Disconnect non compatable witnesses

prandnum commented

2021-04-26 16:12:30 +00:00

(Migrated from gitlab.com)

From Sofie:

Hi all. Since Beatrice halted today and it'll be another PITA to get it up and running again can we implement something to prevent this please?
Right now we have no way of telling which peerplays version witnesses are using to sign blocks. This causes people to not update until they're HF kicked off the chain or (in this case) the chain halts.
Is it possible to add a version to the log?
Now: Got block X from witness Y ...
Wishful thinking: Got block X from witness Y signed by version Z

That way we can anticipate who's going to be dropped after the hardforks.

From Sofie: ``` Hi all. Since Beatrice halted today and it'll be another PITA to get it up and running again can we implement something to prevent this please? Right now we have no way of telling which peerplays version witnesses are using to sign blocks. This causes people to not update until they're HF kicked off the chain or (in this case) the chain halts. Is it possible to add a version to the log? Now: Got block X from witness Y ... Wishful thinking: Got block X from witness Y signed by version Z That way we can anticipate who's going to be dropped after the hardforks. ```

prandnum commented

2021-04-26 16:14:16 +00:00

(Migrated from gitlab.com)

@bobinson @serkixenos @sierra19XX - Please provide your view

CC: @hbelakon

@bobinson @serkixenos @sierra19XX - Please provide your view CC: @hbelakon

bobinson commented

2021-04-27 11:46:09 +00:00

(Migrated from gitlab.com)

We need to take up this us a feature and add meaningful log messages indicating time, version, function or operation being performed, plugin details, account/witness details etc. It will be beneficial to look at logs generated by libraries like Log4J or https://logging.apache.org/log4cxx and design the logs. The way logs are handled is not the best & this needs to be addressed.

hbelakon commented

2021-04-30 02:41:57 +00:00

(Migrated from gitlab.com)

assigned to @sierra19XX

cryptosig commented

2021-05-12 00:37:40 +00:00

(Migrated from gitlab.com)

Maybe the way to handle this is to add it into the blockchain with the witness signature. Perhaps we could make a modification to the witness signature field to add some hex for version or perhaps we could add some data in the optional extensions area to the blocks?

{"previous":"00000002e8e9f28a62c48abbbd9a3ed76928ff29","timestamp":"2017-06-06T16:00:18","witness":"1.6.4","next_secret_hash":"86b9b6883182c63a63bbcd76fedb0a36d5e569ca","previous_secret":"0000000000000000000000000000000000000000","transaction_merkle_root":"0000000000000000000000000000000000000000","extensions":[ MAYBE HERE?],"witness_signature":"HEXVALUE+203abe2ae98f4b266071aa7ead51832f77cd43d05cc91cf192992f34f2bae42b402dfcac59c332aab67f9c029e779eef179c117f92ebc06f614ee50133160c8118","transactions":[]}}

Maybe the way to handle this is to add it into the blockchain with the witness signature. Perhaps we could make a modification to the witness signature field to add some hex for version or perhaps we could add some data in the optional extensions area to the blocks? ``` {"previous":"00000002e8e9f28a62c48abbbd9a3ed76928ff29","timestamp":"2017-06-06T16:00:18","witness":"1.6.4","next_secret_hash":"86b9b6883182c63a63bbcd76fedb0a36d5e569ca","previous_secret":"0000000000000000000000000000000000000000","transaction_merkle_root":"0000000000000000000000000000000000000000","extensions":[ MAYBE HERE?],"witness_signature":"HEXVALUE+203abe2ae98f4b266071aa7ead51832f77cd43d05cc91cf192992f34f2bae42b402dfcac59c332aab67f9c029e779eef179c117f92ebc06f614ee50133160c8118","transactions":[]}} ```

sierra19XX commented

2021-06-29 00:42:31 +00:00

(Migrated from gitlab.com)

changed time estimate to 32h

hbelakon commented

2021-08-23 12:49:04 +00:00

(Migrated from gitlab.com)

Lets de-scope it from current milestone (too many tickets in there)

hbelakon commented

2021-09-09 03:08:48 +00:00

(Migrated from gitlab.com)

De-scoped from current milestone

serkixenos commented

2021-12-31 00:08:26 +00:00

(Migrated from gitlab.com)

unassigned @sierra19XX

serkixenos commented

2022-01-19 23:34:26 +00:00

(Migrated from gitlab.com)

Looks like there is a incomplete feature that allows inspecting hard forks of remote node.
It disrupts sync of two nodes beyond supported hard forks.

https://gitlab.com/PBSA/peerplays/-/blob/master/libraries/net/include/graphene/net/node.hpp

virtual uint32_t estimate_last_known_fork_from_git_revision_timestamp(uint32_t unix_timestamp) const = 0;

https://gitlab.com/PBSA/peerplays/-/blob/master/libraries/app/application.cpp

There are no attempts to estimate supported hard forks

   virtual uint32_t estimate_last_known_fork_from_git_revision_timestamp(uint32_t unix_timestamp) const override {
      return 0; // there are no forks in graphene
   }

Looks like there is a incomplete feature that allows inspecting hard forks of remote node. It disrupts sync of two nodes beyond supported hard forks. https://gitlab.com/PBSA/peerplays/-/blob/master/libraries/net/include/graphene/net/node.hpp ``` virtual uint32_t estimate_last_known_fork_from_git_revision_timestamp(uint32_t unix_timestamp) const = 0; ``` https://gitlab.com/PBSA/peerplays/-/blob/master/libraries/app/application.cpp There are no attempts to estimate supported hard forks ``` virtual uint32_t estimate_last_known_fork_from_git_revision_timestamp(uint32_t unix_timestamp) const override { return 0; // there are no forks in graphene } ```

serkixenos commented

2022-01-20 14:50:42 +00:00

(Migrated from gitlab.com)

assigned to @pavel.baykov

hirunda commented

2022-03-25 12:20:45 +00:00

(Migrated from gitlab.com)

assigned to @hirunda and unassigned @pavel.baykov

hirunda commented

2022-04-10 22:40:18 +00:00

(Migrated from gitlab.com)

changed title from {-enhanc-}e witness{- logging-} to {+Disconnect non compatabl+}e witness{+es+}

changed title from **{-enhanc-}e witness{- logging-}** to **{+Disconnect non compatabl+}e witness{+es+}**

hirunda commented

2022-04-10 22:54:02 +00:00

(Migrated from gitlab.com)

Current solution is going to disconnect witnesses which are not hardfork compatible. Compatibility is determined with last hardfork. The list of hardforks should be updated with the new release. If hardfork is not there, then no damage is expected, but if we introduce the new operation or something similar we usually introduce the hardfork. If witness A updates the software and witness B don't, witness A will communicate with witness B only until the time comes defined with the last hardfork introduced in peerplays. After the last hardfork A and B will remain disconnected until witness B updates the software , after which the connection could be re-established.

Current solution is going to disconnect witnesses which are not hardfork compatible. Compatibility is determined with last hardfork. The list of hardforks should be updated with the new release. If hardfork is not there, then no damage is expected, but if we introduce the new operation or something similar we usually introduce the hardfork. If witness **A** updates the software and witness **B** don't, witness **A** will communicate with witness **B** only until the time comes defined with the last hardfork introduced in peerplays. After the last hardfork **A** and **B** will remain disconnected until witness **B** updates the software , after which the connection could be re-established.

serkixenos commented

2022-04-12 17:18:10 +00:00

(Migrated from gitlab.com)

Remove HARDFORK_CORE_210_TIME file, as it is not used anywhere in the code
Add log messages, so when hardfork lists are compared, we have an output in a witness log to show incompatible nodes (a message like "witness node with ip addr aaa.bbb.ccc.ddd has incompatible hardforks/is running previous software version, or something like that)

Related MR https://gitlab.com/PBSA/peerplays/-/merge_requests/96 1. Remove HARDFORK_CORE_210_TIME file, as it is not used anywhere in the code 2. Add log messages, so when hardfork lists are compared, we have an output in a witness log to show incompatible nodes (a message like "witness node with ip addr aaa.bbb.ccc.ddd has incompatible hardforks/is running previous software version, or something like that)

serkixenos commented

2022-04-26 19:04:57 +00:00

(Migrated from gitlab.com)

assigned to @prandnum

prandnum commented

2022-04-27 14:43:39 +00:00

(Migrated from gitlab.com)

@hirunda @serkixenos

For testing this we would need to change the docker files of at least one witness node to pick an older version of the peerplays codebase right?

@hirunda @serkixenos For testing this we would need to change the docker files of at least one witness node to pick an older version of the peerplays codebase right?

serkixenos commented

2022-05-17 16:05:11 +00:00

(Migrated from gitlab.com)

mentioned in issue #367

serkixenos commented

2022-05-18 12:12:00 +00:00

(Migrated from gitlab.com)

mentioned in issue #368

serkixenos commented

2022-05-18 12:12:36 +00:00

(Migrated from gitlab.com)

Related tickets - #367 and #368

serkixenos commented

2022-05-18 12:13:21 +00:00

(Migrated from gitlab.com)

@hirunda Please provide detailed test instructions here.

prandnum commented

2022-05-23 16:21:15 +00:00

(Migrated from gitlab.com)

attaching the logs. https://gitlab.com/PBSA/documentation/working-documents/-/blob/develop/88/containers.zip

prandnum commented

2022-05-23 17:26:15 +00:00

(Migrated from gitlab.com)

@hirunda @serkixenos From logs what i can infer is that peerplays03 and peerplays04 are also part of the chain and producing blocks.

prandnum commented

2022-05-23 17:31:47 +00:00

(Migrated from gitlab.com)

qa@PBSA-Dev:~$ docker exec -it peerplays-qa-environment_peerplays01_1 bash
root@c816ab9cd4c2:~/peerplays-network# cd src/peerplays/
root@c816ab9cd4c2:~/peerplays-network/src/peerplays# git log -1
fatal: unsafe repository ('/home/peerplays/peerplays-network/src/peerplays' is owned by someone else)
To add an exception for this directory, call:

        git config --global --add safe.directory /home/peerplays/peerplays-network/src/peerplays
root@c816ab9cd4c2:~/peerplays-network/src/peerplays# git config --global --add safe.directory /home/peerplays/peerplays-network/src/peerplays
root@c816ab9cd4c2:~/peerplays-network/src/peerplays# git log -1
commit 398051254396b13b889602d4f92be18a8239f843 (HEAD -> develop, origin/develop)
Merge: f16aa73b fc324559
Author: serkixenos <serki.xenos@gmail.com>
Date:   Tue May 17 00:52:35 2022 +0200

    Merge branch 'develop' of https://gitlab.com/PBSA/peerplays into develop
root@c816ab9cd4c2:~/peerplays-network/src/peerplays#
root@c816ab9cd4c2:~/peerplays-network/src/peerplays#
root@c816ab9cd4c2:~/peerplays-network/src/peerplays#
root@c816ab9cd4c2:~/peerplays-network/src/peerplays#
root@c816ab9cd4c2:~/peerplays-network/src/peerplays# exit
exit
qa@PBSA-Dev:~$ docker exec -it peerplays-qa-environment_peerplays03_1 bash
root@ca0cda3229cb:~/peerplays-network# cd src/peerplays/
root@ca0cda3229cb:~/peerplays-network/src/peerplays# git log -1
fatal: unsafe repository ('/home/peerplays/peerplays-network/src/peerplays' is owned by someone else)
To add an exception for this directory, call:

        git config --global --add safe.directory /home/peerplays/peerplays-network/src/peerplays
root@ca0cda3229cb:~/peerplays-network/src/peerplays# git config --global --add safe.directory /home/peerplays/peerplays-network/src/peerplays
root@ca0cda3229cb:~/peerplays-network/src/peerplays# git log -1
commit de2a89ebceba5a8957068eb0b89d8c7b95a36ed1 (HEAD)
Author: serkixenos <serki.xenos@gmail.com>
Date:   Wed Dec 15 09:27:30 2021 -0400

    Set SON for Hive Mainnet hardfork date to 2021-12-21T00:00:00
root@ca0cda3229cb:~/peerplays-network/src/peerplays#
root@ca0cda3229cb:~/peerplays-network/src/peerplays#
root@ca0cda3229cb:~/peerplays-network/src/peerplays# exit
exit
qa@PBSA-Dev:~$ docker exec -it peerplays-qa-environment_peerplays04_1 bash
root@610cf6c92f80:~/peerplays-network# cd src/peerplays
root@610cf6c92f80:~/peerplays-network/src/peerplays#
root@610cf6c92f80:~/peerplays-network/src/peerplays#
root@610cf6c92f80:~/peerplays-network/src/peerplays# git log -1
fatal: unsafe repository ('/home/peerplays/peerplays-network/src/peerplays' is owned by someone else)
To add an exception for this directory, call:

        git config --global --add safe.directory /home/peerplays/peerplays-network/src/peerplays
root@610cf6c92f80:~/peerplays-network/src/peerplays# git config --global --add safe.directory /home/peerplays/peerplays-network/src/peerplays
root@610cf6c92f80:~/peerplays-network/src/peerplays#
root@610cf6c92f80:~/peerplays-network/src/peerplays#
root@610cf6c92f80:~/peerplays-network/src/peerplays# git log -1
commit de2a89ebceba5a8957068eb0b89d8c7b95a36ed1 (HEAD)
Author: serkixenos <serki.xenos@gmail.com>
Date:   Wed Dec 15 09:27:30 2021 -0400

    Set SON for Hive Mainnet hardfork date to 2021-12-21T00:00:00
root@610cf6c92f80:~/peerplays-network/src/peerplays# exit
exit
qa@PBSA-Dev:~$
qa@PBSA-Dev:~$
qa@PBSA-Dev:~$ docker exec -it peerplays-qa-environment_peerplays02_1 bash
root@0da7f9643c04:~/peerplays-network# cd src/peerplays/
root@0da7f9643c04:~/peerplays-network/src/peerplays# git log -1
fatal: unsafe repository ('/home/peerplays/peerplays-network/src/peerplays' is owned by someone else)
To add an exception for this directory, call:

        git config --global --add safe.directory /home/peerplays/peerplays-network/src/peerplays
root@0da7f9643c04:~/peerplays-network/src/peerplays# git config --global --add safe.directory /home/peerplays/peerplays-network/src/peerplays
root@0da7f9643c04:~/peerplays-network/src/peerplays# git log -1
commit 398051254396b13b889602d4f92be18a8239f843 (HEAD -> develop, origin/develop)
Merge: f16aa73b fc324559
Author: serkixenos <serki.xenos@gmail.com>
Date:   Tue May 17 00:52:35 2022 +0200

    Merge branch 'develop' of https://gitlab.com/PBSA/peerplays into develop
root@0da7f9643c04:~/peerplays-network/src/peerplays#
root@0da7f9643c04:~/peerplays-network/src/peerplays#
root@0da7f9643c04:~/peerplays-network/src/peerplays#

``` qa@PBSA-Dev:~$ docker exec -it peerplays-qa-environment_peerplays01_1 bash root@c816ab9cd4c2:~/peerplays-network# cd src/peerplays/ root@c816ab9cd4c2:~/peerplays-network/src/peerplays# git log -1 fatal: unsafe repository ('/home/peerplays/peerplays-network/src/peerplays' is owned by someone else) To add an exception for this directory, call: git config --global --add safe.directory /home/peerplays/peerplays-network/src/peerplays root@c816ab9cd4c2:~/peerplays-network/src/peerplays# git config --global --add safe.directory /home/peerplays/peerplays-network/src/peerplays root@c816ab9cd4c2:~/peerplays-network/src/peerplays# git log -1 commit 398051254396b13b889602d4f92be18a8239f843 (HEAD -> develop, origin/develop) Merge: f16aa73b fc324559 Author: serkixenos <serki.xenos@gmail.com> Date: Tue May 17 00:52:35 2022 +0200 Merge branch 'develop' of https://gitlab.com/PBSA/peerplays into develop root@c816ab9cd4c2:~/peerplays-network/src/peerplays# root@c816ab9cd4c2:~/peerplays-network/src/peerplays# root@c816ab9cd4c2:~/peerplays-network/src/peerplays# root@c816ab9cd4c2:~/peerplays-network/src/peerplays# root@c816ab9cd4c2:~/peerplays-network/src/peerplays# exit exit qa@PBSA-Dev:~$ docker exec -it peerplays-qa-environment_peerplays03_1 bash root@ca0cda3229cb:~/peerplays-network# cd src/peerplays/ root@ca0cda3229cb:~/peerplays-network/src/peerplays# git log -1 fatal: unsafe repository ('/home/peerplays/peerplays-network/src/peerplays' is owned by someone else) To add an exception for this directory, call: git config --global --add safe.directory /home/peerplays/peerplays-network/src/peerplays root@ca0cda3229cb:~/peerplays-network/src/peerplays# git config --global --add safe.directory /home/peerplays/peerplays-network/src/peerplays root@ca0cda3229cb:~/peerplays-network/src/peerplays# git log -1 commit de2a89ebceba5a8957068eb0b89d8c7b95a36ed1 (HEAD) Author: serkixenos <serki.xenos@gmail.com> Date: Wed Dec 15 09:27:30 2021 -0400 Set SON for Hive Mainnet hardfork date to 2021-12-21T00:00:00 root@ca0cda3229cb:~/peerplays-network/src/peerplays# root@ca0cda3229cb:~/peerplays-network/src/peerplays# root@ca0cda3229cb:~/peerplays-network/src/peerplays# exit exit qa@PBSA-Dev:~$ docker exec -it peerplays-qa-environment_peerplays04_1 bash root@610cf6c92f80:~/peerplays-network# cd src/peerplays root@610cf6c92f80:~/peerplays-network/src/peerplays# root@610cf6c92f80:~/peerplays-network/src/peerplays# root@610cf6c92f80:~/peerplays-network/src/peerplays# git log -1 fatal: unsafe repository ('/home/peerplays/peerplays-network/src/peerplays' is owned by someone else) To add an exception for this directory, call: git config --global --add safe.directory /home/peerplays/peerplays-network/src/peerplays root@610cf6c92f80:~/peerplays-network/src/peerplays# git config --global --add safe.directory /home/peerplays/peerplays-network/src/peerplays root@610cf6c92f80:~/peerplays-network/src/peerplays# root@610cf6c92f80:~/peerplays-network/src/peerplays# root@610cf6c92f80:~/peerplays-network/src/peerplays# git log -1 commit de2a89ebceba5a8957068eb0b89d8c7b95a36ed1 (HEAD) Author: serkixenos <serki.xenos@gmail.com> Date: Wed Dec 15 09:27:30 2021 -0400 Set SON for Hive Mainnet hardfork date to 2021-12-21T00:00:00 root@610cf6c92f80:~/peerplays-network/src/peerplays# exit exit qa@PBSA-Dev:~$ qa@PBSA-Dev:~$ qa@PBSA-Dev:~$ docker exec -it peerplays-qa-environment_peerplays02_1 bash root@0da7f9643c04:~/peerplays-network# cd src/peerplays/ root@0da7f9643c04:~/peerplays-network/src/peerplays# git log -1 fatal: unsafe repository ('/home/peerplays/peerplays-network/src/peerplays' is owned by someone else) To add an exception for this directory, call: git config --global --add safe.directory /home/peerplays/peerplays-network/src/peerplays root@0da7f9643c04:~/peerplays-network/src/peerplays# git config --global --add safe.directory /home/peerplays/peerplays-network/src/peerplays root@0da7f9643c04:~/peerplays-network/src/peerplays# git log -1 commit 398051254396b13b889602d4f92be18a8239f843 (HEAD -> develop, origin/develop) Merge: f16aa73b fc324559 Author: serkixenos <serki.xenos@gmail.com> Date: Tue May 17 00:52:35 2022 +0200 Merge branch 'develop' of https://gitlab.com/PBSA/peerplays into develop root@0da7f9643c04:~/peerplays-network/src/peerplays# root@0da7f9643c04:~/peerplays-network/src/peerplays# root@0da7f9643c04:~/peerplays-network/src/peerplays# ```

prandnum commented

2022-05-30 18:13:30 +00:00

(Migrated from gitlab.com)

@hirunda @serkixenos

Changed peerplyas03 and peerplays04 to build from an older peerplyas commit and when the docker qa env is brought up the witness peerplays03 and peerplays04 are unable to join as expected

2935000ms th_a       witness.cpp:195               block_production_loo ] Not producing block because production is disabled until we receive a recent block (see: --enable-stale-production)
2936000ms th_a       witness.cpp:195               block_production_loo ] Not producing block because production is disabled until we receive a recent block (see: --enable-stale-production)
2937000ms th_a       witness.cpp:195               block_production_loo ] Not producing block because production is disabled until we receive a recent block (see: --enable-stale-production)
2938000ms th_a       witness.cpp:195               block_production_loo ] Not producing block because production is disabled until we receive a recent block (see: --enable-stale-production)
2939000ms th_a       witness.cpp:195               block_production_loo ] Not producing block because production is disabled until we receive a recent block (see: --enable-stale-production)
2940000ms th_a       witness.cpp:195               block_production_loo ] Not producing block because production is disabled until we receive a recent block (see: --enable-stale-production)
2941000ms th_a       witness.cpp:195               block_production_loo ] Not producing block because production is disabled until we receive a recent block (see: --enable-stale-production)
2942000ms th_a       witness.cpp:195               block_production_loo ] Not producing block because production is disabled until we receive a recent block (see: --enable-stale-production)

but the witness nodes were seen as crashing, not all, but 8.

qa@PBSA-Dev:/var/kiran$ docker ps -a
CONTAINER ID   IMAGE                   COMMAND                  CREATED        STATUS          PORTS                                                                                                                                             NAMES
1f66e7677a1c   peerplays04             "./witness_node -d .���"   16 hours ago   Up 16 hours     22/tcp, 0.0.0.0:8093->8090/tcp, :::8093->8090/tcp, 0.0.0.0:9780->9777/tcp, :::9780->9777/tcp                                                      peerplays-qa-environment_peerplays04_1
ba60bbbdd3d1   peerplays03             "./witness_node -d .���"   16 hours ago   Up 16 hours     22/tcp, 0.0.0.0:8092->8090/tcp, :::8092->8090/tcp, 0.0.0.0:9779->9777/tcp, :::9779->9777/tcp                                                      peerplays-qa-environment_peerplays03_1
7fcc118fb531   peerplays01             "./witness_node -d .���"   16 hours ago   Up 16 hours     0.0.0.0:8090->8090/tcp, :::8090->8090/tcp, 22/tcp, 0.0.0.0:9777->9777/tcp, :::9777->9777/tcp                                                      peerplays-qa-environment_peerplays01_1
6500036194c3   peerplays16             "./witness_node -d .���"   16 hours ago   Up 16 hours     22/tcp, 0.0.0.0:8105->8090/tcp, :::8105->8090/tcp, 0.0.0.0:9792->9777/tcp, :::9792->9777/tcp                                                      peerplays-qa-environment_peerplays16_1
5a36c7e40d2c   peerplays06             "./witness_node -d .���"   16 hours ago   Up 20 seconds   22/tcp, 0.0.0.0:8095->8090/tcp, :::8095->8090/tcp, 0.0.0.0:9782->9777/tcp, :::9782->9777/tcp                                                      peerplays-qa-environment_peerplays06_1
13655be8c586   peerplays11             "./witness_node -d .���"   16 hours ago   Up 47 seconds   22/tcp, 0.0.0.0:8100->8090/tcp, :::8100->8090/tcp, 0.0.0.0:9787->9777/tcp, :::9787->9777/tcp                                                      peerplays-qa-environment_peerplays11_1
734ba453e0a8   peerplays02             "./witness_node -d .���"   16 hours ago   Up 8 seconds    22/tcp, 0.0.0.0:8091->8090/tcp, :::8091->8090/tcp, 0.0.0.0:9778->9777/tcp, :::9778->9777/tcp                                                      peerplays-qa-environment_peerplays02_1
b83281ac1f4d   peerplays05             "./witness_node -d .���"   16 hours ago   Up 5 seconds    22/tcp, 0.0.0.0:8094->8090/tcp, :::8094->8090/tcp, 0.0.0.0:9781->9777/tcp, :::9781->9777/tcp                                                      peerplays-qa-environment_peerplays05_1
6618b60fed9f   peerplays14             "./witness_node -d .���"   16 hours ago   Up 16 hours     22/tcp, 0.0.0.0:8103->8090/tcp, :::8103->8090/tcp, 0.0.0.0:9790->9777/tcp, :::9790->9777/tcp                                                      peerplays-qa-environment_peerplays14_1
fa8840c33cf8   peerplays13             "./witness_node -d .���"   16 hours ago   Up 16 hours     22/tcp, 0.0.0.0:8102->8090/tcp, :::8102->8090/tcp, 0.0.0.0:9789->9777/tcp, :::9789->9777/tcp                                                      peerplays-qa-environment_peerplays13_1
f587abd37350   peerplays07             "./witness_node -d .���"   16 hours ago   Up 32 seconds   22/tcp, 0.0.0.0:8096->8090/tcp, :::8096->8090/tcp, 0.0.0.0:9783->9777/tcp, :::9783->9777/tcp                                                      peerplays-qa-environment_peerplays07_1
a792f43c12ef   peerplays10             "./witness_node -d .���"   16 hours ago   Up 8 hours      22/tcp, 0.0.0.0:8099->8090/tcp, :::8099->8090/tcp, 0.0.0.0:9786->9777/tcp, :::9786->9777/tcp                                                      peerplays-qa-environment_peerplays10_1
c6000959ac8d   peerplays08             "./witness_node -d .���"   16 hours ago   Up 4 hours      22/tcp, 0.0.0.0:8097->8090/tcp, :::8097->8090/tcp, 0.0.0.0:9784->9777/tcp, :::9784->9777/tcp                                                      peerplays-qa-environment_peerplays08_1
5bc2da987417   peerplays12             "./witness_node -d .���"   16 hours ago   Up 16 hours     22/tcp, 0.0.0.0:8101->8090/tcp, :::8101->8090/tcp, 0.0.0.0:9788->9777/tcp, :::9788->9777/tcp                                                      peerplays-qa-environment_peerplays12_1
286e1837ae73   peerplays09             "./witness_node -d .���"   16 hours ago   Up 11 hours     22/tcp, 0.0.0.0:8098->8090/tcp, :::8098->8090/tcp, 0.0.0.0:9785->9777/tcp, :::9785->9777/tcp                                                      peerplays-qa-environment_peerplays09_1
8cefacfe1387   peerplays15             "./witness_node -d .���"   16 hours ago   Up 16 hours     22/tcp, 0.0.0.0:8104->8090/tcp, :::8104->8090/tcp, 0.0.0.0:9791->9777/tcp, :::9791->9777/tcp                                                      peerplays-qa-environment_peerplays15_1
b1a010c1cae2   peerplays-baseabc       "/usr/sbin/sshd -D"      16 hours ago   Up 16 hours     22/tcp, 8090/tcp, 9777/tcp                                                                                                                        peerplays-qa-environment_peerplays-baseabc_1
89de65ba8e9e   hive-for-peerplays      "./hived -d ./hived_���"   16 hours ago   Up 16 hours     22/tcp, 0.0.0.0:28090-28091->28090-28091/tcp, :::28090-28091->28090-28091/tcp                                                                     peerplays-qa-environment_hive-for-peerplays_1
21ca2f21a2cd   bitcoin-for-peerplays   "./bitcoind -datadir���"   16 hours ago   Up 16 hours     0.0.0.0:8332->8332/tcp, :::8332->8332/tcp, 0.0.0.0:11111->11111/tcp, :::11111->11111/tcp, 22/tcp, 0.0.0.0:18444->18444/tcp, :::18444->18444/tcp   peerplays-qa-environment_bitcoin-for-peerplays_1
qa@PBSA-Dev:/var/kiran$

peerplays05 and peerplays08 logs are attached: available here - https://gitlab.com/PBSA/documentation/working-documents/-/tree/develop/88/30052022

@hirunda @serkixenos Changed peerplyas03 and peerplays04 to build from an older peerplyas commit and when the docker qa env is brought up the witness peerplays03 and peerplays04 are unable to join as expected ``` 2935000ms th_a witness.cpp:195 block_production_loo ] Not producing block because production is disabled until we receive a recent block (see: --enable-stale-production) 2936000ms th_a witness.cpp:195 block_production_loo ] Not producing block because production is disabled until we receive a recent block (see: --enable-stale-production) 2937000ms th_a witness.cpp:195 block_production_loo ] Not producing block because production is disabled until we receive a recent block (see: --enable-stale-production) 2938000ms th_a witness.cpp:195 block_production_loo ] Not producing block because production is disabled until we receive a recent block (see: --enable-stale-production) 2939000ms th_a witness.cpp:195 block_production_loo ] Not producing block because production is disabled until we receive a recent block (see: --enable-stale-production) 2940000ms th_a witness.cpp:195 block_production_loo ] Not producing block because production is disabled until we receive a recent block (see: --enable-stale-production) 2941000ms th_a witness.cpp:195 block_production_loo ] Not producing block because production is disabled until we receive a recent block (see: --enable-stale-production) 2942000ms th_a witness.cpp:195 block_production_loo ] Not producing block because production is disabled until we receive a recent block (see: --enable-stale-production) ``` but the witness nodes were seen as crashing, not all, but 8. ``` qa@PBSA-Dev:/var/kiran$ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 1f66e7677a1c peerplays04 "./witness_node -d .��" 16 hours ago Up 16 hours 22/tcp, 0.0.0.0:8093->8090/tcp, :::8093->8090/tcp, 0.0.0.0:9780->9777/tcp, :::9780->9777/tcp peerplays-qa-environment_peerplays04_1 ba60bbbdd3d1 peerplays03 "./witness_node -d .��" 16 hours ago Up 16 hours 22/tcp, 0.0.0.0:8092->8090/tcp, :::8092->8090/tcp, 0.0.0.0:9779->9777/tcp, :::9779->9777/tcp peerplays-qa-environment_peerplays03_1 7fcc118fb531 peerplays01 "./witness_node -d .��" 16 hours ago Up 16 hours 0.0.0.0:8090->8090/tcp, :::8090->8090/tcp, 22/tcp, 0.0.0.0:9777->9777/tcp, :::9777->9777/tcp peerplays-qa-environment_peerplays01_1 6500036194c3 peerplays16 "./witness_node -d .��" 16 hours ago Up 16 hours 22/tcp, 0.0.0.0:8105->8090/tcp, :::8105->8090/tcp, 0.0.0.0:9792->9777/tcp, :::9792->9777/tcp peerplays-qa-environment_peerplays16_1 5a36c7e40d2c peerplays06 "./witness_node -d .��" 16 hours ago Up 20 seconds 22/tcp, 0.0.0.0:8095->8090/tcp, :::8095->8090/tcp, 0.0.0.0:9782->9777/tcp, :::9782->9777/tcp peerplays-qa-environment_peerplays06_1 13655be8c586 peerplays11 "./witness_node -d .��" 16 hours ago Up 47 seconds 22/tcp, 0.0.0.0:8100->8090/tcp, :::8100->8090/tcp, 0.0.0.0:9787->9777/tcp, :::9787->9777/tcp peerplays-qa-environment_peerplays11_1 734ba453e0a8 peerplays02 "./witness_node -d .��" 16 hours ago Up 8 seconds 22/tcp, 0.0.0.0:8091->8090/tcp, :::8091->8090/tcp, 0.0.0.0:9778->9777/tcp, :::9778->9777/tcp peerplays-qa-environment_peerplays02_1 b83281ac1f4d peerplays05 "./witness_node -d .��" 16 hours ago Up 5 seconds 22/tcp, 0.0.0.0:8094->8090/tcp, :::8094->8090/tcp, 0.0.0.0:9781->9777/tcp, :::9781->9777/tcp peerplays-qa-environment_peerplays05_1 6618b60fed9f peerplays14 "./witness_node -d .��" 16 hours ago Up 16 hours 22/tcp, 0.0.0.0:8103->8090/tcp, :::8103->8090/tcp, 0.0.0.0:9790->9777/tcp, :::9790->9777/tcp peerplays-qa-environment_peerplays14_1 fa8840c33cf8 peerplays13 "./witness_node -d .��" 16 hours ago Up 16 hours 22/tcp, 0.0.0.0:8102->8090/tcp, :::8102->8090/tcp, 0.0.0.0:9789->9777/tcp, :::9789->9777/tcp peerplays-qa-environment_peerplays13_1 f587abd37350 peerplays07 "./witness_node -d .��" 16 hours ago Up 32 seconds 22/tcp, 0.0.0.0:8096->8090/tcp, :::8096->8090/tcp, 0.0.0.0:9783->9777/tcp, :::9783->9777/tcp peerplays-qa-environment_peerplays07_1 a792f43c12ef peerplays10 "./witness_node -d .��" 16 hours ago Up 8 hours 22/tcp, 0.0.0.0:8099->8090/tcp, :::8099->8090/tcp, 0.0.0.0:9786->9777/tcp, :::9786->9777/tcp peerplays-qa-environment_peerplays10_1 c6000959ac8d peerplays08 "./witness_node -d .��" 16 hours ago Up 4 hours 22/tcp, 0.0.0.0:8097->8090/tcp, :::8097->8090/tcp, 0.0.0.0:9784->9777/tcp, :::9784->9777/tcp peerplays-qa-environment_peerplays08_1 5bc2da987417 peerplays12 "./witness_node -d .��" 16 hours ago Up 16 hours 22/tcp, 0.0.0.0:8101->8090/tcp, :::8101->8090/tcp, 0.0.0.0:9788->9777/tcp, :::9788->9777/tcp peerplays-qa-environment_peerplays12_1 286e1837ae73 peerplays09 "./witness_node -d .��" 16 hours ago Up 11 hours 22/tcp, 0.0.0.0:8098->8090/tcp, :::8098->8090/tcp, 0.0.0.0:9785->9777/tcp, :::9785->9777/tcp peerplays-qa-environment_peerplays09_1 8cefacfe1387 peerplays15 "./witness_node -d .��" 16 hours ago Up 16 hours 22/tcp, 0.0.0.0:8104->8090/tcp, :::8104->8090/tcp, 0.0.0.0:9791->9777/tcp, :::9791->9777/tcp peerplays-qa-environment_peerplays15_1 b1a010c1cae2 peerplays-baseabc "/usr/sbin/sshd -D" 16 hours ago Up 16 hours 22/tcp, 8090/tcp, 9777/tcp peerplays-qa-environment_peerplays-baseabc_1 89de65ba8e9e hive-for-peerplays "./hived -d ./hived_��" 16 hours ago Up 16 hours 22/tcp, 0.0.0.0:28090-28091->28090-28091/tcp, :::28090-28091->28090-28091/tcp peerplays-qa-environment_hive-for-peerplays_1 21ca2f21a2cd bitcoin-for-peerplays "./bitcoind -datadir��" 16 hours ago Up 16 hours 0.0.0.0:8332->8332/tcp, :::8332->8332/tcp, 0.0.0.0:11111->11111/tcp, :::11111->11111/tcp, 22/tcp, 0.0.0.0:18444->18444/tcp, :::18444->18444/tcp peerplays-qa-environment_bitcoin-for-peerplays_1 qa@PBSA-Dev:/var/kiran$ ``` peerplays05 and peerplays08 logs are attached: available here - https://gitlab.com/PBSA/documentation/working-documents/-/tree/develop/88/30052022

prandnum commented

2022-07-19 18:42:09 +00:00

(Migrated from gitlab.com)

The outdated witnesses(peerplays02 and peerplays04) have not joined the cluster as seen in logs below:

peerplays10_1                 | 2205510ms th_a       peerplays_sidechain_plugin.cpp:458 son_processing       ] Scheduled SON: 1.33.4 Now: 2022-07-19T18:36:42 
peerplays07_1                 | 2205510ms th_a       peerplays_sidechain_plugin.cpp:458 son_processing       ] Scheduled SON: 1.33.4 Now: 2022-07-19T18:36:42 
peerplays08_1                 | 2205511ms th_a       peerplays_sidechain_plugin.cpp:458 son_processing       ] Scheduled SON: 1.33.4 Now: 2022-07-19T18:36:42 
peerplays03_1                 | 2205512ms th_a       peerplays_sidechain_plugin.cpp:458 son_processing       ] Scheduled SON: 1.33.4 Now: 2022-07-19T18:36:42 
peerplays11_1                 | 2205511ms th_a       peerplays_sidechain_plugin.cpp:458 son_processing       ] Scheduled SON: 1.33.4 Now: 2022-07-19T18:36:42 
peerplays01_1                 | 2205512ms th_a       peerplays_sidechain_plugin.cpp:458 son_processing       ] Scheduled SON: 1.33.4 Now: 2022-07-19T18:36:42 
peerplays14_1                 | 2205511ms th_a       peerplays_sidechain_plugin.cpp:458 son_processing       ] Scheduled SON: 1.33.4 Now: 2022-07-19T18:36:42 
peerplays12_1                 | 2205512ms th_a       peerplays_sidechain_plugin.cpp:458 son_processing       ] Scheduled SON: 1.33.4 Now: 2022-07-19T18:36:42 
peerplays15_1                 | 2205512ms th_a       peerplays_sidechain_plugin.cpp:458 son_processing       ] Scheduled SON: 1.33.4 Now: 2022-07-19T18:36:42 
peerplays13_1                 | 2205511ms th_a       peerplays_sidechain_plugin.cpp:458 son_processing       ] Scheduled SON: 1.33.4 Now: 2022-07-19T18:36:42 
peerplays06_1                 | 2205513ms th_a       peerplays_sidechain_plugin.cpp:458 son_processing       ] Scheduled SON: 1.33.4 Now: 2022-07-19T18:36:42 
peerplays05_1                 | 2205514ms th_a       peerplays_sidechain_plugin.cpp:458 son_processing       ] Scheduled SON: 1.33.4 Now: 2022-07-19T18:36:42 
peerplays02_1                 | 2206000ms th_a       witness.cpp:195               block_production_loo ] Not producing block because production is disabled until we receive a recent block (see: --enable-stale-production)
peerplays04_1                 | 2206000ms th_a       witness.cpp:195               block_production_loo ] Not producing block because production is disabled until we receive a recent block (see: --enable-stale-production)
peerplays02_1                 | 2207000ms th_a       witness.cpp:195               block_production_loo ] Not producing block because production is disabled until we receive a recent block (see: --enable-stale-production)
peerplays04_1                 | 2207000ms th_a       witness.cpp:195               block_production_loo ] Not producing block because production is disabled until we receive a recent block (see: --enable-stale-production)

peerplays08_1                 | 2208000ms th_a       witness.cpp:267               maybe_produce_block  ] slot: 1 scheduled_witness: 1.6.8 scheduled_time: 2022-07-19T18:36:48 now: 2022-07-19T18:36:48 
peerplays02_1                 | 2208000ms th_a       witness.cpp:195               block_production_loo ] Not producing block because production is disabled until we receive a recent block (see: --enable-stale-production)
peerplays04_1                 | 2208000ms th_a       witness.cpp:195               block_production_loo ] Not producing block because production is disabled until we receive a recent block (see: --enable-stale-production)
peerplays08_1                 | 2208008ms th_a       witness.cpp:191               block_production_loo ] Generated block #351 with timestamp 2022-07-19T18:36:48 at time 2022-07-19T18:36:48
peerplays01_1                 | 2208013ms th_a       application.cpp:429           handle_block         ] Got block: #351 time: 2022-07-19T18:36:48 latency: 13 ms from: init7  irreversible: 343 (-8)
peerplays13_1                 | 2208013ms th_a       application.cpp:429           handle_block         ] Got block: #351 time: 2022-07-19T18:36:48 latency: 13 ms from: init7  irreversible: 343 (-8)
peerplays15_1                 | 2208013ms th_a       application.cpp:429           handle_block         ] Got block: #351 time: 2022-07-19T18:36:48 latency: 13 ms from: init7  irreversible: 343 (-8)
peerplays10_1                 | 2208013ms th_a       application.cpp:429           handle_block         ] Got block: #351 time: 2022-07-19T18:36:48 latency: 13 ms from: init7  irreversible: 343 (-8)
peerplays03_1                 | 2208013ms th_a       application.cpp:429           handle_block         ] Got block: #351 time: 2022-07-19T18:36:48 latency: 13 ms from: init7  irreversible: 343 (-8)
peerplays16_1                 | 2208015ms th_a       application.cpp:429           handle_block         ] Got block: #351 time: 2022-07-19T18:36:48 latency: 14 ms from: init7  irreversible: 343 (-8)
peerplays11_1                 | 2208016ms th_a       application.cpp:429           handle_block         ] Got block: #351 time: 2022-07-19T18:36:48 latency: 16 ms from: init7  irreversible: 343 (-8)
peerplays09_1                 | 2208016ms th_a       application.cpp:429           handle_block         ] Got block: #351 time: 2022-07-19T18:36:48 latency: 15 ms from: init7  irreversible: 343 (-8)
peerplays07_1                 | 2208016ms th_a       application.cpp:429           handle_block         ] Got block: #351 time: 2022-07-19T18:36:48 latency: 16 ms from: init7  irreversible: 343 (-8)
peerplays12_1                 | 2208016ms th_a       application.cpp:429           handle_block         ] Got block: #351 time: 2022-07-19T18:36:48 latency: 16 ms from: init7  irreversible: 343 (-8)
peerplays06_1                 | 2208015ms th_a       application.cpp:429           handle_block         ] Got block: #351 time: 2022-07-19T18:36:48 latency: 15 ms from: init7  irreversible: 343 (-8)
peerplays05_1                 | 2208016ms th_a       application.cpp:429           handle_block         ] Got block: #351 time: 2022-07-19T18:36:48 latency: 16 ms from: init7  irreversible: 343 (-8)
peerplays14_1                 | 2208017ms th_a       application.cpp:429           handle_block         ] Got block: #351 time: 2022-07-19T18:36:48 latency: 17 ms from: init7  irreversible: 343 (-8)
peerplays08_1                 | 2208502ms th_a       peerplays_sidechain_plugin.cpp:458 son_processing       ] Scheduled SON: 1.33.3 Now: 2022-07-19T18:36:45 
peerplays10_1                 | 2208515ms th_a       peerplays_sidechain_plugin.cpp:458 son_processing       ] Scheduled SON: 1.33.3 Now: 2022-07-19T18:36:45 
peerplays01_1                 | 2208515ms th_a       peerplays_sidechain_plugin.cpp:458 son_processing       ] Scheduled SON: 1.33.3 Now: 2022-07-19T18:36:45 
peerplays15_1                 | 2208515ms th_a       peerplays_sidechain_plugin.cpp:458 son_processing       ] Scheduled SON: 1.33.3 Now: 2022-07-19T18:36:45 
peerplays03_1                 | 2208515ms th_a       peerplays_sidechain_plugin.cpp:458 son_processing       ] Scheduled SON: 1.33.3 Now: 2022-07-19T18:36:45 
peerplays13_1                 | 2208515ms th_a       peerplays_sidechain_plugin.cpp:458 son_processing       ] Scheduled SON: 1.33.3 Now: 2022-07-19T18:36:45 
peerplays16_1                 | 2208516ms th_a       peerplays_sidechain_plugin.cpp:458 son_processing       ] Scheduled SON: 1.33.3 Now: 2022-07-19T18:36:45 
peerplays06_1                 | 2208517ms th_a       peerplays_sidechain_plugin.cpp:458 son_processing       ] Scheduled SON: 1.33.3 Now: 2022-07-19T18:36:45 
peerplays11_1                 | 2208518ms th_a       peerplays_sidechain_plugin.cpp:458 son_processing       ] Scheduled SON: 1.33.3 Now: 2022-07-19T18:36:45 
peerplays12_1                 | 2208518ms th_a       peerplays_sidechain_plugin.cpp:458 son_processing       ] Scheduled SON: 1.33.3 Now: 2022-07-19T18:36:45 
peerplays07_1                 | 2208518ms th_a       peerplays_sidechain_plugin.cpp:458 son_processing       ] Scheduled SON: 1.33.3 Now: 2022-07-19T18:36:45 
peerplays14_1                 | 2208519ms th_a       peerplays_sidechain_plugin.cpp:458 son_processing       ] Scheduled SON: 1.33.3 Now: 2022-07-19T18:36:45 
peerplays05_1                 | 2208519ms th_a       peerplays_sidechain_plugin.cpp:458 son_processing       ] Scheduled SON: 1.33.3 Now: 2022-07-19T18:36:45 
peerplays09_1                 | 2208524ms th_a       peerplays_sidechain_plugin.cpp:458 son_processing       ] Scheduled SON: 1.33.3 Now: 2022-07-19T18:36:45 
peerplays02_1                 | 2209000ms th_a       witness.cpp:195               block_production_loo ] Not producing block because production is disabled until we receive a recent block (see: -

@hirunda @serkixenos

witness crash needs to be seen, hence keeping the system under observation

The outdated witnesses(peerplays02 and peerplays04) have not joined the cluster as seen in logs below: ``` peerplays10_1 | 2205510ms th_a peerplays_sidechain_plugin.cpp:458 son_processing ] Scheduled SON: 1.33.4 Now: 2022-07-19T18:36:42 peerplays07_1 | 2205510ms th_a peerplays_sidechain_plugin.cpp:458 son_processing ] Scheduled SON: 1.33.4 Now: 2022-07-19T18:36:42 peerplays08_1 | 2205511ms th_a peerplays_sidechain_plugin.cpp:458 son_processing ] Scheduled SON: 1.33.4 Now: 2022-07-19T18:36:42 peerplays03_1 | 2205512ms th_a peerplays_sidechain_plugin.cpp:458 son_processing ] Scheduled SON: 1.33.4 Now: 2022-07-19T18:36:42 peerplays11_1 | 2205511ms th_a peerplays_sidechain_plugin.cpp:458 son_processing ] Scheduled SON: 1.33.4 Now: 2022-07-19T18:36:42 peerplays01_1 | 2205512ms th_a peerplays_sidechain_plugin.cpp:458 son_processing ] Scheduled SON: 1.33.4 Now: 2022-07-19T18:36:42 peerplays14_1 | 2205511ms th_a peerplays_sidechain_plugin.cpp:458 son_processing ] Scheduled SON: 1.33.4 Now: 2022-07-19T18:36:42 peerplays12_1 | 2205512ms th_a peerplays_sidechain_plugin.cpp:458 son_processing ] Scheduled SON: 1.33.4 Now: 2022-07-19T18:36:42 peerplays15_1 | 2205512ms th_a peerplays_sidechain_plugin.cpp:458 son_processing ] Scheduled SON: 1.33.4 Now: 2022-07-19T18:36:42 peerplays13_1 | 2205511ms th_a peerplays_sidechain_plugin.cpp:458 son_processing ] Scheduled SON: 1.33.4 Now: 2022-07-19T18:36:42 peerplays06_1 | 2205513ms th_a peerplays_sidechain_plugin.cpp:458 son_processing ] Scheduled SON: 1.33.4 Now: 2022-07-19T18:36:42 peerplays05_1 | 2205514ms th_a peerplays_sidechain_plugin.cpp:458 son_processing ] Scheduled SON: 1.33.4 Now: 2022-07-19T18:36:42 peerplays02_1 | 2206000ms th_a witness.cpp:195 block_production_loo ] Not producing block because production is disabled until we receive a recent block (see: --enable-stale-production) peerplays04_1 | 2206000ms th_a witness.cpp:195 block_production_loo ] Not producing block because production is disabled until we receive a recent block (see: --enable-stale-production) peerplays02_1 | 2207000ms th_a witness.cpp:195 block_production_loo ] Not producing block because production is disabled until we receive a recent block (see: --enable-stale-production) peerplays04_1 | 2207000ms th_a witness.cpp:195 block_production_loo ] Not producing block because production is disabled until we receive a recent block (see: --enable-stale-production) peerplays08_1 | 2208000ms th_a witness.cpp:267 maybe_produce_block ] slot: 1 scheduled_witness: 1.6.8 scheduled_time: 2022-07-19T18:36:48 now: 2022-07-19T18:36:48 peerplays02_1 | 2208000ms th_a witness.cpp:195 block_production_loo ] Not producing block because production is disabled until we receive a recent block (see: --enable-stale-production) peerplays04_1 | 2208000ms th_a witness.cpp:195 block_production_loo ] Not producing block because production is disabled until we receive a recent block (see: --enable-stale-production) peerplays08_1 | 2208008ms th_a witness.cpp:191 block_production_loo ] Generated block #351 with timestamp 2022-07-19T18:36:48 at time 2022-07-19T18:36:48 peerplays01_1 | 2208013ms th_a application.cpp:429 handle_block ] Got block: #351 time: 2022-07-19T18:36:48 latency: 13 ms from: init7 irreversible: 343 (-8) peerplays13_1 | 2208013ms th_a application.cpp:429 handle_block ] Got block: #351 time: 2022-07-19T18:36:48 latency: 13 ms from: init7 irreversible: 343 (-8) peerplays15_1 | 2208013ms th_a application.cpp:429 handle_block ] Got block: #351 time: 2022-07-19T18:36:48 latency: 13 ms from: init7 irreversible: 343 (-8) peerplays10_1 | 2208013ms th_a application.cpp:429 handle_block ] Got block: #351 time: 2022-07-19T18:36:48 latency: 13 ms from: init7 irreversible: 343 (-8) peerplays03_1 | 2208013ms th_a application.cpp:429 handle_block ] Got block: #351 time: 2022-07-19T18:36:48 latency: 13 ms from: init7 irreversible: 343 (-8) peerplays16_1 | 2208015ms th_a application.cpp:429 handle_block ] Got block: #351 time: 2022-07-19T18:36:48 latency: 14 ms from: init7 irreversible: 343 (-8) peerplays11_1 | 2208016ms th_a application.cpp:429 handle_block ] Got block: #351 time: 2022-07-19T18:36:48 latency: 16 ms from: init7 irreversible: 343 (-8) peerplays09_1 | 2208016ms th_a application.cpp:429 handle_block ] Got block: #351 time: 2022-07-19T18:36:48 latency: 15 ms from: init7 irreversible: 343 (-8) peerplays07_1 | 2208016ms th_a application.cpp:429 handle_block ] Got block: #351 time: 2022-07-19T18:36:48 latency: 16 ms from: init7 irreversible: 343 (-8) peerplays12_1 | 2208016ms th_a application.cpp:429 handle_block ] Got block: #351 time: 2022-07-19T18:36:48 latency: 16 ms from: init7 irreversible: 343 (-8) peerplays06_1 | 2208015ms th_a application.cpp:429 handle_block ] Got block: #351 time: 2022-07-19T18:36:48 latency: 15 ms from: init7 irreversible: 343 (-8) peerplays05_1 | 2208016ms th_a application.cpp:429 handle_block ] Got block: #351 time: 2022-07-19T18:36:48 latency: 16 ms from: init7 irreversible: 343 (-8) peerplays14_1 | 2208017ms th_a application.cpp:429 handle_block ] Got block: #351 time: 2022-07-19T18:36:48 latency: 17 ms from: init7 irreversible: 343 (-8) peerplays08_1 | 2208502ms th_a peerplays_sidechain_plugin.cpp:458 son_processing ] Scheduled SON: 1.33.3 Now: 2022-07-19T18:36:45 peerplays10_1 | 2208515ms th_a peerplays_sidechain_plugin.cpp:458 son_processing ] Scheduled SON: 1.33.3 Now: 2022-07-19T18:36:45 peerplays01_1 | 2208515ms th_a peerplays_sidechain_plugin.cpp:458 son_processing ] Scheduled SON: 1.33.3 Now: 2022-07-19T18:36:45 peerplays15_1 | 2208515ms th_a peerplays_sidechain_plugin.cpp:458 son_processing ] Scheduled SON: 1.33.3 Now: 2022-07-19T18:36:45 peerplays03_1 | 2208515ms th_a peerplays_sidechain_plugin.cpp:458 son_processing ] Scheduled SON: 1.33.3 Now: 2022-07-19T18:36:45 peerplays13_1 | 2208515ms th_a peerplays_sidechain_plugin.cpp:458 son_processing ] Scheduled SON: 1.33.3 Now: 2022-07-19T18:36:45 peerplays16_1 | 2208516ms th_a peerplays_sidechain_plugin.cpp:458 son_processing ] Scheduled SON: 1.33.3 Now: 2022-07-19T18:36:45 peerplays06_1 | 2208517ms th_a peerplays_sidechain_plugin.cpp:458 son_processing ] Scheduled SON: 1.33.3 Now: 2022-07-19T18:36:45 peerplays11_1 | 2208518ms th_a peerplays_sidechain_plugin.cpp:458 son_processing ] Scheduled SON: 1.33.3 Now: 2022-07-19T18:36:45 peerplays12_1 | 2208518ms th_a peerplays_sidechain_plugin.cpp:458 son_processing ] Scheduled SON: 1.33.3 Now: 2022-07-19T18:36:45 peerplays07_1 | 2208518ms th_a peerplays_sidechain_plugin.cpp:458 son_processing ] Scheduled SON: 1.33.3 Now: 2022-07-19T18:36:45 peerplays14_1 | 2208519ms th_a peerplays_sidechain_plugin.cpp:458 son_processing ] Scheduled SON: 1.33.3 Now: 2022-07-19T18:36:45 peerplays05_1 | 2208519ms th_a peerplays_sidechain_plugin.cpp:458 son_processing ] Scheduled SON: 1.33.3 Now: 2022-07-19T18:36:45 peerplays09_1 | 2208524ms th_a peerplays_sidechain_plugin.cpp:458 son_processing ] Scheduled SON: 1.33.3 Now: 2022-07-19T18:36:45 peerplays02_1 | 2209000ms th_a witness.cpp:195 block_production_loo ] Not producing block because production is disabled until we receive a recent block (see: - ``` @hirunda @serkixenos witness crash needs to be seen, hence keeping the system under observation

bobinson commented

2022-07-20 06:50:30 +00:00

(Migrated from gitlab.com)

hi @prandnum , when you say the witness crash, you are expecting the outdated nodes to crash ?

prandnum commented

2022-07-20 09:46:32 +00:00

(Migrated from gitlab.com)

@bobinson no the outdated ones will unable to join the chain, but other witnesses participating in the chain are crashing (peerplays02 and peerplays04 are the outdated ones here) you can see from below other witnesses are crashing.

CONTAINER ID   IMAGE                   COMMAND                  CREATED        STATUS                            PORTS                                                                                                                                             NAMES
f8dc3b3b9659   peerplays11             "./witness_node -d .…"   15 hours ago   Restarting (139) 55 seconds ago                                                                                                                                                     peerplays-qa-environment_peerplays11_1
193d1bc6a2b7   peerplays08             "./witness_node -d .…"   15 hours ago   Up 8 hours                        22/tcp, 0.0.0.0:8097->8090/tcp, :::8097->8090/tcp, 0.0.0.0:9784->9777/tcp, :::9784->9777/tcp                                                      peerplays-qa-environment_peerplays08_1
ea9d69ff9c29   peerplays07             "./witness_node -d .…"   15 hours ago   Up 29 seconds                     22/tcp, 0.0.0.0:8096->8090/tcp, :::8096->8090/tcp, 0.0.0.0:9783->9777/tcp, :::9783->9777/tcp                                                      peerplays-qa-environment_peerplays07_1
cdb02f3c084b   peerplays16             "./witness_node -d .…"   15 hours ago   Up 15 hours                       22/tcp, 0.0.0.0:8105->8090/tcp, :::8105->8090/tcp, 0.0.0.0:9792->9777/tcp, :::9792->9777/tcp                                                      peerplays-qa-environment_peerplays16_1
f012f0e4f42a   peerplays02             "./witness_node -d .…"   15 hours ago   Up 15 hours                       22/tcp, 0.0.0.0:8091->8090/tcp, :::8091->8090/tcp, 0.0.0.0:9778->9777/tcp, :::9778->9777/tcp                                                      peerplays-qa-environment_peerplays02_1
cc4068c7186a   peerplays03             "./witness_node -d .…"   15 hours ago   Up 15 hours                       22/tcp, 0.0.0.0:8092->8090/tcp, :::8092->8090/tcp, 0.0.0.0:9779->9777/tcp, :::9779->9777/tcp                                                      peerplays-qa-environment_peerplays03_1
a9bd4f735894   peerplays06             "./witness_node -d .…"   15 hours ago   Up About an hour                  22/tcp, 0.0.0.0:8095->8090/tcp, :::8095->8090/tcp, 0.0.0.0:9782->9777/tcp, :::9782->9777/tcp                                                      peerplays-qa-environment_peerplays06_1
da5c092d9e79   peerplays15             "./witness_node -d .…"   15 hours ago   Up 15 hours                       22/tcp, 0.0.0.0:8104->8090/tcp, :::8104->8090/tcp, 0.0.0.0:9791->9777/tcp, :::9791->9777/tcp                                                      peerplays-qa-environment_peerplays15_1
0bc000b77b51   peerplays12             "./witness_node -d .…"   15 hours ago   Up 15 hours                       22/tcp, 0.0.0.0:8101->8090/tcp, :::8101->8090/tcp, 0.0.0.0:9788->9777/tcp, :::9788->9777/tcp                                                      peerplays-qa-environment_peerplays12_1
5da3329de5f7   peerplays04             "./witness_node -d .…"   15 hours ago   Up 15 hours                       22/tcp, 0.0.0.0:8093->8090/tcp, :::8093->8090/tcp, 0.0.0.0:9780->9777/tcp, :::9780->9777/tcp                                                      peerplays-qa-environment_peerplays04_1
a2e65b641486   peerplays13             "./witness_node -d .…"   15 hours ago   Up 15 hours                       22/tcp, 0.0.0.0:8102->8090/tcp, :::8102->8090/tcp, 0.0.0.0:9789->9777/tcp, :::9789->9777/tcp                                                      peerplays-qa-environment_peerplays13_1
ab12c68367db   peerplays09             "./witness_node -d .…"   15 hours ago   Up 51 minutes                     22/tcp, 0.0.0.0:8098->8090/tcp, :::8098->8090/tcp, 0.0.0.0:9785->9777/tcp, :::9785->9777/tcp                                                      peerplays-qa-environment_peerplays09_1
5caf62db6925   peerplays05             "./witness_node -d .…"   15 hours ago   Up 15 hours                       22/tcp, 0.0.0.0:8094->8090/tcp, :::8094->8090/tcp, 0.0.0.0:9781->9777/tcp, :::9781->9777/tcp                                                      peerplays-qa-environment_peerplays05_1
de10871a7bc8   peerplays01             "./witness_node -d .…"   15 hours ago   Up 3 hours                        0.0.0.0:8090->8090/tcp, :::8090->8090/tcp, 22/tcp, 0.0.0.0:9777->9777/tcp, :::9777->9777/tcp                                                      peerplays-qa-environment_peerplays01_1
418cd275a01f   peerplays14             "./witness_node -d .…"   15 hours ago   Up 15 hours                       22/tcp, 0.0.0.0:8103->8090/tcp, :::8103->8090/tcp, 0.0.0.0:9790->9777/tcp, :::9790->9777/tcp                                                      peerplays-qa-environment_peerplays14_1
18ca3f20663b   peerplays10             "./witness_node -d .…"   15 hours ago   Up 9 seconds                      22/tcp, 0.0.0.0:8099->8090/tcp, :::8099->8090/tcp, 0.0.0.0:9786->9777/tcp, :::9786->9777/tcp                                                      peerplays-qa-environment_peerplays10_1
fb750a441449   peerplays-baseabc       "bash"                   15 hours ago   Exited (0) 15 hours ago                                                                                                                                                             peerplays-qa-environment_peerplays-baseabc_1
170abb3eb9bd   peerplays-base          "bash"                   15 hours ago   Exited (0) 15 hours ago                                                                                                                                                             peerplays-qa-environment_peerplays-base_1
bcfe6d82a49a   ubuntu-for-peerplays    "bash"                   15 hours ago   Exited (0) 15 hours ago                                                                                                                                                             peerplays-qa-environment_ubuntu-for-peerplays_1
b79a553dee03   hive-for-peerplays      "./hived -d ./hived_…"   39 hours ago   Up 39 hours                       22/tcp, 0.0.0.0:28090-28091->28090-28091/tcp, :::28090-28091->28090-28091/tcp                                                                     peerplays-qa-environment_hive-for-peerplays_1
0a0ec90f8dc9   bitcoin-for-peerplays   "./bitcoind -datadir…"   39 hours ago   Up 39 hours                       0.0.0.0:8332->8332/tcp, :::8332->8332/tcp, 0.0.0.0:11111->11111/tcp, :::11111->11111/tcp, 22/tcp, 0.0.0.0:18444->18444/tcp, :::18444->18444/tcp   peerplays-qa-environment_bitcoin-for-peerplays_1
kiran@warpspeed:~/PBSA/19072022/peerplays-utils/peerplays-qa-environment$

@bobinson no the outdated ones will unable to join the chain, but other witnesses participating in the chain are crashing (peerplays02 and peerplays04 are the outdated ones here) you can see from below other witnesses are crashing. ``` CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES f8dc3b3b9659 peerplays11 "./witness_node -d .…" 15 hours ago Restarting (139) 55 seconds ago peerplays-qa-environment_peerplays11_1 193d1bc6a2b7 peerplays08 "./witness_node -d .…" 15 hours ago Up 8 hours 22/tcp, 0.0.0.0:8097->8090/tcp, :::8097->8090/tcp, 0.0.0.0:9784->9777/tcp, :::9784->9777/tcp peerplays-qa-environment_peerplays08_1 ea9d69ff9c29 peerplays07 "./witness_node -d .…" 15 hours ago Up 29 seconds 22/tcp, 0.0.0.0:8096->8090/tcp, :::8096->8090/tcp, 0.0.0.0:9783->9777/tcp, :::9783->9777/tcp peerplays-qa-environment_peerplays07_1 cdb02f3c084b peerplays16 "./witness_node -d .…" 15 hours ago Up 15 hours 22/tcp, 0.0.0.0:8105->8090/tcp, :::8105->8090/tcp, 0.0.0.0:9792->9777/tcp, :::9792->9777/tcp peerplays-qa-environment_peerplays16_1 f012f0e4f42a peerplays02 "./witness_node -d .…" 15 hours ago Up 15 hours 22/tcp, 0.0.0.0:8091->8090/tcp, :::8091->8090/tcp, 0.0.0.0:9778->9777/tcp, :::9778->9777/tcp peerplays-qa-environment_peerplays02_1 cc4068c7186a peerplays03 "./witness_node -d .…" 15 hours ago Up 15 hours 22/tcp, 0.0.0.0:8092->8090/tcp, :::8092->8090/tcp, 0.0.0.0:9779->9777/tcp, :::9779->9777/tcp peerplays-qa-environment_peerplays03_1 a9bd4f735894 peerplays06 "./witness_node -d .…" 15 hours ago Up About an hour 22/tcp, 0.0.0.0:8095->8090/tcp, :::8095->8090/tcp, 0.0.0.0:9782->9777/tcp, :::9782->9777/tcp peerplays-qa-environment_peerplays06_1 da5c092d9e79 peerplays15 "./witness_node -d .…" 15 hours ago Up 15 hours 22/tcp, 0.0.0.0:8104->8090/tcp, :::8104->8090/tcp, 0.0.0.0:9791->9777/tcp, :::9791->9777/tcp peerplays-qa-environment_peerplays15_1 0bc000b77b51 peerplays12 "./witness_node -d .…" 15 hours ago Up 15 hours 22/tcp, 0.0.0.0:8101->8090/tcp, :::8101->8090/tcp, 0.0.0.0:9788->9777/tcp, :::9788->9777/tcp peerplays-qa-environment_peerplays12_1 5da3329de5f7 peerplays04 "./witness_node -d .…" 15 hours ago Up 15 hours 22/tcp, 0.0.0.0:8093->8090/tcp, :::8093->8090/tcp, 0.0.0.0:9780->9777/tcp, :::9780->9777/tcp peerplays-qa-environment_peerplays04_1 a2e65b641486 peerplays13 "./witness_node -d .…" 15 hours ago Up 15 hours 22/tcp, 0.0.0.0:8102->8090/tcp, :::8102->8090/tcp, 0.0.0.0:9789->9777/tcp, :::9789->9777/tcp peerplays-qa-environment_peerplays13_1 ab12c68367db peerplays09 "./witness_node -d .…" 15 hours ago Up 51 minutes 22/tcp, 0.0.0.0:8098->8090/tcp, :::8098->8090/tcp, 0.0.0.0:9785->9777/tcp, :::9785->9777/tcp peerplays-qa-environment_peerplays09_1 5caf62db6925 peerplays05 "./witness_node -d .…" 15 hours ago Up 15 hours 22/tcp, 0.0.0.0:8094->8090/tcp, :::8094->8090/tcp, 0.0.0.0:9781->9777/tcp, :::9781->9777/tcp peerplays-qa-environment_peerplays05_1 de10871a7bc8 peerplays01 "./witness_node -d .…" 15 hours ago Up 3 hours 0.0.0.0:8090->8090/tcp, :::8090->8090/tcp, 22/tcp, 0.0.0.0:9777->9777/tcp, :::9777->9777/tcp peerplays-qa-environment_peerplays01_1 418cd275a01f peerplays14 "./witness_node -d .…" 15 hours ago Up 15 hours 22/tcp, 0.0.0.0:8103->8090/tcp, :::8103->8090/tcp, 0.0.0.0:9790->9777/tcp, :::9790->9777/tcp peerplays-qa-environment_peerplays14_1 18ca3f20663b peerplays10 "./witness_node -d .…" 15 hours ago Up 9 seconds 22/tcp, 0.0.0.0:8099->8090/tcp, :::8099->8090/tcp, 0.0.0.0:9786->9777/tcp, :::9786->9777/tcp peerplays-qa-environment_peerplays10_1 fb750a441449 peerplays-baseabc "bash" 15 hours ago Exited (0) 15 hours ago peerplays-qa-environment_peerplays-baseabc_1 170abb3eb9bd peerplays-base "bash" 15 hours ago Exited (0) 15 hours ago peerplays-qa-environment_peerplays-base_1 bcfe6d82a49a ubuntu-for-peerplays "bash" 15 hours ago Exited (0) 15 hours ago peerplays-qa-environment_ubuntu-for-peerplays_1 b79a553dee03 hive-for-peerplays "./hived -d ./hived_…" 39 hours ago Up 39 hours 22/tcp, 0.0.0.0:28090-28091->28090-28091/tcp, :::28090-28091->28090-28091/tcp peerplays-qa-environment_hive-for-peerplays_1 0a0ec90f8dc9 bitcoin-for-peerplays "./bitcoind -datadir…" 39 hours ago Up 39 hours 0.0.0.0:8332->8332/tcp, :::8332->8332/tcp, 0.0.0.0:11111->11111/tcp, :::11111->11111/tcp, 22/tcp, 0.0.0.0:18444->18444/tcp, :::18444->18444/tcp peerplays-qa-environment_bitcoin-for-peerplays_1 kiran@warpspeed:~/PBSA/19072022/peerplays-utils/peerplays-qa-environment$ ```

prandnum commented

2022-07-20 18:53:03 +00:00

(Migrated from gitlab.com)

witness logs are available at
https://gitlab.com/PBSA/documentation/working-documents/-/tree/develop/88/20072022-witness-crash

witness logs are available at https://gitlab.com/PBSA/documentation/working-documents/-/tree/develop/88/20072022-witness-crash

prandnum commented

2022-07-25 15:32:06 +00:00

(Migrated from gitlab.com)

Following scenarios were tested:

All 20.04 witness + init script, with two witnesses running old version(running 20.04): Witnesses crashed (older version witnesses didnt crash)
All 18.04 witness + init script: No Witness crash
20.04 and 18.04 witness + init script: No Witness crash
20.04 and 18.04 witness + init script, with two witnesses running old version(running 18.04): Witnesses crashed (older version witnesses didnt crash)

@hirunda @serkixenos

Following scenarios were tested: - All 20.04 witness + init script, with two witnesses running old version(running 20.04): Witnesses crashed (older version witnesses didnt crash) - All 18.04 witness + init script: No Witness crash - 20.04 and 18.04 witness + init script: No Witness crash - 20.04 and 18.04 witness + init script, with two witnesses running old version(running 18.04): Witnesses crashed (older version witnesses didnt crash) @hirunda @serkixenos

hirunda commented

2022-07-26 18:01:50 +00:00

(Migrated from gitlab.com)

@prandnum Let me try on my side to see if related to changes I made ...

prandnum commented

2022-08-01 16:30:50 +00:00

(Migrated from gitlab.com)

mentioned in issue #409

prandnum (Migrated from gitlab.com) closed this issue

2022-08-25 18:32:10 +00:00

serkixenos commented

2022-08-28 13:55:37 +00:00