stopping witness_node with control-C corrupts chain file

bobinson commented

2022-01-01 14:51:26 +00:00

(Migrated from gitlab.com)

Stopped the witness_node with Control-C

2846957ms asio       main.cpp:168                  operator()           ] Caught SIGINT attempting to exit cleanly
2846957ms th_a       main.cpp:181                  main                 ] Exiting from signal 2
2846979ms th_a       undo_database.cpp:294         pop_commit           ] error popping commit 10 assert_exception: Assert Exception
maybe_found != nullptr: Unable to find Object
    {"id":"1.11.9415482"}
    th_a  index.hpp:113 get
2846979ms th_a       db_management.cpp:272         close                ] Database close unexpected exception: {"code":10,"name":"assert_exception","message":"Assert Exception","stack":[{"context":{"level":"error","file":"index.hpp","line":113,"method":"get","hostname":"","thread_name":"th_a","timestamp":"2022-01-01T14:47:26"},"format":"maybe_found != nullptr: Unable to find Object","data":{"id":"1.11.9415482"}},{"context":{"level":"warn","file":"object_database.cpp","line":119,"method":"pop_undo","hostname":"","thread_name":"th_a","timestamp":"2022-01-01T14:47:26"},"format":"","data":{}},{"context":{"level":"warn","file":"db_block.cpp","line":600,"method":"pop_block","hostname":"","thread_name":"th_a","timestamp":"2022-01-01T14:47:26"},"format":"","data":{}}]}

Tried to resume with replay

2929730ms th_a       accounts_list_plugin.cpp:128  list_accounts        ] accounts list plugin:  list_accounts()
2929730ms th_a       bookie_plugin.cpp:468         plugin_initialize    ] bookie plugin: plugin_startup() begin
2929730ms th_a       db_management.cpp:294         force_slow_replays   ] enabling slow replays
2929730ms th_a       bookie_plugin.cpp:501         plugin_initialize    ] bookie plugin: plugin_startup() end
2929730ms th_a       witness.cpp:91                plugin_initialize    ] witness plugin:  plugin_initialize() begin
2929731ms th_a       witness.cpp:105               plugin_initialize    ] Public Key: PPY6rP7UNDEyD99NAzWTm9xTTUDu4wpkPiAugpcHvaogDNbtzeG3H
2929731ms th_a       witness.cpp:123               plugin_initialize    ] witness plugin:  plugin_initialize() end
2929731ms th_a       object_database.cpp:106       open                 ] Opening object database from /root/src/peerplays/witness_node_data_dir/blockchain ...
2929885ms th_a       db_management.cpp:244         open                 ] 10 assert_exception: Assert Exception
insert_result.second: Could not insert object, most likely a uniqueness constraint was violated
    {}
    th_a  generic_index.hpp:53 insert

    {"data_dir":"/root/src/peerplays/witness_node_data_dir/blockchain"}
    th_a  object_database.cpp:113 open
2929885ms th_a       db_management.cpp:244         open                 ] data_dir: /root/src/peerplays/witness_node_data_dir/blockchain
2929885ms th_a       application.cpp:371           startup              ] Caught exception 10 assert_exception: Assert Exception
insert_result.second: Could not insert object, most likely a uniqueness constraint was violated
    {}
    th_a  generic_index.hpp:53 insert

    {"data_dir":"/root/src/peerplays/witness_node_data_dir/blockchain"}
    th_a  object_database.cpp:113 open
rethrow
    {"data_dir":"/root/src/peerplays/witness_node_data_dir/blockchain"}
    th_a  db_management.cpp:244 open in open(), you might want to force a replay
2929885ms th_a       application.cpp:411           startup              ] 10 assert_exception: Assert Exception
insert_result.second: Could not insert object, most likely a uniqueness constraint was violated
    {}
    th_a  generic_index.hpp:53 insert

    {"data_dir":"/root/src/peerplays/witness_node_data_dir/blockchain"}
    th_a  object_database.cpp:113 open
rethrow
    {"data_dir":"/root/src/peerplays/witness_node_data_dir/blockchain"}
    th_a  db_management.cpp:244 open
2929885ms th_a       application.cpp:954           startup              ] 10 assert_exception: Assert Exception
insert_result.second: Could not insert object, most likely a uniqueness constraint was violated
    {}
    th_a  generic_index.hpp:53 insert

    {"data_dir":"/root/src/peerplays/witness_node_data_dir/blockchain"}
    th_a  object_database.cpp:113 open
rethrow
    {"data_dir":"/root/src/peerplays/witness_node_data_dir/blockchain"}
    th_a  db_management.cpp:244 open
rethrow
    {}
    th_a  application.cpp:411 startup


    th_a  application.cpp:411 startup
2929885ms th_a       main.cpp:193                  main                 ] Exiting with error:
10 assert_exception: Assert Exception
insert_result.second: Could not insert object, most likely a uniqueness constraint was violated
    {}
    th_a  generic_index.hpp:53 insert

    {"data_dir":"/root/src/peerplays/witness_node_data_dir/blockchain"}
    th_a  object_database.cpp:113 open
rethrow
    {"data_dir":"/root/src/peerplays/witness_node_data_dir/blockchain"}
    th_a  db_management.cpp:244 open
rethrow
    {}
    th_a  application.cpp:411 startup

Stopped the `witness_node` with Control-C ``` 2846957ms asio main.cpp:168 operator() ] Caught SIGINT attempting to exit cleanly 2846957ms th_a main.cpp:181 main ] Exiting from signal 2 2846979ms th_a undo_database.cpp:294 pop_commit ] error popping commit 10 assert_exception: Assert Exception maybe_found != nullptr: Unable to find Object {"id":"1.11.9415482"} th_a index.hpp:113 get 2846979ms th_a db_management.cpp:272 close ] Database close unexpected exception: {"code":10,"name":"assert_exception","message":"Assert Exception","stack":[{"context":{"level":"error","file":"index.hpp","line":113,"method":"get","hostname":"","thread_name":"th_a","timestamp":"2022-01-01T14:47:26"},"format":"maybe_found != nullptr: Unable to find Object","data":{"id":"1.11.9415482"}},{"context":{"level":"warn","file":"object_database.cpp","line":119,"method":"pop_undo","hostname":"","thread_name":"th_a","timestamp":"2022-01-01T14:47:26"},"format":"","data":{}},{"context":{"level":"warn","file":"db_block.cpp","line":600,"method":"pop_block","hostname":"","thread_name":"th_a","timestamp":"2022-01-01T14:47:26"},"format":"","data":{}}]} ``` Tried to resume with replay ``` 2929730ms th_a accounts_list_plugin.cpp:128 list_accounts ] accounts list plugin: list_accounts() 2929730ms th_a bookie_plugin.cpp:468 plugin_initialize ] bookie plugin: plugin_startup() begin 2929730ms th_a db_management.cpp:294 force_slow_replays ] enabling slow replays 2929730ms th_a bookie_plugin.cpp:501 plugin_initialize ] bookie plugin: plugin_startup() end 2929730ms th_a witness.cpp:91 plugin_initialize ] witness plugin: plugin_initialize() begin 2929731ms th_a witness.cpp:105 plugin_initialize ] Public Key: PPY6rP7UNDEyD99NAzWTm9xTTUDu4wpkPiAugpcHvaogDNbtzeG3H 2929731ms th_a witness.cpp:123 plugin_initialize ] witness plugin: plugin_initialize() end 2929731ms th_a object_database.cpp:106 open ] Opening object database from /root/src/peerplays/witness_node_data_dir/blockchain ... 2929885ms th_a db_management.cpp:244 open ] 10 assert_exception: Assert Exception insert_result.second: Could not insert object, most likely a uniqueness constraint was violated {} th_a generic_index.hpp:53 insert {"data_dir":"/root/src/peerplays/witness_node_data_dir/blockchain"} th_a object_database.cpp:113 open 2929885ms th_a db_management.cpp:244 open ] data_dir: /root/src/peerplays/witness_node_data_dir/blockchain 2929885ms th_a application.cpp:371 startup ] Caught exception 10 assert_exception: Assert Exception insert_result.second: Could not insert object, most likely a uniqueness constraint was violated {} th_a generic_index.hpp:53 insert {"data_dir":"/root/src/peerplays/witness_node_data_dir/blockchain"} th_a object_database.cpp:113 open rethrow {"data_dir":"/root/src/peerplays/witness_node_data_dir/blockchain"} th_a db_management.cpp:244 open in open(), you might want to force a replay 2929885ms th_a application.cpp:411 startup ] 10 assert_exception: Assert Exception insert_result.second: Could not insert object, most likely a uniqueness constraint was violated {} th_a generic_index.hpp:53 insert {"data_dir":"/root/src/peerplays/witness_node_data_dir/blockchain"} th_a object_database.cpp:113 open rethrow {"data_dir":"/root/src/peerplays/witness_node_data_dir/blockchain"} th_a db_management.cpp:244 open 2929885ms th_a application.cpp:954 startup ] 10 assert_exception: Assert Exception insert_result.second: Could not insert object, most likely a uniqueness constraint was violated {} th_a generic_index.hpp:53 insert {"data_dir":"/root/src/peerplays/witness_node_data_dir/blockchain"} th_a object_database.cpp:113 open rethrow {"data_dir":"/root/src/peerplays/witness_node_data_dir/blockchain"} th_a db_management.cpp:244 open rethrow {} th_a application.cpp:411 startup th_a application.cpp:411 startup 2929885ms th_a main.cpp:193 main ] Exiting with error: 10 assert_exception: Assert Exception insert_result.second: Could not insert object, most likely a uniqueness constraint was violated {} th_a generic_index.hpp:53 insert {"data_dir":"/root/src/peerplays/witness_node_data_dir/blockchain"} th_a object_database.cpp:113 open rethrow {"data_dir":"/root/src/peerplays/witness_node_data_dir/blockchain"} th_a db_management.cpp:244 open rethrow {} th_a application.cpp:411 startup ```

bobinson commented

2022-01-04 15:37:57 +00:00

(Migrated from gitlab.com)

assigned to @yviter

yviter commented

2022-01-18 11:11:37 +00:00

(Migrated from gitlab.com)

Created two types of database: Happy and Faily.
Happy never fails and Faily fails always.
Next step: to reproduce error condition syncing with mainnet

Created two types of database: Happy and Faily. Happy never fails and Faily fails always. Next step: to reproduce error condition syncing with mainnet

yviter commented

2022-01-19 13:20:44 +00:00

(Migrated from gitlab.com)

There are two basic strategies to investigate:
-second time block replay (surgeon's idea)
-dirty write (mine idea)
Debugging witness_node step-by-step

There are two basic strategies to investigate: -second time block replay (surgeon's idea) -dirty write (mine idea) Debugging witness_node step-by-step

yviter commented

2022-01-20 13:28:41 +00:00

(Migrated from gitlab.com)

trying to create a new one instance of "faulty" database I created it with error: Unable to find Object {"id":"1.6.17"}

yviter commented

2022-01-26 13:10:24 +00:00

(Migrated from gitlab.com)

Learning FC thread library;
Preparing report/diagram of witness_node threading scheme

Learning FC thread library; Preparing report/diagram of witness_node threading scheme

yviter commented

2022-01-28 13:12:41 +00:00

(Migrated from gitlab.com)

There are combination of two approaches: cooperative multitasking + preemptive multitasking.
Main thread "witness_node" do all initialization / finalization stuff. It also starts "p2p" thread which starts preemptive part, the thread pool of 8 threads. The purpose of preemptive threads pool is to handle ASIO stuff

There are combination of two approaches: cooperative multitasking + preemptive multitasking. Main thread "witness_node" do all initialization / finalization stuff. It also starts "p2p" thread which starts preemptive part, the thread pool of 8 threads. The purpose of preemptive threads pool is to handle ASIO stuff

serkixenos commented

2022-01-31 13:58:44 +00:00

(Migrated from gitlab.com)

unassigned @yviter

serkixenos commented

2022-02-10 21:47:17 +00:00

(Migrated from gitlab.com)

assigned to @vampik

vampik commented

2022-02-15 13:44:11 +00:00

(Migrated from gitlab.com)

mentioned in commit 065c9d1546

mentioned in commit 065c9d1546cdb7860a5cf6e6cb4527ba7b98ff51

vampik commented

2022-02-16 10:04:13 +00:00

(Migrated from gitlab.com)

mentioned in commit 7ae3138863

mentioned in commit 7ae3138863480eba443870bc94da65bab85b36c4

vampik commented

2022-02-16 10:04:13 +00:00

(Migrated from gitlab.com)

mentioned in commit 42504ab65b

mentioned in commit 42504ab65b57d5a2a689e2f2c5bb72a9a695a02e

vampik commented

2022-02-16 10:04:13 +00:00

(Migrated from gitlab.com)

mentioned in commit 1f6e6dc65a

mentioned in commit 1f6e6dc65a482f7cf7bdefd5f2436cd63955f169

vampik commented

2022-02-16 10:04:13 +00:00

(Migrated from gitlab.com)

mentioned in commit 3f205a5722

mentioned in commit 3f205a572297fd8c33778d6147fb3fd4b927f524

vampik commented

2022-02-16 10:04:13 +00:00

(Migrated from gitlab.com)

mentioned in commit 4267e6323a

mentioned in commit 4267e6323a3c1ca66c7e36059d061b3094117718

vampik commented

2022-02-16 11:48:49 +00:00

(Migrated from gitlab.com)

mentioned in commit cca322573b

mentioned in commit cca322573b229493622686ea951e3343aec212a7

vampik commented

2022-02-21 06:11:08 +00:00

(Migrated from gitlab.com)

mentioned in commit ac1be63ddd

mentioned in commit ac1be63dddb7c7b4302392affc4b13307a5e1b03

vampik commented

2022-02-22 10:07:59 +00:00

(Migrated from gitlab.com)

mentioned in commit 2cb4705374

mentioned in commit 2cb470537446ddd0096e87d566b15db22b9c62f6

vampik commented

2022-02-23 09:54:21 +00:00

(Migrated from gitlab.com)

mentioned in commit a685a63453

mentioned in commit a685a63453abc2312dafa1758c0ddb8779f2b9a2

vampik commented

2022-02-23 09:54:22 +00:00

(Migrated from gitlab.com)

mentioned in commit 6f9c6fa8f0

mentioned in commit 6f9c6fa8f089e699de8305793c9ad560e83ca143

vampik commented

2022-02-24 06:44:35 +00:00

(Migrated from gitlab.com)

mentioned in commit 224eedee39

mentioned in commit 224eedee394f2c7300c4cc405a3b8ac943bbf87e

vampik commented

2022-02-24 06:44:35 +00:00

(Migrated from gitlab.com)

mentioned in commit a6d26572a1

mentioned in commit a6d26572a1a0983e3b5a73c753399ea689838724

vampik commented

2022-02-24 06:44:35 +00:00

(Migrated from gitlab.com)

mentioned in commit 1d32b18102

mentioned in commit 1d32b18102384fb731e3acd358471c94497c6cc4

vampik commented

2022-02-24 06:44:35 +00:00

(Migrated from gitlab.com)

mentioned in commit 82aedaf937

mentioned in commit 82aedaf93767df8b3e2149f8c02565c11c2242a8

vampik commented

2022-02-24 06:44:36 +00:00

(Migrated from gitlab.com)

mentioned in commit 37c9b057df

mentioned in commit 37c9b057df6d942bbca90dd42faa5740d25c5277

serkixenos commented

2022-02-26 22:25:02 +00:00

(Migrated from gitlab.com)

mentioned in merge request !75

serkixenos commented

2022-02-28 22:56:21 +00:00

(Migrated from gitlab.com)

mentioned in commit 8dc8ac0aec

mentioned in commit 8dc8ac0aecc25fd53daeb26446618fccf34d144e

serkixenos commented

2022-02-28 22:56:21 +00:00

(Migrated from gitlab.com)

mentioned in commit 49c39afbe1

mentioned in commit 49c39afbe16ac0bd75e4cdd038015d578b0c3e81

serkixenos commented

2022-02-28 22:57:17 +00:00

(Migrated from gitlab.com)

assigned to @prandnum

serkixenos commented

2022-02-28 23:03:27 +00:00

(Migrated from gitlab.com)

@prandnum

Start syncing with the mainnet.
Randomly stop and restart software at different points during synchronization
Verify that the software is not crashing on restart
Verify that the software continues syncing, and does not require --replay-blockchain after restart
Verify that the software syncs properly with the mainnet.

@prandnum 1. Start syncing with the mainnet. 2. Randomly stop and restart software at different points during synchronization 3. Verify that the software is not crashing on restart 4. Verify that the software continues syncing, and does not require --replay-blockchain after restart 4. Verify that the software syncs properly with the mainnet.

serkixenos commented

2022-02-28 23:59:13 +00:00

(Migrated from gitlab.com)

Note: This is not an error, its waiting for OS to release configured port.

Unable to listen for connections on port 9777, retrying in a few seconds
You can wait for it to become available, or restart this program using
the --p2p-port option to specify another port

Still waiting for port 9777 to become available

Still waiting for port 9777 to become available

Note: This is not an error, its waiting for OS to release configured port. ``` Unable to listen for connections on port 9777, retrying in a few seconds You can wait for it to become available, or restart this program using the --p2p-port option to specify another port Still waiting for port 9777 to become available Still waiting for port 9777 to become available ```

prandnum commented

2022-03-09 18:39:59 +00:00

(Migrated from gitlab.com)

Hi @vampik @serkixenos changing the state back to pending as the logs are given(via RocketChat) for the issue seen.

prandnum commented

2022-03-10 18:56:25 +00:00

(Migrated from gitlab.com)

The issue is not seen anymore. When the process is killed with ctrl+c, kill -9(both process and service) the witness node process is able to restart without any issue(provided the chain is fully synced atleast once).and kill #316 is raised to track chain halt issue when the witness process is killed multiple times during initial sync.

prandnum (Migrated from gitlab.com) closed this issue

2022-03-10 18:56:26 +00:00

bobinson commented

2022-03-11 06:35:08 +00:00

(Migrated from gitlab.com)

@vampik @prandnum and @serkixenos - appreciate the work done on this!

stopping witness_node with control-C corrupts chain file #235