MAINNET halted #246

New issue

Closed

opened 2022-01-14 08:44:46 +00:00 by prandnum · 7 comments

prandnum commented

2022-01-14 08:44:46 +00:00

(Migrated from gitlab.com)

Two of the MAINNET nodes run by me are halted
on AWS VM

1133007ms th_a       application.cpp:490           handle_block         ] Error when pushing block:
10 assert_exception: Assert Exception
second_branch_itr != _index.get<block_id>().end():
    {}
    th_a  fork_database.cpp:209 fetch_branch_from

    {"first":"02c596d9711b2be3b3f9e6412f5f080a4bd1240f","second":"02c59673b2c21677e29c919adef13a3f1a6f1585"}
    th_a  fork_database.cpp:240 fetch_branch_from

    {"new_block":{"previous":"02c596abc73c2cee26d6439a72d1605631133195","timestamp":"2022-01-13T21:57:48","witness":"1.6.86","next_secret_hash":"7d785d72d9699591048329e5a1df7636f7301187","previous_secret":"1bdf77122310a5ca97e92fd7d47f179e4d411c6a","transaction_merkle_root":"0000000000000000000000000000000000000000","extensions":[],"witness_signature":"2056e8a0820d3472bbb274192e5470e2be6dbd24c96a2d472f53fe53bb737ef7d1758d14a1501bc0e726c9b7f63eeb43843106c92c83ec0e4ae3288adfa184a80b","transactions":[]}}
    th_a  db_block.cpp:293 _push_block
1506123ms th_a       application.cpp:487           handle_block         ] Error when pushing block:
3080000 unlinkable_block_exception: unlinkable block
block does not link to known chain
    {}
    th_a  db_block.cpp:206 _push_block

    {"new_block":{"previous":"02c59673b2c21677e29c919adef13a3f1a6f1585","timestamp":"2022-01-13T21:53:54","witness":"1.6.81","next_secret_hash":"2411084de2054de00b6d09722a0791722180ae44","previous_secret":"119be45a655156d5adb46a75b66cf9003c08fce5","transaction_merkle_root":"0000000000000000000000000000000000000000","extensions":[],"witness_signature":"20758b2270309e5f467d61e6ee07c2b9b774a3fdf572985ae8a473eb412898d20814f502323e2354639eaed55dac6b46e4a22877bd3be7841beaac2d685a939d8d","transactions":[]}}
    th_a  db_block.cpp:293 _push_block
1506124ms th_a       application.cpp:487           handle_block         ] Error when pushing block:
3080000 unlinkable_block_exception: unlinkable block
block does not link to known chain
    {}
    th_a  db_block.cpp:206 _push_block

    {"new_block":{"previous":"02c596742ae33ed83e15c921f4b0b7c57ba8e998","timestamp":"2022-01-13T21:54:00","witness":"1.6.93","next_secret_hash":"17a9e97df9954c97ca03aeff81e747cb643dc215","previous_secret":"29fc4c1049d092d93d284c1c4a5bfb048abc8837","transaction_merkle_root":"0000000000000000000000000000000000000000","extensions":[],"witness_signature":"2033e113f318a818d4e2b99da892f38b219180c32ffe91a3fd3f3141f9adc4c9557453628c54f88c67a7771d90c7033b2eee716f2017f3bc395fab81100d67409d","transactions":[]}}
    th_a  db_block.cpp:293 _push_block
1506124ms th_a       application.cpp:487           handle_block         ] Error when pushing block:
3080000 unlinkable_block_exception: unlinkable block
block does not link to known chain
    {}
    th_a  db_block.cpp:206 _push_block

    {"new_block":{"previous":"02c59675b8db2e9c8777280dbe33e330bd8c709d","timestamp":"2022-01-13T21:54:06","witness":"1.6.87","next_secret_hash":"202b390b581e052a6e9eb0b182452001753bf658","previous_secret":"fe543ef20c0a92fa0b0b2cabc03ab2b8284c1c1d","transaction_merkle_root":"0000000000000000000000000000000000000000","extensions":[],"witness_signature":"1f286f186ad06eb95b5c70db133e2e040942edbfe517df0d848cc332f0564061f057ae9b09fd3654ffdd41ee453b5088b0aeafb673e574a60e2c6c8ef2a3c9cc53","transactions":[]}}
    th_a  db_block.cpp:293 _push_block
874851ms th_a       application.cpp:487           handle_block         ] Error when pushing block:
3080000 unlinkable_block_exception: unlinkable block
block does not link to known chain
    {}
    th_a  db_block.cpp:206 _push_block

    {"new_block":{"previous":"02c59681b5153fc4306f05c1c5edf8f8352775a9","timestamp":"2022-01-13T21:53:48","witness":"1.6.87","next_secret_hash":"934f26a50c4e92e1e7f8a2031e908531da9f22a4","previous_secret":"2ccc6006701df743339400fe5b5386ad00417201","transaction_merkle_root":"0000000000000000000000000000000000000000","extensions":[],"witness_signature":"20488e4aaeae0c9cb28fa9aab33e66ee0c7bd9f311a0194f8bf035a9bdba2506d245a68b7c8c50571ef6e3570f15297b2ea6800f016d6741b2118e00caf1fa7ee9","transactions":[]}}
    th_a  db_block.cpp:293 _push_block

on Hetzner VM

3080000 unlinkable_block_exception: unlinkable block
block does not link to known chain
    {}
    th_a  db_block.cpp:206 _push_block

    {"new_block":{"previous":"02c596b09618f0f7a0dfc91ff5cc81735c27e589","timestamp":"2022-01-13T21:58:24","witness":"1.6.92","next_secret_hash":"ad3e428005a6d7dd97008fe1668cadc1fbea80fe","previous_secret":"6ffee3b9f7030421df659f94c53af204fdfd200d","transaction_merkle_root":"0000000000000000000000000000000000000000","extensions":[],"witness_signature":"1f7b8baff1db0d30490f5e7fcda10fb48612946342230356d24b7defc98c269f3d3420545c48a6e4fdc8c2dd89edcfa784a679ef978b21e91ba9a7080040356a48","transactions":[]}}
    th_a  db_block.cpp:293 _push_block
2509715ms th_a       application.cpp:487           handle_block         ] Error when pushing block:
3080000 unlinkable_block_exception: unlinkable block
block does not link to known chain
    {}
    th_a  db_block.cpp:206 _push_block

    {"new_block":{"previous":"02c596b1c4f77130977ec28819d2af836fef3027","timestamp":"2022-01-13T21:58:27","witness":"1.6.1","next_secret_hash":"95d16669a5902b77a7354168880e8bb6ce05a75e","previous_secret":"870b7512f1bfc17686dda0f7e3fe372eead9e527","transaction_merkle_root":"0000000000000000000000000000000000000000","extensions":[],"witness_signature":"205bae1b3ef40292f2ac3d23c14f637a5a00b71b0ef3547e28c0812c7223835c361af575bbd99de90a904f05d2254dcf8b1b3c62d0e1c27ae6f688b1a33975596a","transactions":[]}}
    th_a  db_block.cpp:293 _push_block
2509715ms th_a       application.cpp:487           handle_block         ] Error when pushing block:
3080000 unlinkable_block_exception: unlinkable block
block does not link to known chain
    {}
    th_a  db_block.cpp:206 _push_block

    {"new_block":{"previous":"02c596b29d699647fdd33e2027603a9a142938b5","timestamp":"2022-01-13T21:58:30","witness":"1.6.86","next_secret_hash":"e532350ae07c89de64de6721fb8a9916178f3cd8","previous_secret":"35f38da997d2a068c4289f656372a8c352d0bef2","transaction_merkle_root":"0000000000000000000000000000000000000000","extensions":[],"witness_signature":"2042f28f231e5dcd9164a09741d6374d7f598e253a8f7b94644fbfbb5c649d6475476e4a1311cfc0e638ea8bbf288d259490ccb59288f4eff171d0fc72134007fc","transactions":[]}}
    th_a  db_block.cpp:293 _push_block
2509716ms th_a       application.cpp:487           handle_block         ] Error when pushing block:
3080000 unlinkable_block_exception: unlinkable block
block does not link to known chain
    {}
    th_a  db_block.cpp:206 _push_block

    {"new_block":{"previous":"02c596b3233f1fd1335a1c36ff73ce6cafe60d17","timestamp":"2022-01-13T21:58:39","witness":"1.6.89","next_secret_hash":"1c66f7b20526b8a9baf7b970837dc5bf071d5ffb","previous_secret":"c7948d0205b2e31e3e18508fc2c6b8084cd10500","transaction_merkle_root":"0000000000000000000000000000000000000000","extensions":[],"witness_signature":"1f1248f7a3127cafc60e963022d913097d94a9a9c88300f4bf34847ec62aae0d7020f26fcd03bce41ed720b5e80c4499e2d470f841f7f5d38cfc64f81cf3d684ad","transactions":[]}}
    th_a  db_block.cpp:293 _push_block

Two of the MAINNET nodes run by me are halted on AWS VM ``` 1133007ms th_a application.cpp:490 handle_block ] Error when pushing block: 10 assert_exception: Assert Exception second_branch_itr != _index.get<block_id>().end(): {} th_a fork_database.cpp:209 fetch_branch_from {"first":"02c596d9711b2be3b3f9e6412f5f080a4bd1240f","second":"02c59673b2c21677e29c919adef13a3f1a6f1585"} th_a fork_database.cpp:240 fetch_branch_from {"new_block":{"previous":"02c596abc73c2cee26d6439a72d1605631133195","timestamp":"2022-01-13T21:57:48","witness":"1.6.86","next_secret_hash":"7d785d72d9699591048329e5a1df7636f7301187","previous_secret":"1bdf77122310a5ca97e92fd7d47f179e4d411c6a","transaction_merkle_root":"0000000000000000000000000000000000000000","extensions":[],"witness_signature":"2056e8a0820d3472bbb274192e5470e2be6dbd24c96a2d472f53fe53bb737ef7d1758d14a1501bc0e726c9b7f63eeb43843106c92c83ec0e4ae3288adfa184a80b","transactions":[]}} th_a db_block.cpp:293 _push_block 1506123ms th_a application.cpp:487 handle_block ] Error when pushing block: 3080000 unlinkable_block_exception: unlinkable block block does not link to known chain {} th_a db_block.cpp:206 _push_block {"new_block":{"previous":"02c59673b2c21677e29c919adef13a3f1a6f1585","timestamp":"2022-01-13T21:53:54","witness":"1.6.81","next_secret_hash":"2411084de2054de00b6d09722a0791722180ae44","previous_secret":"119be45a655156d5adb46a75b66cf9003c08fce5","transaction_merkle_root":"0000000000000000000000000000000000000000","extensions":[],"witness_signature":"20758b2270309e5f467d61e6ee07c2b9b774a3fdf572985ae8a473eb412898d20814f502323e2354639eaed55dac6b46e4a22877bd3be7841beaac2d685a939d8d","transactions":[]}} th_a db_block.cpp:293 _push_block 1506124ms th_a application.cpp:487 handle_block ] Error when pushing block: 3080000 unlinkable_block_exception: unlinkable block block does not link to known chain {} th_a db_block.cpp:206 _push_block {"new_block":{"previous":"02c596742ae33ed83e15c921f4b0b7c57ba8e998","timestamp":"2022-01-13T21:54:00","witness":"1.6.93","next_secret_hash":"17a9e97df9954c97ca03aeff81e747cb643dc215","previous_secret":"29fc4c1049d092d93d284c1c4a5bfb048abc8837","transaction_merkle_root":"0000000000000000000000000000000000000000","extensions":[],"witness_signature":"2033e113f318a818d4e2b99da892f38b219180c32ffe91a3fd3f3141f9adc4c9557453628c54f88c67a7771d90c7033b2eee716f2017f3bc395fab81100d67409d","transactions":[]}} th_a db_block.cpp:293 _push_block 1506124ms th_a application.cpp:487 handle_block ] Error when pushing block: 3080000 unlinkable_block_exception: unlinkable block block does not link to known chain {} th_a db_block.cpp:206 _push_block {"new_block":{"previous":"02c59675b8db2e9c8777280dbe33e330bd8c709d","timestamp":"2022-01-13T21:54:06","witness":"1.6.87","next_secret_hash":"202b390b581e052a6e9eb0b182452001753bf658","previous_secret":"fe543ef20c0a92fa0b0b2cabc03ab2b8284c1c1d","transaction_merkle_root":"0000000000000000000000000000000000000000","extensions":[],"witness_signature":"1f286f186ad06eb95b5c70db133e2e040942edbfe517df0d848cc332f0564061f057ae9b09fd3654ffdd41ee453b5088b0aeafb673e574a60e2c6c8ef2a3c9cc53","transactions":[]}} th_a db_block.cpp:293 _push_block 874851ms th_a application.cpp:487 handle_block ] Error when pushing block: 3080000 unlinkable_block_exception: unlinkable block block does not link to known chain {} th_a db_block.cpp:206 _push_block {"new_block":{"previous":"02c59681b5153fc4306f05c1c5edf8f8352775a9","timestamp":"2022-01-13T21:53:48","witness":"1.6.87","next_secret_hash":"934f26a50c4e92e1e7f8a2031e908531da9f22a4","previous_secret":"2ccc6006701df743339400fe5b5386ad00417201","transaction_merkle_root":"0000000000000000000000000000000000000000","extensions":[],"witness_signature":"20488e4aaeae0c9cb28fa9aab33e66ee0c7bd9f311a0194f8bf035a9bdba2506d245a68b7c8c50571ef6e3570f15297b2ea6800f016d6741b2118e00caf1fa7ee9","transactions":[]}} th_a db_block.cpp:293 _push_block ``` on Hetzner VM ``` 3080000 unlinkable_block_exception: unlinkable block block does not link to known chain {} th_a db_block.cpp:206 _push_block {"new_block":{"previous":"02c596b09618f0f7a0dfc91ff5cc81735c27e589","timestamp":"2022-01-13T21:58:24","witness":"1.6.92","next_secret_hash":"ad3e428005a6d7dd97008fe1668cadc1fbea80fe","previous_secret":"6ffee3b9f7030421df659f94c53af204fdfd200d","transaction_merkle_root":"0000000000000000000000000000000000000000","extensions":[],"witness_signature":"1f7b8baff1db0d30490f5e7fcda10fb48612946342230356d24b7defc98c269f3d3420545c48a6e4fdc8c2dd89edcfa784a679ef978b21e91ba9a7080040356a48","transactions":[]}} th_a db_block.cpp:293 _push_block 2509715ms th_a application.cpp:487 handle_block ] Error when pushing block: 3080000 unlinkable_block_exception: unlinkable block block does not link to known chain {} th_a db_block.cpp:206 _push_block {"new_block":{"previous":"02c596b1c4f77130977ec28819d2af836fef3027","timestamp":"2022-01-13T21:58:27","witness":"1.6.1","next_secret_hash":"95d16669a5902b77a7354168880e8bb6ce05a75e","previous_secret":"870b7512f1bfc17686dda0f7e3fe372eead9e527","transaction_merkle_root":"0000000000000000000000000000000000000000","extensions":[],"witness_signature":"205bae1b3ef40292f2ac3d23c14f637a5a00b71b0ef3547e28c0812c7223835c361af575bbd99de90a904f05d2254dcf8b1b3c62d0e1c27ae6f688b1a33975596a","transactions":[]}} th_a db_block.cpp:293 _push_block 2509715ms th_a application.cpp:487 handle_block ] Error when pushing block: 3080000 unlinkable_block_exception: unlinkable block block does not link to known chain {} th_a db_block.cpp:206 _push_block {"new_block":{"previous":"02c596b29d699647fdd33e2027603a9a142938b5","timestamp":"2022-01-13T21:58:30","witness":"1.6.86","next_secret_hash":"e532350ae07c89de64de6721fb8a9916178f3cd8","previous_secret":"35f38da997d2a068c4289f656372a8c352d0bef2","transaction_merkle_root":"0000000000000000000000000000000000000000","extensions":[],"witness_signature":"2042f28f231e5dcd9164a09741d6374d7f598e253a8f7b94644fbfbb5c649d6475476e4a1311cfc0e638ea8bbf288d259490ccb59288f4eff171d0fc72134007fc","transactions":[]}} th_a db_block.cpp:293 _push_block 2509716ms th_a application.cpp:487 handle_block ] Error when pushing block: 3080000 unlinkable_block_exception: unlinkable block block does not link to known chain {} th_a db_block.cpp:206 _push_block {"new_block":{"previous":"02c596b3233f1fd1335a1c36ff73ce6cafe60d17","timestamp":"2022-01-13T21:58:39","witness":"1.6.89","next_secret_hash":"1c66f7b20526b8a9baf7b970837dc5bf071d5ffb","previous_secret":"c7948d0205b2e31e3e18508fc2c6b8084cd10500","transaction_merkle_root":"0000000000000000000000000000000000000000","extensions":[],"witness_signature":"1f1248f7a3127cafc60e963022d913097d94a9a9c88300f4bf34847ec62aae0d7020f26fcd03bce41ed720b5e80c4499e2d470f841f7f5d38cfc64f81cf3d684ad","transactions":[]}} th_a db_block.cpp:293 _push_block ```

👀 1

prandnum commented

2022-01-14 08:44:54 +00:00

(Migrated from gitlab.com)

made the issue confidential

prandnum commented

2022-01-14 08:45:15 +00:00

(Migrated from gitlab.com)

@bobinson @serkixenos

CC: @hbelakon

@bobinson @serkixenos CC: @hbelakon

bobinson commented

2022-01-14 15:01:07 +00:00

(Migrated from gitlab.com)

assigned to @Hiltos1

Hiltos1 commented

2022-01-14 21:26:56 +00:00

(Migrated from gitlab.com)

Mainnet is back up.
The solution is to do the following:

Stop the witness_node.
Truncate the block database back to the most recently experienced last irreversible block (LIB) number. This can be found in your logs. The truncate operation can be calculated and formed as follows:

# Example...
# If LIB was 46503509, multiply by 32 like this: 46503509 * 32 = 1488112288
# Then the truncate command in linux will be this:
truncate -s 1488112288 ./witness_node_data_dir/blockchain/database/block_num_to_block/index

Collaborate with other witnesses to find a suitable seed node to sync with. The seed node should not be in an error state. Use the seed node in your config.ini file like so:

seed-node = 3.20.153.96:9777
seed-nodes = []
# It's important to set "seed-nodes" to an empty array to avoid connecting to seed nodes in an error state!

Ensure the rest of the config.ini file is properly configured (unique keys, etc.)
Restart your node with the "--replay-blockchain" option like so:
witness_node --replay-blockchain
Your node should replay up to the lib and then rejoin the network.

Mainnet is back up. The solution is to do the following: 1. Stop the witness_node. 2. Truncate the block database back to the most recently experienced last irreversible block (LIB) number. This can be found in your logs. The truncate operation can be calculated and formed as follows: ``` # Example... # If LIB was 46503509, multiply by 32 like this: 46503509 * 32 = 1488112288 # Then the truncate command in linux will be this: truncate -s 1488112288 ./witness_node_data_dir/blockchain/database/block_num_to_block/index ``` 3. Collaborate with other witnesses to find a suitable seed node to sync with. The seed node should not be in an error state. Use the seed node in your config.ini file like so: ``` seed-node = 3.20.153.96:9777 seed-nodes = [] # It's important to set "seed-nodes" to an empty array to avoid connecting to seed nodes in an error state! ``` 4. Ensure the rest of the config.ini file is properly configured (unique keys, etc.) 5. Restart your node with the "--replay-blockchain" option like so: `witness_node --replay-blockchain` 6. Your node should replay up to the lib and then rejoin the network.

Hiltos1 commented

2022-01-14 21:29:57 +00:00

(Migrated from gitlab.com)

assigned to @bobinson

Hiltos1 commented

2022-01-14 21:30:37 +00:00

(Migrated from gitlab.com)

I'll put this ticket in review with @bobinson

Thanks,
Sean

I'll put this ticket in review with @bobinson Thanks, Sean

prandnum commented

2022-01-15 06:40:24 +00:00