Resolve sync problem with IRONA #393

New issue

Closed

opened 2022-06-18 21:06:28 +00:00 by hirunda · 4 comments

hirunda commented

2022-06-18 21:06:28 +00:00

(Migrated from gitlab.com)

On IRONA during sync process halt is happening on the block as described bellow:

1403497ms th_a fork_database.cpp:65 push_block ] Pushing block to fork database that failed to link: 000e3fad1a6d7edf6faaadb4c6d0df8cd0f7244c, 933805 1403497ms th_a fork_database.cpp:66 push_block ] Head: 933802, 000e3faa13a374ed4b0c76d53447531c41976002 1403497ms th_a application.cpp:496 handle_block ] Error when pushing block:

The block is rejected due the wrong scheduled witness:

10 assert_exception: Assert Exception next_block.witness == scheduled_witness: Witness produced block at wrong time {"block witness":"1.6.5","scheduled":"1.6.4","slot_num":19893} th_a db_block.cpp:903 validate_block_header

Find the root cause of the issue

On IRONA during sync process halt is happening on the block as described bellow: `1403497ms th_a fork_database.cpp:65 push_block ] Pushing block to fork database that failed to link: 000e3fad1a6d7edf6faaadb4c6d0df8cd0f7244c, 933805 1403497ms th_a fork_database.cpp:66 push_block ] Head: 933802, 000e3faa13a374ed4b0c76d53447531c41976002 1403497ms th_a application.cpp:496 handle_block ] Error when pushing block:` The block is rejected due the wrong scheduled witness: `10 assert_exception: Assert Exception next_block.witness == scheduled_witness: Witness produced block at wrong time {"block witness":"1.6.5","scheduled":"1.6.4","slot_num":19893} th_a db_block.cpp:903 validate_block_header` Find the root cause of the issue

hirunda commented

2022-06-18 21:06:28 +00:00

(Migrated from gitlab.com)

assigned to @hirunda

hirunda commented

2022-06-18 21:06:45 +00:00

(Migrated from gitlab.com)

changed the description

hirunda commented

2022-07-04 11:52:47 +00:00

(Migrated from gitlab.com)

The only way to resolve this bug is to restart witness node on IRONA.
--replay-blockchain will not help.
The problem is that witness node was restarted without --replay-blockchain and now we have wrong scheduler state
which will break the sync on the specific block and after that block.
The second approach would be making the changes in the code specific to IRONA case, by detecting problematic block
and move scheduler state to be aligned with IRONA.

Important note: This issue is only specific to IRONA. With all fixes on the master and develop branch, there is no need anymore for --replay-blockchain flag when witness node is restarted.

1) The only way to resolve this bug is to restart witness node on IRONA. --replay-blockchain will not help. The problem is that witness node was restarted without --replay-blockchain and now we have wrong scheduler state which will break the sync on the specific block and after that block. 2) The second approach would be making the changes in the code specific to IRONA case, by detecting problematic block and move scheduler state to be aligned with IRONA. Important note: This issue is only specific to IRONA. With all fixes on the master and develop branch, there is no need anymore for --replay-blockchain flag when witness node is restarted.

serkixenos commented

2022-07-05 12:29:30 +00:00

(Migrated from gitlab.com)

Wont fix, as this problem is specific to Irona only. The problem is caused by invalid witness scheduler initialization, which is fixed in develop branch. The problem will disappear if Irona is restarted from scratch.

serkixenos (Migrated from gitlab.com) closed this issue

2022-07-05 12:29:37 +00:00