Peerplays_Blockchain/peerplays_migrated

Fork 0

when blockchain replay is attempted "blocks" file increases in size #35

New issue

Closed

opened 2020-11-27 18:34:51 +00:00 by prandnum · 7 comments

prandnum commented

2020-11-27 18:34:51 +00:00

(Migrated from gitlab.com)

Steps:

Installed peerplays docker container (connection to mainnet)
Allowed the witness node to be run for a day so that it is synced and catches up with rest of the nodes
Tried replay the blockchain again, which started properly but it is seen that the "blocks" file increases from the size before replay was started

Issue Seen:

Say that the block size at T1 is 1G
Replay takes 1 hr in which new blocks size of 10Mb is created

Then if we run repay 5 times the block size would be ~6.15Gb whereas actually it should have been ~1.05GB

Ideally when the replay is started, the existing blocks file should be used to create a new blocks file and when the replay is finished the original blocks file should be removed

Logic for replay:
-- Docker container is started using the following command
sudo docker run -p0.0.0.0:2001:2001 -v /dev/shm:/shm -v /home/qa/latest/peerplays-docker/data:/peerplays -d --name peerplays -t datasecuritynode/peerplays:latest witness_node --data-dir=/peerplays/witness_node_data_dir
-- to replay the following steps are done

Stops the running peerplays docker container, if any and removes the container
Started it with the following command
sudo docker run -p0.0.0.0:2001:2001 -v /dev/shm:/shm -v /home/qa/latest/config-file-from-build/peerplays-docker/data:/peerplays -d --name peerplays -t peerplays witness_node --replay-blockchain --data-dir=/peerplays/witness_node_data_dir

-- Before replay

qa@PBSA-Dev:~/latest/config-file-from-build/peerplays-docker$ while true; do ls -lrt data/witness_node_data_dir/blockchain/database/block_num_to_block/*;date;sleep 2; done
-rw-r--r-- 1 root root 1146269760 Nov 27 08:05 data/witness_node_data_dir/blockchain/database/block_num_to_block/index
-rw-r--r-- 1 root root 5839105369 Nov 27 08:05 data/witness_node_data_dir/blockchain/database/block_num_to_block/blocks
Fri Nov 27 08:05:22 EST 2020
-rw-r--r-- 1 root root 1146269760 Nov 27 08:05 data/witness_node_data_dir/blockchain/database/block_num_to_block/index
-rw-r--r-- 1 root root 5839105369 Nov 27 08:05 data/witness_node_data_dir/blockchain/database/block_num_to_block/		
Fri Nov 27 08:05:24 EST 2020

-- Started the replay

3595306ms th_a       accounts_list_plugin.cpp:129  list_accounts        ] accounts list plugin:  list_accounts()
3595306ms th_a       bookie_plugin.cpp:469         plugin_initialize    ] bookie plugin: plugin_startup() begin
3595306ms th_a       db_management.cpp:290         force_slow_replays   ] enabling slow replays
3595306ms th_a       bookie_plugin.cpp:502         plugin_initialize    ] bookie plugin: plugin_startup() end
3595307ms th_a       witness.cpp:92                plugin_initialize    ] witness plugin:  plugin_initialize() begin
3595307ms th_a       witness.cpp:104               plugin_initialize    ] Public Key: PPY6MRyAjQq8ud7hVNYcfnVPJqcVpscN5So8BhtHuGYqET5GDW5CV
3595307ms th_a       witness.cpp:122               plugin_initialize    ] witness plugin:  plugin_initialize() end
3595307ms th_a       db_management.cpp:180         wipe                 ] Wiping database
3595307ms th_a       object_database.cpp:93        wipe                 ] Wiping object database...
3595397ms th_a       object_database.cpp:95        wipe                 ] Done wiping object databse.
3595397ms th_a       object_database.cpp:106       open                 ] Opening object database from /peerplays/witness_node_data_dir/blockchain ...
3595397ms th_a       object_database.cpp:111       open                 ] Done opening object database.
3595398ms th_a       application.cpp:349           operator()           ] Initializing database...
3597155ms th_a       db_debug.cpp:85               debug_dump           ] total_balances[asset_id_type()].value: 239781396552 core_asset_data.current_supply.value: 546051993031 
3597157ms th_a       db_management.cpp:107         reindex              ] reindexing blockchain
3597157ms th_a       db_management.cpp:113         reindex              ] Replaying blocks, starting at 1...
3597160ms th_a       db_maint.cpp:1321             process_dividend_ass ] In process_dividend_assets time 2017-06-06T16:00:03
3597475ms th_a       db_maint.cpp:1321             process_dividend_ass ] In process_dividend_assets time 2017-06-06T17:00:00
3597688ms th_a       db_maint.cpp:1321             process_dividend_ass ] In process_dividend_assets time 2017-06-06T18:00:00

-- During replay

qa@PBSA-Dev:~/latest/config-file-from-build/peerplays-docker$ while true; do ls -lrt data/witness_node_data_dir/blockchain/database/block_num_to_block/*;date;sleep 2; done
-rw-r--r-- 1 root root 6248687095 Nov 27 12:06 data/witness_node_data_dir/blockchain/database/block_num_to_block/blocks
-rw-r--r-- 1 root root 1146269952 Nov 27 12:06 data/witness_node_data_dir/blockchain/database/block_num_to_block/index
Fri Nov 27 12:06:57 EST 2020
-rw-r--r-- 1 root root 1146269952 Nov 27 12:06 data/witness_node_data_dir/blockchain/database/block_num_to_block/index
-rw-r--r-- 1 root root 6250662887 Nov 27 12:06 data/witness_node_data_dir/blockchain/database/block_num_to_block/blocks
Fri Nov 27 12:06:59 EST 2020
-rw-r--r-- 1 root root 1146269952 Nov 27 12:07 data/witness_node_data_dir/blockchain/database/block_num_to_block/index
-rw-r--r-- 1 root root 6252597960 Nov 27 12:07 data/witness_node_data_dir/blockchain/database/block_num_to_block/blocks
Fri Nov 27 12:07:01 EST 2020
-rw-r--r-- 1 root root 1146269952 Nov 27 12:07 data/witness_node_data_dir/blockchain/database/block_num_to_block/index
-rw-r--r-- 1 root root 6254548444 Nov 27 12:07 data/witness_node_data_dir/blockchain/database/block_num_to_block/blocks
Fri Nov 27 12:07:03 EST 2020
-rw-r--r-- 1 root root 1146269952 Nov 27 12:07 data/witness_node_data_dir/blockchain/database/block_num_to_block/index
-rw-r--r-- 1 root root 6256454395 Nov 27 12:07 data/witness_node_data_dir/blockchain/database/block_num_to_block/blocks
Fri Nov 27 12:07:05 EST 2020
-rw-r--r-- 1 root root 1146269952 Nov 27 12:07 data/witness_node_data_dir/blockchain/database/block_num_to_block/index
-rw-r--r-- 1 root root 6258449823 Nov 27 12:07 data/witness_node_data_dir/blockchain/database/block_num_to_block/blocks
Fri Nov 27 12:07:07 EST 2020
-rw-r--r-- 1 root root 1146269952 Nov 27 12:07 data/witness_node_data_dir/blockchain/database/block_num_to_block/index
-rw-r--r-- 1 root root 6260414011 Nov 27 12:07 data/witness_node_data_dir/blockchain/database/block_num_to_block/blocks
Fri Nov 27 12:07:09 EST 2020
-rw-r--r-- 1 root root 1146269952 Nov 27 12:07 data/witness_node_data_dir/blockchain/database/block_num_to_block/index
-rw-r--r-- 1 root root 6262065692 Nov 27 12:07 data/witness_node_data_dir/blockchain/database/block_num_to_block/blocks
Fri Nov 27 12:07:11 EST 2020
-rw-r--r-- 1 root root 1146269952 Nov 27 12:07 data/witness_node_data_dir/blockchain/database/block_num_to_block/index
-rw-r--r-- 1 root root 6263677500 Nov 27 12:07 data/witness_node_data_dir/blockchain/database/block_num_to_block/blocks
Fri Nov 27 12:07:13 EST 2020

Steps: 1) Installed peerplays docker container (connection to mainnet) 2) Allowed the witness node to be run for a day so that it is synced and catches up with rest of the nodes 3) Tried replay the blockchain again, which started properly but it is seen that the "blocks" file increases from the size before replay was started Issue Seen: 1) Say that the block size at T1 is 1G 2) Replay takes 1 hr in which new blocks size of 10Mb is created Then if we run repay 5 times the block size would be ~6.15Gb whereas actually it should have been ~1.05GB Ideally when the replay is started, the existing blocks file should be used to create a new blocks file and when the replay is finished the original blocks file should be removed Logic for replay: -- Docker container is started using the following command sudo docker run -p0.0.0.0:2001:2001 -v /dev/shm:/shm -v /home/qa/latest/peerplays-docker/data:/peerplays -d --name peerplays -t datasecuritynode/peerplays:latest witness_node --data-dir=/peerplays/witness_node_data_dir -- to replay the following steps are done 1) Stops the running peerplays docker container, if any and removes the container 2) Started it with the following command sudo docker run -p0.0.0.0:2001:2001 -v /dev/shm:/shm -v /home/qa/latest/config-file-from-build/peerplays-docker/data:/peerplays -d --name peerplays -t peerplays witness_node --replay-blockchain --data-dir=/peerplays/witness_node_data_dir --------------------------------------------------------------------------------------------------------------------- -- Before replay ``` qa@PBSA-Dev:~/latest/config-file-from-build/peerplays-docker$ while true; do ls -lrt data/witness_node_data_dir/blockchain/database/block_num_to_block/*;date;sleep 2; done -rw-r--r-- 1 root root 1146269760 Nov 27 08:05 data/witness_node_data_dir/blockchain/database/block_num_to_block/index -rw-r--r-- 1 root root 5839105369 Nov 27 08:05 data/witness_node_data_dir/blockchain/database/block_num_to_block/blocks Fri Nov 27 08:05:22 EST 2020 -rw-r--r-- 1 root root 1146269760 Nov 27 08:05 data/witness_node_data_dir/blockchain/database/block_num_to_block/index -rw-r--r-- 1 root root 5839105369 Nov 27 08:05 data/witness_node_data_dir/blockchain/database/block_num_to_block/ Fri Nov 27 08:05:24 EST 2020 ``` -- Started the replay ``` 3595306ms th_a accounts_list_plugin.cpp:129 list_accounts ] accounts list plugin: list_accounts() 3595306ms th_a bookie_plugin.cpp:469 plugin_initialize ] bookie plugin: plugin_startup() begin 3595306ms th_a db_management.cpp:290 force_slow_replays ] enabling slow replays 3595306ms th_a bookie_plugin.cpp:502 plugin_initialize ] bookie plugin: plugin_startup() end 3595307ms th_a witness.cpp:92 plugin_initialize ] witness plugin: plugin_initialize() begin 3595307ms th_a witness.cpp:104 plugin_initialize ] Public Key: PPY6MRyAjQq8ud7hVNYcfnVPJqcVpscN5So8BhtHuGYqET5GDW5CV 3595307ms th_a witness.cpp:122 plugin_initialize ] witness plugin: plugin_initialize() end 3595307ms th_a db_management.cpp:180 wipe ] Wiping database 3595307ms th_a object_database.cpp:93 wipe ] Wiping object database... 3595397ms th_a object_database.cpp:95 wipe ] Done wiping object databse. 3595397ms th_a object_database.cpp:106 open ] Opening object database from /peerplays/witness_node_data_dir/blockchain ... 3595397ms th_a object_database.cpp:111 open ] Done opening object database. 3595398ms th_a application.cpp:349 operator() ] Initializing database... 3597155ms th_a db_debug.cpp:85 debug_dump ] total_balances[asset_id_type()].value: 239781396552 core_asset_data.current_supply.value: 546051993031 3597157ms th_a db_management.cpp:107 reindex ] reindexing blockchain 3597157ms th_a db_management.cpp:113 reindex ] Replaying blocks, starting at 1... 3597160ms th_a db_maint.cpp:1321 process_dividend_ass ] In process_dividend_assets time 2017-06-06T16:00:03 3597475ms th_a db_maint.cpp:1321 process_dividend_ass ] In process_dividend_assets time 2017-06-06T17:00:00 3597688ms th_a db_maint.cpp:1321 process_dividend_ass ] In process_dividend_assets time 2017-06-06T18:00:00 ``` -- During replay ``` qa@PBSA-Dev:~/latest/config-file-from-build/peerplays-docker$ while true; do ls -lrt data/witness_node_data_dir/blockchain/database/block_num_to_block/*;date;sleep 2; done -rw-r--r-- 1 root root 6248687095 Nov 27 12:06 data/witness_node_data_dir/blockchain/database/block_num_to_block/blocks -rw-r--r-- 1 root root 1146269952 Nov 27 12:06 data/witness_node_data_dir/blockchain/database/block_num_to_block/index Fri Nov 27 12:06:57 EST 2020 -rw-r--r-- 1 root root 1146269952 Nov 27 12:06 data/witness_node_data_dir/blockchain/database/block_num_to_block/index -rw-r--r-- 1 root root 6250662887 Nov 27 12:06 data/witness_node_data_dir/blockchain/database/block_num_to_block/blocks Fri Nov 27 12:06:59 EST 2020 -rw-r--r-- 1 root root 1146269952 Nov 27 12:07 data/witness_node_data_dir/blockchain/database/block_num_to_block/index -rw-r--r-- 1 root root 6252597960 Nov 27 12:07 data/witness_node_data_dir/blockchain/database/block_num_to_block/blocks Fri Nov 27 12:07:01 EST 2020 -rw-r--r-- 1 root root 1146269952 Nov 27 12:07 data/witness_node_data_dir/blockchain/database/block_num_to_block/index -rw-r--r-- 1 root root 6254548444 Nov 27 12:07 data/witness_node_data_dir/blockchain/database/block_num_to_block/blocks Fri Nov 27 12:07:03 EST 2020 -rw-r--r-- 1 root root 1146269952 Nov 27 12:07 data/witness_node_data_dir/blockchain/database/block_num_to_block/index -rw-r--r-- 1 root root 6256454395 Nov 27 12:07 data/witness_node_data_dir/blockchain/database/block_num_to_block/blocks Fri Nov 27 12:07:05 EST 2020 -rw-r--r-- 1 root root 1146269952 Nov 27 12:07 data/witness_node_data_dir/blockchain/database/block_num_to_block/index -rw-r--r-- 1 root root 6258449823 Nov 27 12:07 data/witness_node_data_dir/blockchain/database/block_num_to_block/blocks Fri Nov 27 12:07:07 EST 2020 -rw-r--r-- 1 root root 1146269952 Nov 27 12:07 data/witness_node_data_dir/blockchain/database/block_num_to_block/index -rw-r--r-- 1 root root 6260414011 Nov 27 12:07 data/witness_node_data_dir/blockchain/database/block_num_to_block/blocks Fri Nov 27 12:07:09 EST 2020 -rw-r--r-- 1 root root 1146269952 Nov 27 12:07 data/witness_node_data_dir/blockchain/database/block_num_to_block/index -rw-r--r-- 1 root root 6262065692 Nov 27 12:07 data/witness_node_data_dir/blockchain/database/block_num_to_block/blocks Fri Nov 27 12:07:11 EST 2020 -rw-r--r-- 1 root root 1146269952 Nov 27 12:07 data/witness_node_data_dir/blockchain/database/block_num_to_block/index -rw-r--r-- 1 root root 6263677500 Nov 27 12:07 data/witness_node_data_dir/blockchain/database/block_num_to_block/blocks Fri Nov 27 12:07:13 EST 2020 ```

bobinson commented

2021-02-15 05:49:55 +00:00

(Migrated from gitlab.com)

assigned to @serkixenos

serkixenos commented

2021-02-22 13:03:42 +00:00

(Migrated from gitlab.com)

Bug is confirmed and reproduced.

Blocks database is defined in two files, at location witness_node_data_dir/blockchain/block_num_to_block:

blocks - a list of downloaded blocks
index - index file, holding information about block number, start position and length in blocks file.

At the fully synced chain, first item in index file says that block 0 starts at blocks file position 0, and has length X, second item - block 1 starts at position X, length Y, block 2 starts at X + Y, length Z, etc...

During replay, we are reading data from these files, and process them, but once the block is processed, we are saving the block at the end of the blocks file, and update the index file to point to the new position.

Example:

This is the structure holding block position info. index file holds array of these structures
struct index_entry

{
   uint64_t      block_pos = 0;
   uint32_t      block_size = 0;
   block_id_type block_id;
};

Current state of the files:

blocks file size iz 1000150.

index[0].block_pos = 0;
index[0].block_size = 150;
index[0].block_id = 0;

index[1].block_pos = 150;
index[1].block_size = 100;
index[1].block_id = 1;

index[2].block_pos = 250;
index[2].block_size = 150;
index[2].block_id = 3;

...

index[n].block_pos = 1000000;
index[n].block_size = 150;
index[n].block_id = n;

Now, we start the replay, and stop it after two blocks. New state would be:

blocks file size iz 1000400.

index[0].block_pos = 1000150;
index[0].block_size = 150;
index[0].block_id = 0;

index[1].block_pos = 1000300;
index[1].block_size = 100;
index[1].block_id = 1;

index[2].block_pos = 250;
index[2].block_size = 150;
index[2].block_id = 3;

...

index[n].block_pos = 1000000;
index[n].block_size = 150;
index[n].block_id = n;

This means that we lost "pointer" to the first instances of the blocks 0 and 1, inside blocks file, as we rewrote their index entries to point to the newly written blocks at the end of the file, while not removing existing block data at the file start.

Solution - make the system aware that it is in a replay mode, and do not append replayed blocks to the end of the blocks file.

Bug is confirmed and reproduced. Blocks database is defined in two files, at location witness_node_data_dir/blockchain/block_num_to_block: blocks - a list of downloaded blocks index - index file, holding information about block number, start position and length in blocks file. At the fully synced chain, first item in index file says that block 0 starts at blocks file position 0, and has length X, second item - block 1 starts at position X, length Y, block 2 starts at X + Y, length Z, etc... During replay, we are reading data from these files, and process them, but once the block is processed, we are saving the block at the end of the blocks file, and update the index file to point to the new position. Example: This is the structure holding block position info. index file holds array of these structures struct index_entry ``` { uint64_t block_pos = 0; uint32_t block_size = 0; block_id_type block_id; }; ``` Current state of the files: ``` blocks file size iz 1000150. index[0].block_pos = 0; index[0].block_size = 150; index[0].block_id = 0; index[1].block_pos = 150; index[1].block_size = 100; index[1].block_id = 1; index[2].block_pos = 250; index[2].block_size = 150; index[2].block_id = 3; ... index[n].block_pos = 1000000; index[n].block_size = 150; index[n].block_id = n; ``` Now, we start the replay, and stop it after two blocks. New state would be: ``` blocks file size iz 1000400. index[0].block_pos = 1000150; index[0].block_size = 150; index[0].block_id = 0; index[1].block_pos = 1000300; index[1].block_size = 100; index[1].block_id = 1; index[2].block_pos = 250; index[2].block_size = 150; index[2].block_id = 3; ... index[n].block_pos = 1000000; index[n].block_size = 150; index[n].block_id = n; ``` This means that we lost "pointer" to the first instances of the blocks 0 and 1, inside blocks file, as we rewrote their index entries to point to the newly written blocks at the end of the file, while not removing existing block data at the file start. Solution - make the system aware that it is in a replay mode, and do not append replayed blocks to the end of the blocks file.

serkixenos commented

2021-11-23 13:30:29 +00:00

(Migrated from gitlab.com)

assigned to @pavel.baykov and unassigned @serkixenos

prandnum commented

2021-12-01 15:57:52 +00:00

(Migrated from gitlab.com)

@pavel.baykov Could not reproduce the issue. the fix is working. please merge the code then we can close the ticket.

CC: @serkixenos @bobinson @hbelakon

@pavel.baykov Could not reproduce the issue. the fix is working. please merge the code then we can close the ticket. CC: @serkixenos @bobinson @hbelakon

yviter commented

2021-12-01 17:49:35 +00:00

(Migrated from gitlab.com)

assigned to @yviter

serkixenos (Migrated from gitlab.com) closed this issue

2021-12-01 21:58:32 +00:00

yviter (Migrated from gitlab.com) reopened this issue

2021-12-02 22:02:16 +00:00

yviter commented

2021-12-02 22:02:35 +00:00

(Migrated from gitlab.com)

still testing on irona