Peerplays_Blockchain/peerplays_migrated

Fork 0

"Error when pushing block" when the directory is full #389

New issue

Closed

opened 2022-06-13 16:26:14 +00:00 by prandnum · 11 comments

prandnum commented

2022-06-13 16:26:14 +00:00

(Migrated from gitlab.com)

"Error when pushing block" when the directory is full.

Expected outcome: When the disk is full and the witness process is not able to write to the disk, "Disk Full" text in the error message would help in troubleshooting the issue.

Logs attached. diskfull.tar.gz

"Error when pushing block" when the directory is full. Expected outcome: When the disk is full and the witness process is not able to write to the disk, "Disk Full" text in the error message would help in troubleshooting the issue. Logs attached. [diskfull.tar.gz](/uploads/14488bb4c5c08fb29f4ee1d672415690/diskfull.tar.gz)

prandnum commented

2022-06-13 16:26:42 +00:00

(Migrated from gitlab.com)

@bobinson @serkixenos

serkixenos commented

2022-07-26 23:47:06 +00:00

(Migrated from gitlab.com)

Usefull function
https://linux.die.net/man/2/statfs64

Usefull function https://linux.die.net/man/2/statfs64

serkixenos commented

2022-07-26 23:47:14 +00:00

(Migrated from gitlab.com)

assigned to @hirunda

serkixenos commented

2022-09-07 13:54:33 +00:00

(Migrated from gitlab.com)

unassigned @hirunda

serkixenos commented

2022-09-07 13:54:52 +00:00

(Migrated from gitlab.com)

assigned to @mkhan17

mkhan17 commented

2022-09-20 14:56:49 +00:00

(Migrated from gitlab.com)

Test Instructions for the fix:

copy large amount of data in your /home directory or /root directory wherever the witness_node_data_dir is present.
You can also create large files using this link (https://www.cyberciti.biz/faq/howto-create-lage-files-with-dd-command/)
Make sure you have even less than 50 mb space available, by running command sudo df -h
Run the witness node on baremetal
It will give you an exception as seen in this log (baremetal-runlog.txt)

Test Instructions for the fix: 1) copy large amount of data in your /home directory or /root directory wherever the witness_node_data_dir is present. You can also create large files using this link (https://www.cyberciti.biz/faq/howto-create-lage-files-with-dd-command/) 2) Make sure you have even less than 50 mb space available, by running command sudo df -h 3) Run the witness node on baremetal 4) It will give you an exception as seen in this log ([baremetal-runlog.txt](/uploads/713b46b7be063e563b4a19b570fff33e/baremetal-runlog.txt))

mkhan17 commented

2022-09-21 12:28:54 +00:00

(Migrated from gitlab.com)

Please also test this below scenario and see the behaviour

Test scenario created as follows:

Have space of around 120 MB in the /home directory
start the witness node, It is observed that witness node generates blocks with no issues
Make the space available to around 43 MB
It is seen that there are exceptions such as "Rejecting block due to low disk space" in the witness node from push_block function in db_block.cpp
Delete some files to make more space available
It is seen that block generation goes back to normal and new blocks are generated

Log for this test tesdt_log.txt

Please also test this below scenario and see the behaviour Test scenario created as follows: 1. Have space of around 120 MB in the /home directory 2. start the witness node, It is observed that witness node generates blocks with no issues 3. Make the space available to around 43 MB 4. It is seen that there are exceptions such as "Rejecting block due to low disk space" in the witness node from push_block function in db_block.cpp 5. Delete some files to make more space available 6. It is seen that block generation goes back to normal and new blocks are generated Log for this test [tesdt_log.txt](/uploads/91c0389a3e4624a40c50afe0671d20bb/tesdt_log.txt)

serkixenos commented

2022-09-26 19:27:08 +00:00

(Migrated from gitlab.com)

assigned to @prandnum and @wsalloum

serkixenos commented

2022-09-26 19:44:18 +00:00

(Migrated from gitlab.com)

Please note that free space limit is increased to 100 MB.

Current test instructions are covering only all in one QA scenario where single node generates all blocks. Make sure to cover scenario with multiple nodes, where non witness node (the one not generating blocks) stops receiving blocks due to low disk space, and check what happens when the space is available again.

Eg: Sync node stops accepting blocks at block 1000, but witnesses are continuing to build blocks, up to 1020. Then ffee disk space on sync node, and verify that sync node downloads blocks from 1001 to 1020, and then continues receiving blocks as usual.

Another scenario is with a witness node building a block when its disk space is low. Check is this block broadcasted to the network, will it be accepted by other nodes, or not.

@mkhan17 Please provide test instructions for these scenarios too. This is about taking/freeing up disk space on different nodes in QA environment. You might need to investigate docker options for limiting volume sizes.

Please note that free space limit is increased to 100 MB. Current test instructions are covering only all in one QA scenario where single node generates all blocks. Make sure to cover scenario with multiple nodes, where non witness node (the one not generating blocks) stops receiving blocks due to low disk space, and check what happens when the space is available again. Eg: Sync node stops accepting blocks at block 1000, but witnesses are continuing to build blocks, up to 1020. Then ffee disk space on sync node, and verify that sync node downloads blocks from 1001 to 1020, and then continues receiving blocks as usual. Another scenario is with a witness node building a block when its disk space is low. Check is this block broadcasted to the network, will it be accepted by other nodes, or not. @mkhan17 Please provide test instructions for these scenarios too. This is about taking/freeing up disk space on different nodes in QA environment. You might need to investigate docker options for limiting volume sizes.

mkhan17 commented

2022-09-27 18:38:11 +00:00

(Migrated from gitlab.com)

Test scenario created for Peerplays QA Environment as follows:

Start Docker Environment with all 16 Peerplays nodes
Ensure you have space of more than 100 MB in the /root directory
Make the space available to around 90 MB in the /root directory since docker is running in /var/lib/docker directory
It is seen that there are exceptions such as "Rejecting block due to low disk space" from all the nodes.
Delete some files to make more space available
It is seen that nodes reject the transaction with messages such as below and after sometime block production goes back to normal. Please see log for more details (docker-space-test-log.txt)

peerplays15_1 | 3080000 unlinkable_block_exception: unlinkable block peerplays15_1 | block does not link to known chain

and also like this message

peerplays15_1 | 1383566ms th_a application.cpp:460 handle_block ] Error when pushing block: peerplays15_1 | 10 assert_exception: Assert Exception peerplays15_1 | new_block.witness == wid: Witness produced block at wrong time peerplays15_1 | {"block witness":"1.6.4","scheduled":"1.6.1","slot_num":5} peerplays15_1 | th_a db_block.cpp:342 verify_signing_witness

Test scenario created for Peerplays QA Environment as follows: 1. Start Docker Environment with all 16 Peerplays nodes 2. Ensure you have space of more than 100 MB in the /root directory 3. Make the space available to around 90 MB in the /root directory since docker is running in /var/lib/docker directory 4. It is seen that there are exceptions such as "Rejecting block due to low disk space" from all the nodes. 5. Delete some files to make more space available 6. It is seen that nodes reject the transaction with messages such as below and after sometime block production goes back to normal. Please see log for more details ([docker-space-test-log.txt](/uploads/b5bc4d2c66479429767ce189ab9956e0/docker-space-test-log.txt)) `` peerplays15_1 | 3080000 unlinkable_block_exception: unlinkable block peerplays15_1 | block does not link to known chain `` and also like this message `` peerplays15_1 | 1383566ms th_a application.cpp:460 handle_block ] Error when pushing block: peerplays15_1 | 10 assert_exception: Assert Exception peerplays15_1 | new_block.witness == wid: Witness produced block at wrong time peerplays15_1 | {"block witness":"1.6.4","scheduled":"1.6.1","slot_num":5} peerplays15_1 | th_a db_block.cpp:342 verify_signing_witness ``

wsalloum commented

2022-09-29 09:58:13 +00:00

(Migrated from gitlab.com)

Run QA Environment with all 16 Peerplays nodes
docker-compose up peerplays01 peerplays02 peerplays03 peerplays04 peerplays05 peerplays06 peerplays07 peerplays08 peerplays09 peerplays10 peerplays11
Check the log, it is working normally

peerplays10_1                     | 1638003ms th_a       witness.cpp:191               block_production_loo ] Generated block #30 with timestamp 2022-09-29T09:27:18 at time 2022-09-29T09:27:18
peerplays12_1                     | 1638011ms th_a       application.cpp:429           handle_block         ] Got block: #30 time: 2022-09-29T09:27:18 latency: 11 ms from: init9  irreversible: 22 (-8)
peerplays09_1                     | 1638011ms th_a       application.cpp:429           handle_block         ] Got block: #30 time: 2022-09-29T09:27:18 latency: 11 ms from: init9  irreversible: 22 (-8)
peerplays02_1                     | 1638011ms th_a       application.cpp:429           handle_block         ] Got block: #30 time: 2022-09-29T09:27:18 latency: 11 ms from: init9  irreversible: 22 (-8)
peerplays05_1                     | 1638011ms th_a       application.cpp:429           handle_block         ] Got block: #30 time: 2022-09-29T09:27:18 latency: 11 ms from: init9  irreversible: 22 (-8)
peerplays06_1                     | 1638014ms th_a       application.cpp:429           handle_block         ] Got block: #30 time: 2022-09-29T09:27:18 latency: 14 ms from: init9  irreversible: 22 (-8)
peerplays15_1                     | 1638016ms th_a       application.cpp:429           handle_block         ] Got block: #30 time: 2022-09-29T09:27:18 latency: 16 ms from: init9  irreversible: 22 (-8)
peerplays04_1                     | 1638016ms th_a       application.cpp:429           handle_block         ] Got block: #30 time: 2022-09-29T09:27:18 latency: 16 ms from: init9  irreversible: 22 (-8)
peerplays08_1                     | 1638017ms th_a       application.cpp:429           handle_block         ] Got block: #30 time: 2022-09-29T09:27:18 latency: 17 ms from: init9  irreversible: 22 (-8)
peerplays13_1                     | 1638017ms th_a       application.cpp:429           handle_block         ] Got block: #30 time: 2022-09-29T09:27:18 latency: 17 ms from: init9  irreversible: 22 (-8)
peerplays03_1                     | 1638018ms th_a       application.cpp:429           handle_block         ] Got block: #30 time: 2022-09-29T09:27:18 latency: 18 ms from: init9  irreversible: 22 (-8)
peerplays01_1                     | 1638019ms th_a       application.cpp:429           handle_block         ] Got block: #30 time: 2022-09-29T09:27:18 latency: 19 ms from: init9  irreversible: 22 (-8)
peerplays14_1                     | 1638020ms th_a       application.cpp:429           handle_block         ] Got block: #30 time: 2022-09-29T09:27:18 latency: 20 ms from: init9  irreversible: 22 (-8)
peerplays07_1                     | 1638027ms th_a       application.cpp:429           handle_block         ] Got block: #30 time: 2022-09-29T09:27:18 latency: 27 ms from: init9  irreversible: 22 (-8)
peerplays11_1                     | 1638027ms th_a       application.cpp:429           handle_block         ] Got block: #30 time: 2022-09-29T09:27:18 latency: 27 ms from: init9  irreversible: 22 (-8)
peerplays16_1                     | 1638029ms th_a       application.cpp:429           handle_block         ] Got block: #30 time: 2022-09-29T09:27:18 latency: 29 ms from: init9  irreversible: 22 (-8)
peerplays03_1                     | 1641000ms th_a       witness.cpp:267               maybe_produce_block  ] slot: 1 scheduled_witness: 1.6.3 scheduled_time: 2022-09-29T09:27:21 now: 2022-09-29T09:27:21 
peerplays03_1                     | 1641002ms th_a       witness.cpp:191               block_production_loo ] Generated block #31 with timestamp 2022-09-29T09:27:21 at time 2022-09-29T09:27:21

Create a large file on /var and make the space 15MB fallocate -l 7.77G test2.img
Check the log, the error message Rejecting block due to low disk space is there

peerplays09_1                     | (si.available) > 104857600: Rejecting block due to low disk space
peerplays09_1                     |     {}
peerplays09_1                     |     th_a  db_block.cpp:202 _push_block
peerplays09_1                     | 
peerplays09_1                     |     {"new_block":{"previous":"0000013d8048ed635f5508b5e7777d3178398c07","timestamp":"2022-09-29T09:42:30","witness":"1.6.9","next_secret_hash":"219cedb62d7ed191979f4677ef92f6ca45111b85","previous_secret":"092deea00c65151f27d7fa5e08e6ec397e2f95fc","transaction_merkle_root":"0000000000000000000000000000000000000000","extensions":[],"witness_signature":"20181bffcd048d8fa4da35ec2c14eb8f11c54081d6acf42fddafd193d21638bb015e16ba6d9289f73799ca75d76a13c652658180e0fa8a6fc8dc3cdced33769e24","transactions":[]}}
peerplays09_1                     |     th_a  db_block.cpp:298 _push_block
peerplays09_1                     | 
peerplays09_1                     |     {"witness_id":"1.6.9"}
peerplays09_1                     |     th_a  db_block.cpp:592 _generate_block
peerplays09_1                     | 
peerplays09_1                     |     {}
peerplays09_1                     |     th_a  db_block.cpp:475 generate_block
peerplays09_1                     | 2550001ms th_a       witness.cpp:184               block_production_loo ] Discarding all pending transactions in an attempt to prevent the same error from occurring the next time we try to produce a block
peerplays09_1                     | 2550001ms th_a       witness.cpp:216               block_production_loo ] exception producing block
peerplays09_1                     | 2551000ms th_a       witness.cpp:267               maybe_produce_block  ] slot: 11 scheduled_witness: 1.6.9 scheduled_time: 2022-09-29T09:42:30 now: 2022-09-29T09:42:31 
peerplays09_1                     | 2551001ms th_a       witness.cpp:182               block_production_loo ] Got exception while generating block:
peerplays09_1                     | 10 assert_exception: Assert Exception
peerplays09_1                     | (si.available) > 104857600: Rejecting block due to low disk space
peerplays09_1                     |     {}
peerplays09_1                     |     th_a  db_block.cpp:202 _push_block
peerplays09_1                     | 
peerplays09_1                     |     {"new_block":{"previous":"0000013d8048ed635f5508b5e7777d3178398c07","timestamp":"2022-09-29T09:42:30","witness":"1.6.9","next_secret_hash":"219cedb62d7ed191979f4677ef92f6ca45111b85","previous_secret":"092deea00c65151f27d7fa5e08e6ec397e2f95fc","transaction_merkle_root":"0000000000000000000000000000000000000000","extensions":[],"witness_signature":"20181bffcd048d8fa4da35ec2c14eb8f11c54081d6acf42fddafd193d21638bb015e16ba6d9289f73799ca75d76a13c652658180e0fa8a6fc8dc3cdced33769e24","transactions":[]}}
peerplays09_1                     |     th_a  db_block.cpp:298 _push_block
peerplays09_1                     | 
peerplays09_1                     |     {"witness_id":"1.6.9"}
peerplays09_1                     |     th_a  db_block.cpp:592 _generate_block
peerplays09_1                     | 
peerplays09_1                     |     {}
peerplays09_1                     |     th_a  db_block.cpp:475 generate_block
peerplays09_1                     | 2551001ms th_a       witness.cpp:184               block_production_loo ] Discarding all pending transactions in an attempt to prevent the same error from occurring the next time we try to produce a block
peerplays09_1                     | 2551001ms th_a       witness.cpp:216               block_production_loo ] exception producing block
peerplays09_1                     | 2552000ms th_a       witness.cpp:267               maybe_produce_block  ] slot: 11 scheduled_witness: 1.6.9 scheduled_time: 2022-09-29T09:42:30 now: 2022-09-29T09:42:32 
peerplays09_1                     | 2552001ms th_a       witness.cpp:182               block_production_loo ] Got exception while generating block:
peerplays09_1                     | 10 assert_exception: Assert Exception
peerplays09_1                     | (si.available) > 104857600: Rejecting block due to low disk space

Remove test2.img file on /var (Make more space on /var) and check the log again, and make sure it is back to normal and generating blocks again

peerplays05_1                     | 2844004ms th_a       witness.cpp:191               block_production_loo ] Generated block #411 with timestamp 2022-09-29T09:47:24 at time 2022-09-29T09:47:24
peerplays07_1                     | 2844011ms th_a       application.cpp:429           handle_block         ] Got block: #411 time: 2022-09-29T09:47:24 latency: 11 ms from: init4  irreversible: 403 (-8)
peerplays09_1                     | 2844011ms th_a       application.cpp:429           handle_block         ] Got block: #411 time: 2022-09-29T09:47:24 latency: 11 ms from: init4  irreversible: 403 (-8)
peerplays10_1                     | 2844011ms th_a       application.cpp:429           handle_block         ] Got block: #411 time: 2022-09-29T09:47:24 latency: 11 ms from: init4  irreversible: 403 (-8)
peerplays13_1                     | 2844012ms th_a       application.cpp:429           handle_block         ] Got block: #411 time: 2022-09-29T09:47:24 latency: 12 ms from: init4  irreversible: 403 (-8)
peerplays01_1                     | 2844014ms th_a       application.cpp:429           handle_block         ] Got block: #411 time: 2022-09-29T09:47:24 latency: 14 ms from: init4  irreversible: 403 (-8)
peerplays03_1                     | 2844015ms th_a       application.cpp:429           handle_block         ] Got block: #411 time: 2022-09-29T09:47:24 latency: 15 ms from: init4  irreversible: 403 (-8)
peerplays08_1                     | 2844015ms th_a       application.cpp:429           handle_block         ] Got block: #411 time: 2022-09-29T09:47:24 latency: 15 ms from: init4  irreversible: 403 (-8)
peerplays14_1                     | 2844016ms th_a       application.cpp:429           handle_block         ] Got block: #411 time: 2022-09-29T09:47:24 latency: 16 ms from: init4  irreversible: 403 (-8)
peerplays06_1                     | 2844017ms th_a       application.cpp:429           handle_block         ] Got block: #411 time: 2022-09-29T09:47:24 latency: 16 ms from: init4  irreversible: 403 (-8)
peerplays02_1                     | 2844017ms th_a       application.cpp:429           handle_block         ] Got block: #411 time: 2022-09-29T09:47:24 latency: 17 ms from: init4  irreversible: 403 (-8)
peerplays12_1                     | 2844017ms th_a       application.cpp:429           handle_block         ] Got block: #411 time: 2022-09-29T09:47:24 latency: 17 ms from: init4  irreversible: 403 (-8)
peerplays11_1                     | 2844019ms th_a       application.cpp:429           handle_block         ] Got block: #411 time: 2022-09-29T09:47:24 latency: 17 ms from: init4  irreversible: 403 (-8)
peerplays04_1                     | 2844021ms th_a       application.cpp:429           handle_block         ] Got block: #411 time: 2022-09-29T09:47:24 latency: 21 ms from: init4  irreversible: 403 (-8)
peerplays15_1                     | 2844021ms th_a       application.cpp:429           handle_block         ] Got block: #411 time: 2022-09-29T09:47:24 latency: 21 ms from: init4  irreversible: 403 (-8)
peerplays16_1                     | 2844022ms th_a       application.cpp:429           handle_block         ] Got block: #411 time: 2022-09-29T09:47:24 latency: 22 ms from: init4  irreversible: 403 (-8)

- Run QA Environment with all 16 Peerplays nodes `docker-compose up peerplays01 peerplays02 peerplays03 peerplays04 peerplays05 peerplays06 peerplays07 peerplays08 peerplays09 peerplays10 peerplays11` - Check the log, it is working normally ``` peerplays10_1 | 1638003ms th_a witness.cpp:191 block_production_loo ] Generated block #30 with timestamp 2022-09-29T09:27:18 at time 2022-09-29T09:27:18 peerplays12_1 | 1638011ms th_a application.cpp:429 handle_block ] Got block: #30 time: 2022-09-29T09:27:18 latency: 11 ms from: init9 irreversible: 22 (-8) peerplays09_1 | 1638011ms th_a application.cpp:429 handle_block ] Got block: #30 time: 2022-09-29T09:27:18 latency: 11 ms from: init9 irreversible: 22 (-8) peerplays02_1 | 1638011ms th_a application.cpp:429 handle_block ] Got block: #30 time: 2022-09-29T09:27:18 latency: 11 ms from: init9 irreversible: 22 (-8) peerplays05_1 | 1638011ms th_a application.cpp:429 handle_block ] Got block: #30 time: 2022-09-29T09:27:18 latency: 11 ms from: init9 irreversible: 22 (-8) peerplays06_1 | 1638014ms th_a application.cpp:429 handle_block ] Got block: #30 time: 2022-09-29T09:27:18 latency: 14 ms from: init9 irreversible: 22 (-8) peerplays15_1 | 1638016ms th_a application.cpp:429 handle_block ] Got block: #30 time: 2022-09-29T09:27:18 latency: 16 ms from: init9 irreversible: 22 (-8) peerplays04_1 | 1638016ms th_a application.cpp:429 handle_block ] Got block: #30 time: 2022-09-29T09:27:18 latency: 16 ms from: init9 irreversible: 22 (-8) peerplays08_1 | 1638017ms th_a application.cpp:429 handle_block ] Got block: #30 time: 2022-09-29T09:27:18 latency: 17 ms from: init9 irreversible: 22 (-8) peerplays13_1 | 1638017ms th_a application.cpp:429 handle_block ] Got block: #30 time: 2022-09-29T09:27:18 latency: 17 ms from: init9 irreversible: 22 (-8) peerplays03_1 | 1638018ms th_a application.cpp:429 handle_block ] Got block: #30 time: 2022-09-29T09:27:18 latency: 18 ms from: init9 irreversible: 22 (-8) peerplays01_1 | 1638019ms th_a application.cpp:429 handle_block ] Got block: #30 time: 2022-09-29T09:27:18 latency: 19 ms from: init9 irreversible: 22 (-8) peerplays14_1 | 1638020ms th_a application.cpp:429 handle_block ] Got block: #30 time: 2022-09-29T09:27:18 latency: 20 ms from: init9 irreversible: 22 (-8) peerplays07_1 | 1638027ms th_a application.cpp:429 handle_block ] Got block: #30 time: 2022-09-29T09:27:18 latency: 27 ms from: init9 irreversible: 22 (-8) peerplays11_1 | 1638027ms th_a application.cpp:429 handle_block ] Got block: #30 time: 2022-09-29T09:27:18 latency: 27 ms from: init9 irreversible: 22 (-8) peerplays16_1 | 1638029ms th_a application.cpp:429 handle_block ] Got block: #30 time: 2022-09-29T09:27:18 latency: 29 ms from: init9 irreversible: 22 (-8) peerplays03_1 | 1641000ms th_a witness.cpp:267 maybe_produce_block ] slot: 1 scheduled_witness: 1.6.3 scheduled_time: 2022-09-29T09:27:21 now: 2022-09-29T09:27:21 peerplays03_1 | 1641002ms th_a witness.cpp:191 block_production_loo ] Generated block #31 with timestamp 2022-09-29T09:27:21 at time 2022-09-29T09:27:21 ``` - Create a large file on /var and make the space 15MB `fallocate -l 7.77G test2.img` - Check the log, the error message `Rejecting block due to low disk space` is there ``` peerplays09_1 | (si.available) > 104857600: Rejecting block due to low disk space peerplays09_1 | {} peerplays09_1 | th_a db_block.cpp:202 _push_block peerplays09_1 | peerplays09_1 | {"new_block":{"previous":"0000013d8048ed635f5508b5e7777d3178398c07","timestamp":"2022-09-29T09:42:30","witness":"1.6.9","next_secret_hash":"219cedb62d7ed191979f4677ef92f6ca45111b85","previous_secret":"092deea00c65151f27d7fa5e08e6ec397e2f95fc","transaction_merkle_root":"0000000000000000000000000000000000000000","extensions":[],"witness_signature":"20181bffcd048d8fa4da35ec2c14eb8f11c54081d6acf42fddafd193d21638bb015e16ba6d9289f73799ca75d76a13c652658180e0fa8a6fc8dc3cdced33769e24","transactions":[]}} peerplays09_1 | th_a db_block.cpp:298 _push_block peerplays09_1 | peerplays09_1 | {"witness_id":"1.6.9"} peerplays09_1 | th_a db_block.cpp:592 _generate_block peerplays09_1 | peerplays09_1 | {} peerplays09_1 | th_a db_block.cpp:475 generate_block peerplays09_1 | 2550001ms th_a witness.cpp:184 block_production_loo ] Discarding all pending transactions in an attempt to prevent the same error from occurring the next time we try to produce a block peerplays09_1 | 2550001ms th_a witness.cpp:216 block_production_loo ] exception producing block peerplays09_1 | 2551000ms th_a witness.cpp:267 maybe_produce_block ] slot: 11 scheduled_witness: 1.6.9 scheduled_time: 2022-09-29T09:42:30 now: 2022-09-29T09:42:31 peerplays09_1 | 2551001ms th_a witness.cpp:182 block_production_loo ] Got exception while generating block: peerplays09_1 | 10 assert_exception: Assert Exception peerplays09_1 | (si.available) > 104857600: Rejecting block due to low disk space peerplays09_1 | {} peerplays09_1 | th_a db_block.cpp:202 _push_block peerplays09_1 | peerplays09_1 | {"new_block":{"previous":"0000013d8048ed635f5508b5e7777d3178398c07","timestamp":"2022-09-29T09:42:30","witness":"1.6.9","next_secret_hash":"219cedb62d7ed191979f4677ef92f6ca45111b85","previous_secret":"092deea00c65151f27d7fa5e08e6ec397e2f95fc","transaction_merkle_root":"0000000000000000000000000000000000000000","extensions":[],"witness_signature":"20181bffcd048d8fa4da35ec2c14eb8f11c54081d6acf42fddafd193d21638bb015e16ba6d9289f73799ca75d76a13c652658180e0fa8a6fc8dc3cdced33769e24","transactions":[]}} peerplays09_1 | th_a db_block.cpp:298 _push_block peerplays09_1 | peerplays09_1 | {"witness_id":"1.6.9"} peerplays09_1 | th_a db_block.cpp:592 _generate_block peerplays09_1 | peerplays09_1 | {} peerplays09_1 | th_a db_block.cpp:475 generate_block peerplays09_1 | 2551001ms th_a witness.cpp:184 block_production_loo ] Discarding all pending transactions in an attempt to prevent the same error from occurring the next time we try to produce a block peerplays09_1 | 2551001ms th_a witness.cpp:216 block_production_loo ] exception producing block peerplays09_1 | 2552000ms th_a witness.cpp:267 maybe_produce_block ] slot: 11 scheduled_witness: 1.6.9 scheduled_time: 2022-09-29T09:42:30 now: 2022-09-29T09:42:32 peerplays09_1 | 2552001ms th_a witness.cpp:182 block_production_loo ] Got exception while generating block: peerplays09_1 | 10 assert_exception: Assert Exception peerplays09_1 | (si.available) > 104857600: Rejecting block due to low disk space ``` - Remove test2.img file on /var (Make more space on /var) and check the log again, and make sure it is back to normal and generating blocks again ``` peerplays05_1 | 2844004ms th_a witness.cpp:191 block_production_loo ] Generated block #411 with timestamp 2022-09-29T09:47:24 at time 2022-09-29T09:47:24 peerplays07_1 | 2844011ms th_a application.cpp:429 handle_block ] Got block: #411 time: 2022-09-29T09:47:24 latency: 11 ms from: init4 irreversible: 403 (-8) peerplays09_1 | 2844011ms th_a application.cpp:429 handle_block ] Got block: #411 time: 2022-09-29T09:47:24 latency: 11 ms from: init4 irreversible: 403 (-8) peerplays10_1 | 2844011ms th_a application.cpp:429 handle_block ] Got block: #411 time: 2022-09-29T09:47:24 latency: 11 ms from: init4 irreversible: 403 (-8) peerplays13_1 | 2844012ms th_a application.cpp:429 handle_block ] Got block: #411 time: 2022-09-29T09:47:24 latency: 12 ms from: init4 irreversible: 403 (-8) peerplays01_1 | 2844014ms th_a application.cpp:429 handle_block ] Got block: #411 time: 2022-09-29T09:47:24 latency: 14 ms from: init4 irreversible: 403 (-8) peerplays03_1 | 2844015ms th_a application.cpp:429 handle_block ] Got block: #411 time: 2022-09-29T09:47:24 latency: 15 ms from: init4 irreversible: 403 (-8) peerplays08_1 | 2844015ms th_a application.cpp:429 handle_block ] Got block: #411 time: 2022-09-29T09:47:24 latency: 15 ms from: init4 irreversible: 403 (-8) peerplays14_1 | 2844016ms th_a application.cpp:429 handle_block ] Got block: #411 time: 2022-09-29T09:47:24 latency: 16 ms from: init4 irreversible: 403 (-8) peerplays06_1 | 2844017ms th_a application.cpp:429 handle_block ] Got block: #411 time: 2022-09-29T09:47:24 latency: 16 ms from: init4 irreversible: 403 (-8) peerplays02_1 | 2844017ms th_a application.cpp:429 handle_block ] Got block: #411 time: 2022-09-29T09:47:24 latency: 17 ms from: init4 irreversible: 403 (-8) peerplays12_1 | 2844017ms th_a application.cpp:429 handle_block ] Got block: #411 time: 2022-09-29T09:47:24 latency: 17 ms from: init4 irreversible: 403 (-8) peerplays11_1 | 2844019ms th_a application.cpp:429 handle_block ] Got block: #411 time: 2022-09-29T09:47:24 latency: 17 ms from: init4 irreversible: 403 (-8) peerplays04_1 | 2844021ms th_a application.cpp:429 handle_block ] Got block: #411 time: 2022-09-29T09:47:24 latency: 21 ms from: init4 irreversible: 403 (-8) peerplays15_1 | 2844021ms th_a application.cpp:429 handle_block ] Got block: #411 time: 2022-09-29T09:47:24 latency: 21 ms from: init4 irreversible: 403 (-8) peerplays16_1 | 2844022ms th_a application.cpp:429 handle_block ] Got block: #411 time: 2022-09-29T09:47:24 latency: 22 ms from: init4 irreversible: 403 (-8) ```

wsalloum (Migrated from gitlab.com) closed this issue

2022-09-29 09:58:41 +00:00