witness_node program is crashing on active SON node (Mainnet)

Hiltos1 commented

2021-08-18 14:09:16 +00:00

(Migrated from gitlab.com)

Description:

I have an active SON on Mainnet. After setting up the server, installing/configuring Peerplays and a Bitcoin node, the witness_node program syncs all the blocks up to current. Then when my SON is scheduled, it tries to create "son down" proposals for all the other SONs. Most of the time these new proposals fail. Sometimes the proposals succeed in being created. The SON will operate for a few minutes and then get suddenly disconnected/crash (I can't exactly tell what's happening). Either way, the witness_node program ends.

The first problem I found was that my server time was off. I was getting a p2p disconnect because I was proposing blocks with a timestamp that was "in the future". I installed and configured NTP which solved that problem. But now it's crashing for some other reason.

Relevant info:

Server details

Amazon AWS EC2 instance (r6g.large)
- 2 CPUs, 16 GB RAM, 150 GB storage, 10 Gigabit network
- arm64 architecture
- static public IP
Inbound ports for P2P, RPC, HTTP, HTTPS, and the Bitcoin node are open.
Outbound ports are open for all outbound traffic.

Software details

Peerplays version 1.5.13
- built on the machine from source
Bitcoin node (Bitcoin Core) version 0.21.1
- downloaded & extracted .tar.gz from bitcoin.org

Configuration details

Peerplays is set up as a seed node (opened p2p), API node (opened RPC), and SON node
Peerplays private key in config.ini is the SON account's active private key
SON's signing key is the SON account's active key
son-wallet is set up with a bitcoin pubkey and private key

Logs

Here's the end of my default.log when the program ends:

2021-08-18T02:24:42.400 th_a:invoke handle_block         handle_block ] Got block: #42704062 time: 2021-08-18T02:24:42 latency: 400 ms from: jburr-blocks  irreversible: 42704052 (-10)                 application.cpp:495
2021-08-18T02:24:42.901   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-18T02:24:43                     peerplays_sidechain_plugin.cpp:398
2021-08-18T02:24:45.381 th_a:invoke handle_block         handle_block ] Got block: #42704063 time: 2021-08-18T02:24:45 latency: 381 ms from: robert-hedler  irreversible: 42704053 (-10)                        application.cpp:495
2021-08-18T02:24:45.882   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.4 Now: 2021-08-18T02:24:46                     peerplays_sidechain_plugin.cpp:398
2021-08-18T02:24:48.518 th_a:invoke handle_block         handle_block ] Got block: #42704064 time: 2021-08-18T02:24:48 latency: 518 ms from: eifos-witness  irreversible: 42704056 (-8)                 application.cpp:495
2021-08-18T02:24:49.019   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.0 Now: 2021-08-18T02:24:49                     peerplays_sidechain_plugin.cpp:398
2021-08-18T02:24:54.401 th_a:invoke handle_block         handle_block ] Got block: #42704065 time: 2021-08-18T02:24:54 latency: 401 ms from: hiltos-witness  irreversible: 42704057 (-8)                        application.cpp:495
2021-08-18T02:24:54.902   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.9 Now: 2021-08-18T02:24:55                     peerplays_sidechain_plugin.cpp:398
2021-08-18T02:24:57.452 th_a:invoke handle_block         handle_block ] Got block: #42704066 time: 2021-08-18T02:24:57 latency: 452 ms from: royal-flush  irreversible: 42704058 (-8)                   application.cpp:495
2021-08-18T02:24:57.952   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.7 Now: 2021-08-18T02:24:58                     peerplays_sidechain_plugin.cpp:398
2021-08-18T02:24:57.953   th_a:SON Processing create_son_down_prop ] Sending son down proposal for 1.33.0 from 1.33.7                   peerplays_sidechain_plugin.cpp:558
2021-08-18T02:24:57.953   th_a:SON Processing create_son_down_prop ] Sending son down proposal for 1.33.1 from 1.33.7                   peerplays_sidechain_plugin.cpp:558
2021-08-18T02:24:57.954   th_a:SON Processing create_son_down_prop ] Sending son down proposal for 1.33.2 from 1.33.7                   peerplays_sidechain_plugin.cpp:558
2021-08-18T02:24:57.955   th_a:SON Processing create_son_down_prop ] Sending son down proposal for 1.33.3 from 1.33.7                   peerplays_sidechain_plugin.cpp:558
2021-08-18T02:24:57.955   th_a:SON Processing create_son_down_prop ] Sending son down proposal for 1.33.4 from 1.33.7                   peerplays_sidechain_plugin.cpp:558
2021-08-18T02:24:57.956   th_a:SON Processing create_son_down_prop ] Sending son down proposal for 1.33.9 from 1.33.7                   peerplays_sidechain_plugin.cpp:558

Here's the end of my p2p.log:

2021-08-18T02:24:39.208 p2p:message read_loop on_connection_accept ] I don't know if I'm firewalled.  Sending a firewall check message to peer 104.225.216.198:9779                     node.cpp:2121
2021-08-18T02:24:39.245   p2p:connect_to_task           connect_to ] fatal: error connecting to peer 173.249.23.108:9777: 0 exception: unspecified
Connection refused
    {"message":"Connection refused"}
    asio  asio.cpp:60 error_handler                     peer_connection.cpp:264
2021-08-18T02:24:39.247 p2p:delayed_peer_deletion_task              destroy ] Unexpected exception from peer_connection's accept_or_connect_task : {"code":0,"name":"exception","message":"unspecified","stack":[{"context":{"level":"error","file":"asio.cpp","line":60,"method":"error_handler","hostname":"","thread_name":"asio","timestamp":"2021-08-18T02:24:39"},"format":"${message} ","data":{"message":"Connection refused"}}]}                       peer_connection.cpp:167
2021-08-18T02:24:39.252   p2p:connect_to_task           connect_to ] fatal: error connecting to peer 109.111.245.114:9777: 0 exception: unspecified
No route to host
    {"message":"No route to host"}
    asio  asio.cpp:60 error_handler                     peer_connection.cpp:264
2021-08-18T02:24:39.252 p2p:delayed_peer_deletion_task              destroy ] Unexpected exception from peer_connection's accept_or_connect_task : {"code":0,"name":"exception","message":"unspecified","stack":[{"context":{"level":"error","file":"asio.cpp","line":60,"method":"error_handler","hostname":"","thread_name":"asio","timestamp":"2021-08-18T02:24:39"},"format":"${message} ","data":{"message":"No route to host"}}]}                 peer_connection.cpp:167
2021-08-18T02:24:39.573 p2p:message read_loop            read_loop ] disconnected 11 eof_exception: End Of File
End of file
    {"message":"End of file"}
    asio  asio.cpp:39 operator()

    {"len":16}
    p2p  stcp_socket.cpp:117 readsome                   message_oriented_connection.cpp:218
2021-08-18T02:24:39.609 p2p:message read_loop on_check_firewall_re ] Firewall check we initiated has returned with result: connection_successful, endpoint = 18.116.153.207:9777                        node.cpp:3774
2021-08-18T02:24:45.132 p2p:terminate_inactive_connections_loop terminate_inactive_c ] Forcibly disconnecting from handshaking peer 88.198.13.39:9777 due to inactivity of at least 5 seconds                   node.cpp:1345
2021-08-18T02:24:45.132 p2p:terminate_inactive_connections_loop terminate_inactive_c ] Peer's negotiating status: connecting, bytes sent: 0, bytes received: 0                  node.cpp:1347
2021-08-18T02:24:45.132 p2p:terminate_inactive_connections_loop terminate_inactive_c ] Forcibly disconnecting from handshaking peer 172.17.0.2:39319 due to inactivity of at least 5 seconds                    node.cpp:1345
2021-08-18T02:24:45.132 p2p:terminate_inactive_connections_loop terminate_inactive_c ] Peer's negotiating status: connecting, bytes sent: 0, bytes received: 0                  node.cpp:1347
2021-08-18T02:24:45.133   p2p:connect_to_task           connect_to ] fatal: error connecting to peer 88.198.13.39:9777: 0 exception: unspecified
Operation canceled
    {"message":"Operation canceled"}
    asio  asio.cpp:60 error_handler                     peer_connection.cpp:264
2021-08-18T02:24:45.133 p2p:delayed_peer_deletion_task              destroy ] Unexpected exception from peer_connection's accept_or_connect_task : {"code":0,"name":"exception","message":"unspecified","stack":[{"context":{"level":"error","file":"asio.cpp","line":60,"method":"error_handler","hostname":"","thread_name":"asio","timestamp":"2021-08-18T02:24:45"},"format":"${message} ","data":{"message":"Operation canceled"}}]}                       peer_connection.cpp:167
2021-08-18T02:24:45.133   p2p:connect_to_task           connect_to ] fatal: error connecting to peer 172.17.0.2:39319: 0 exception: unspecified
Operation canceled
    {"message":"Operation canceled"}
    asio  asio.cpp:60 error_handler                     peer_connection.cpp:264
2021-08-18T02:24:45.133 p2p:delayed_peer_deletion_task              destroy ] Unexpected exception from peer_connection's accept_or_connect_task : {"code":0,"name":"exception","message":"unspecified","stack":[{"context":{"level":"error","file":"asio.cpp","line":60,"method":"error_handler","hostname":"","thread_name":"asio","timestamp":"2021-08-18T02:24:45"},"format":"${message} ","data":{"message":"Operation canceled"}}]}                       peer_connection.cpp:167
2021-08-18T02:24:55.133 p2p:terminate_inactive_connections_loop terminate_inactive_c ] Forcibly disconnecting from handshaking peer 95.216.8.243:42765 due to inactivity of at least 5 seconds                  node.cpp:1345
2021-08-18T02:24:55.133 p2p:terminate_inactive_connections_loop terminate_inactive_c ] Peer's negotiating status: connecting, bytes sent: 0, bytes received: 0                  node.cpp:1347
2021-08-18T02:24:55.133 p2p:terminate_inactive_connections_loop terminate_inactive_c ] Forcibly disconnecting from handshaking peer 195.154.83.79:3338 due to inactivity of at least 5 seconds                  node.cpp:1345
2021-08-18T02:24:55.133 p2p:terminate_inactive_connections_loop terminate_inactive_c ] Peer's negotiating status: connecting, bytes sent: 0, bytes received: 0                  node.cpp:1347
2021-08-18T02:24:55.133   p2p:connect_to_task           connect_to ] fatal: error connecting to peer 95.216.8.243:42765: 0 exception: unspecified
Operation canceled
    {"message":"Operation canceled"}
    asio  asio.cpp:60 error_handler                     peer_connection.cpp:264
2021-08-18T02:24:55.133   p2p:connect_to_task           connect_to ] fatal: error connecting to peer 195.154.83.79:3338: 0 exception: unspecified
Operation canceled
    {"message":"Operation canceled"}
    asio  asio.cpp:60 error_handler                     peer_connection.cpp:264
2021-08-18T02:24:55.133 p2p:delayed_peer_deletion_task              destroy ] Unexpected exception from peer_connection's accept_or_connect_task : {"code":0,"name":"exception","message":"unspecified","stack":[{"context":{"level":"error","file":"asio.cpp","line":60,"method":"error_handler","hostname":"","thread_name":"asio","timestamp":"2021-08-18T02:24:55"},"format":"${message} ","data":{"message":"Operation canceled"}}]}                       peer_connection.cpp:167
2021-08-18T02:24:55.133 p2p:delayed_peer_deletion_task              destroy ] Unexpected exception from peer_connection's accept_or_connect_task : {"code":0,"name":"exception","message":"unspecified","stack":[{"context":{"level":"error","file":"asio.cpp","line":60,"method":"error_handler","hostname":"","thread_name":"asio","timestamp":"2021-08-18T02:24:55"},"format":"${message} ","data":{"message":"Operation canceled"}}]}                       peer_connection.cpp:167

There are no entries in the rpc.log.

Expected results:

The witness_node program should continue to run. I don't think the SON should be producing "son down" proposals for the other SONs unless they truly are down.

Actual results:

The witness_node program ends after attempting to create/update proposals.

## Description: I have an active SON on Mainnet. After setting up the server, installing/configuring Peerplays and a Bitcoin node, the witness_node program syncs all the blocks up to current. Then when my SON is scheduled, it tries to create "son down" proposals for all the other SONs. Most of the time these new proposals fail. Sometimes the proposals succeed in being created. The SON will operate for a few minutes and then get suddenly disconnected/crash (I can't exactly tell what's happening). Either way, the witness_node program ends. The first problem I found was that my server time was off. I was getting a p2p disconnect because I was proposing blocks with a timestamp that was "in the future". I installed and configured NTP which solved that problem. But now it's crashing for some other reason. ## Relevant info: ### Server details - Amazon AWS EC2 instance (r6g.large) - 2 CPUs, 16 GB RAM, 150 GB storage, 10 Gigabit network - arm64 architecture - static public IP - Inbound ports for P2P, RPC, HTTP, HTTPS, and the Bitcoin node are open. - Outbound ports are open for all outbound traffic. ### Software details - Peerplays version 1.5.13 - built on the machine from source - Bitcoin node (Bitcoin Core) version 0.21.1 - downloaded & extracted .tar.gz from bitcoin.org ### Configuration details - Peerplays is set up as a seed node (opened p2p), API node (opened RPC), and SON node - Peerplays private key in config.ini is the SON account's active private key - SON's signing key is the SON account's active key - son-wallet is set up with a bitcoin pubkey and private key ### Logs Here's the end of my default.log when the program ends: ``` 2021-08-18T02:24:42.400 th_a:invoke handle_block handle_block ] Got block: #42704062 time: 2021-08-18T02:24:42 latency: 400 ms from: jburr-blocks irreversible: 42704052 (-10) application.cpp:495 2021-08-18T02:24:42.901 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-18T02:24:43 peerplays_sidechain_plugin.cpp:398 2021-08-18T02:24:45.381 th_a:invoke handle_block handle_block ] Got block: #42704063 time: 2021-08-18T02:24:45 latency: 381 ms from: robert-hedler irreversible: 42704053 (-10) application.cpp:495 2021-08-18T02:24:45.882 th_a:SON Processing son_processing ] Scheduled SON: 1.33.4 Now: 2021-08-18T02:24:46 peerplays_sidechain_plugin.cpp:398 2021-08-18T02:24:48.518 th_a:invoke handle_block handle_block ] Got block: #42704064 time: 2021-08-18T02:24:48 latency: 518 ms from: eifos-witness irreversible: 42704056 (-8) application.cpp:495 2021-08-18T02:24:49.019 th_a:SON Processing son_processing ] Scheduled SON: 1.33.0 Now: 2021-08-18T02:24:49 peerplays_sidechain_plugin.cpp:398 2021-08-18T02:24:54.401 th_a:invoke handle_block handle_block ] Got block: #42704065 time: 2021-08-18T02:24:54 latency: 401 ms from: hiltos-witness irreversible: 42704057 (-8) application.cpp:495 2021-08-18T02:24:54.902 th_a:SON Processing son_processing ] Scheduled SON: 1.33.9 Now: 2021-08-18T02:24:55 peerplays_sidechain_plugin.cpp:398 2021-08-18T02:24:57.452 th_a:invoke handle_block handle_block ] Got block: #42704066 time: 2021-08-18T02:24:57 latency: 452 ms from: royal-flush irreversible: 42704058 (-8) application.cpp:495 2021-08-18T02:24:57.952 th_a:SON Processing son_processing ] Scheduled SON: 1.33.7 Now: 2021-08-18T02:24:58 peerplays_sidechain_plugin.cpp:398 2021-08-18T02:24:57.953 th_a:SON Processing create_son_down_prop ] Sending son down proposal for 1.33.0 from 1.33.7 peerplays_sidechain_plugin.cpp:558 2021-08-18T02:24:57.953 th_a:SON Processing create_son_down_prop ] Sending son down proposal for 1.33.1 from 1.33.7 peerplays_sidechain_plugin.cpp:558 2021-08-18T02:24:57.954 th_a:SON Processing create_son_down_prop ] Sending son down proposal for 1.33.2 from 1.33.7 peerplays_sidechain_plugin.cpp:558 2021-08-18T02:24:57.955 th_a:SON Processing create_son_down_prop ] Sending son down proposal for 1.33.3 from 1.33.7 peerplays_sidechain_plugin.cpp:558 2021-08-18T02:24:57.955 th_a:SON Processing create_son_down_prop ] Sending son down proposal for 1.33.4 from 1.33.7 peerplays_sidechain_plugin.cpp:558 2021-08-18T02:24:57.956 th_a:SON Processing create_son_down_prop ] Sending son down proposal for 1.33.9 from 1.33.7 peerplays_sidechain_plugin.cpp:558 ``` Here's the end of my p2p.log: ``` 2021-08-18T02:24:39.208 p2p:message read_loop on_connection_accept ] I don't know if I'm firewalled. Sending a firewall check message to peer 104.225.216.198:9779 node.cpp:2121 2021-08-18T02:24:39.245 p2p:connect_to_task connect_to ] fatal: error connecting to peer 173.249.23.108:9777: 0 exception: unspecified Connection refused {"message":"Connection refused"} asio asio.cpp:60 error_handler peer_connection.cpp:264 2021-08-18T02:24:39.247 p2p:delayed_peer_deletion_task destroy ] Unexpected exception from peer_connection's accept_or_connect_task : {"code":0,"name":"exception","message":"unspecified","stack":[{"context":{"level":"error","file":"asio.cpp","line":60,"method":"error_handler","hostname":"","thread_name":"asio","timestamp":"2021-08-18T02:24:39"},"format":"${message} ","data":{"message":"Connection refused"}}]} peer_connection.cpp:167 2021-08-18T02:24:39.252 p2p:connect_to_task connect_to ] fatal: error connecting to peer 109.111.245.114:9777: 0 exception: unspecified No route to host {"message":"No route to host"} asio asio.cpp:60 error_handler peer_connection.cpp:264 2021-08-18T02:24:39.252 p2p:delayed_peer_deletion_task destroy ] Unexpected exception from peer_connection's accept_or_connect_task : {"code":0,"name":"exception","message":"unspecified","stack":[{"context":{"level":"error","file":"asio.cpp","line":60,"method":"error_handler","hostname":"","thread_name":"asio","timestamp":"2021-08-18T02:24:39"},"format":"${message} ","data":{"message":"No route to host"}}]} peer_connection.cpp:167 2021-08-18T02:24:39.573 p2p:message read_loop read_loop ] disconnected 11 eof_exception: End Of File End of file {"message":"End of file"} asio asio.cpp:39 operator() {"len":16} p2p stcp_socket.cpp:117 readsome message_oriented_connection.cpp:218 2021-08-18T02:24:39.609 p2p:message read_loop on_check_firewall_re ] Firewall check we initiated has returned with result: connection_successful, endpoint = 18.116.153.207:9777 node.cpp:3774 2021-08-18T02:24:45.132 p2p:terminate_inactive_connections_loop terminate_inactive_c ] Forcibly disconnecting from handshaking peer 88.198.13.39:9777 due to inactivity of at least 5 seconds node.cpp:1345 2021-08-18T02:24:45.132 p2p:terminate_inactive_connections_loop terminate_inactive_c ] Peer's negotiating status: connecting, bytes sent: 0, bytes received: 0 node.cpp:1347 2021-08-18T02:24:45.132 p2p:terminate_inactive_connections_loop terminate_inactive_c ] Forcibly disconnecting from handshaking peer 172.17.0.2:39319 due to inactivity of at least 5 seconds node.cpp:1345 2021-08-18T02:24:45.132 p2p:terminate_inactive_connections_loop terminate_inactive_c ] Peer's negotiating status: connecting, bytes sent: 0, bytes received: 0 node.cpp:1347 2021-08-18T02:24:45.133 p2p:connect_to_task connect_to ] fatal: error connecting to peer 88.198.13.39:9777: 0 exception: unspecified Operation canceled {"message":"Operation canceled"} asio asio.cpp:60 error_handler peer_connection.cpp:264 2021-08-18T02:24:45.133 p2p:delayed_peer_deletion_task destroy ] Unexpected exception from peer_connection's accept_or_connect_task : {"code":0,"name":"exception","message":"unspecified","stack":[{"context":{"level":"error","file":"asio.cpp","line":60,"method":"error_handler","hostname":"","thread_name":"asio","timestamp":"2021-08-18T02:24:45"},"format":"${message} ","data":{"message":"Operation canceled"}}]} peer_connection.cpp:167 2021-08-18T02:24:45.133 p2p:connect_to_task connect_to ] fatal: error connecting to peer 172.17.0.2:39319: 0 exception: unspecified Operation canceled {"message":"Operation canceled"} asio asio.cpp:60 error_handler peer_connection.cpp:264 2021-08-18T02:24:45.133 p2p:delayed_peer_deletion_task destroy ] Unexpected exception from peer_connection's accept_or_connect_task : {"code":0,"name":"exception","message":"unspecified","stack":[{"context":{"level":"error","file":"asio.cpp","line":60,"method":"error_handler","hostname":"","thread_name":"asio","timestamp":"2021-08-18T02:24:45"},"format":"${message} ","data":{"message":"Operation canceled"}}]} peer_connection.cpp:167 2021-08-18T02:24:55.133 p2p:terminate_inactive_connections_loop terminate_inactive_c ] Forcibly disconnecting from handshaking peer 95.216.8.243:42765 due to inactivity of at least 5 seconds node.cpp:1345 2021-08-18T02:24:55.133 p2p:terminate_inactive_connections_loop terminate_inactive_c ] Peer's negotiating status: connecting, bytes sent: 0, bytes received: 0 node.cpp:1347 2021-08-18T02:24:55.133 p2p:terminate_inactive_connections_loop terminate_inactive_c ] Forcibly disconnecting from handshaking peer 195.154.83.79:3338 due to inactivity of at least 5 seconds node.cpp:1345 2021-08-18T02:24:55.133 p2p:terminate_inactive_connections_loop terminate_inactive_c ] Peer's negotiating status: connecting, bytes sent: 0, bytes received: 0 node.cpp:1347 2021-08-18T02:24:55.133 p2p:connect_to_task connect_to ] fatal: error connecting to peer 95.216.8.243:42765: 0 exception: unspecified Operation canceled {"message":"Operation canceled"} asio asio.cpp:60 error_handler peer_connection.cpp:264 2021-08-18T02:24:55.133 p2p:connect_to_task connect_to ] fatal: error connecting to peer 195.154.83.79:3338: 0 exception: unspecified Operation canceled {"message":"Operation canceled"} asio asio.cpp:60 error_handler peer_connection.cpp:264 2021-08-18T02:24:55.133 p2p:delayed_peer_deletion_task destroy ] Unexpected exception from peer_connection's accept_or_connect_task : {"code":0,"name":"exception","message":"unspecified","stack":[{"context":{"level":"error","file":"asio.cpp","line":60,"method":"error_handler","hostname":"","thread_name":"asio","timestamp":"2021-08-18T02:24:55"},"format":"${message} ","data":{"message":"Operation canceled"}}]} peer_connection.cpp:167 2021-08-18T02:24:55.133 p2p:delayed_peer_deletion_task destroy ] Unexpected exception from peer_connection's accept_or_connect_task : {"code":0,"name":"exception","message":"unspecified","stack":[{"context":{"level":"error","file":"asio.cpp","line":60,"method":"error_handler","hostname":"","thread_name":"asio","timestamp":"2021-08-18T02:24:55"},"format":"${message} ","data":{"message":"Operation canceled"}}]} peer_connection.cpp:167 ``` There are no entries in the rpc.log. ## Expected results: The witness_node program should continue to run. I don't think the SON should be producing "son down" proposals for the other SONs unless they truly are down. ## Actual results: The witness_node program ends after attempting to create/update proposals.

Hiltos1 commented

2021-08-18 14:09:16 +00:00

(Migrated from gitlab.com)

assigned to @serkixenos and @sierra19XX

Hiltos1 commented

2021-08-18 14:10:22 +00:00

(Migrated from gitlab.com)

FYI @hbelakon, @bobinson

bobinson commented

2021-08-19 09:38:26 +00:00

(Migrated from gitlab.com)

This could be a duplicate of https://gitlab.com/PBSA/peerplays/-/issues/127

hbelakon commented

2021-08-23 12:47:41 +00:00

(Migrated from gitlab.com)

@sierra19XX please estimate! I will put 8h/Weight 1 for now- please change it if you feel it is more complex. T h a n k s

hbelakon commented

2021-08-23 12:47:52 +00:00

(Migrated from gitlab.com)

changed time estimate to 8h

Hiltos1 commented

2021-09-02 13:01:07 +00:00

(Migrated from gitlab.com)

Hi @serkixenos & @sierra19XX,

I tried using a fresh server. This time I used x86_64 architecture instead of ARM to rule out the issue being caused by ARM. Here's the server I used this time:

Server details

Amazon AWS EC2 instance (m5n.2xlarge)
- 8 CPUs, 32 GB RAM, 200 GB storage, 25 Gigabit network
- x86_64 architecture
- static public IP
Inbound ports for P2P, RPC, HTTP, HTTPS, and the Bitcoin node are open.
Outbound ports are open for all outbound traffic.

Everything else is the same as I set it up last time except that Bitcoin Core 0.21.1 was built for x86_64 this time. Unfortunately, the program still halts after all the blocks are synced and the SON tries to make a proposal. Here are the logs...

Logs

default.log

2021-08-28T12:34:56.943        th_a:?unnamed?       reset_p2p_node ] Adding seed node 77.105.15.199:9777                        application.cpp:150
2021-08-28T12:34:56.944        th_a:?unnamed?       reset_p2p_node ] Adding seed node 173.249.23.108:9777                       application.cpp:150
2021-08-28T12:34:56.945        th_a:?unnamed?       reset_p2p_node ] Adding seed node 193.31.27.181:7777                        application.cpp:150
2021-08-28T12:34:56.992        th_a:?unnamed?       reset_p2p_node ] Adding seed node 88.198.13.39:9777                 application.cpp:150
2021-08-28T12:34:56.994        th_a:?unnamed?       reset_p2p_node ] Configured p2p node to listen on 0.0.0.0:9777                      application.cpp:193
2021-08-28T12:34:56.996        th_a:?unnamed? reset_websocket_serv ] Configured websocket rpc to listen on 0.0.0.0:8090                 application.cpp:271
2021-08-28T12:34:56.997        th_a:?unnamed?       plugin_startup ] bookie plugin: plugin_startup()                    bookie_plugin.cpp:506
2021-08-28T12:34:56.997        th_a:?unnamed?       plugin_startup ] peerplays sidechain plugin:  plugin_startup() begin                        peerplays_sidechain_plugin.cpp:682
2021-08-28T12:34:56.997        th_a:?unnamed?       plugin_startup ] Starting 1 SON instances                   peerplays_sidechain_plugin.cpp:212
2021-08-28T12:34:57.000        th_a:?unnamed? sidechain_net_handle ] Bitcoin Public Key: 02f2be7c0784fdc2d379a3925e6408925c70f357e909c28ff6fb93e7f567b6af7f                     sidechain_net_handler_bitcoin.cpp:932
2021-08-28T12:34:57.035        th_a:?unnamed?           loadwallet ] Bitcoin RPC call loadwallet with body {"jsonrpc": "1.0", "id":"loadwallet", "method": "loadwallet", "params": ["son-wallet"] } failed with reply '{"result":null,"error":{"code":-4,"message":"Wallet file verification failed. Refusing to load database. Data file '/home/ubuntu/.bitcoin/son-wallet/wallet.dat' is already loaded."},"id":"loadwallet"}
'                       sidechain_net_handler_bitcoin.cpp:673
2021-08-28T12:34:58.165 th_a:invoke handle_block         handle_block ] Got block: #42990000 time: 2021-08-28T11:56:48 latency: 2290165 ms from: hiltos-witness  irreversible: 42989992 (-8)                    application.cpp:495
2021-08-28T12:34:58.180 th_a:invoke handle_block process_dividend_ass ] In process_dividend_assets time 2021-08-28T12:00:00                     db_maint.cpp:1745
2021-08-28T12:34:58.217 th_a:invoke handle_block schedule_pending_div ] payout_asset_type: 1.3.0 current_balance: 1 previous_balance: 0                         db_maint.cpp:1500
2021-08-28T12:34:58.217 th_a:invoke handle_block schedule_pending_div ] total_fee_per_asset_in_payout_asset: 0 dividend_data.options: ["ERROR_WHILE_CONVERTING_VALUE_TO_STRING"]                        db_maint.cpp:1540
2021-08-28T12:34:58.218 th_a:invoke handle_block schedule_pending_div ] payout_asset_type: 1.3.1 current_balance: 5 previous_balance: 0                         db_maint.cpp:1500
2021-08-28T12:34:58.218 th_a:invoke handle_block schedule_pending_div ] total_fee_per_asset_in_payout_asset: 0 dividend_data.options: ["ERROR_WHILE_CONVERTING_VALUE_TO_STRING"]                        db_maint.cpp:1540
2021-08-28T12:34:58.231 th_a:invoke handle_block   update_active_sons ] Active SONs set NOT CHANGED                     db_maint.cpp:751
2021-08-28T12:34:58.461        th_a:?unnamed?       plugin_startup ] Bitcoin sidechain handler running                  peerplays_sidechain_plugin.cpp:223
2021-08-28T12:34:58.461        th_a:?unnamed? sidechain_net_handle ] Peerplays Public Key: PPY6MkvgJqbTLE5e2j4ySFXDhy4G6Tx957qYDYWMGC5QVDNAyAWvJ                        sidechain_net_handler_peerplays.cpp:33
2021-08-28T12:34:58.461        th_a:?unnamed?       plugin_startup ] Peerplays sidechain handler running                        peerplays_sidechain_plugin.cpp:233
2021-08-28T12:34:58.461        th_a:?unnamed?       plugin_startup ] peerplays sidechain plugin:  plugin_startup() end                  peerplays_sidechain_plugin.cpp:684
2021-08-28T12:34:58.461        th_a:?unnamed?       plugin_startup ] witness plugin:  plugin_startup() begin                    witness.cpp:126
2021-08-28T12:34:58.463        th_a:?unnamed?       plugin_startup ] No witnesses configured! Please add witness IDs and private keys to configuration.                 witness.cpp:141
2021-08-28T12:34:58.463        th_a:?unnamed?       plugin_startup ] witness plugin:  plugin_startup() end                      witness.cpp:142
2021-08-28T12:34:58.463        th_a:?unnamed?                 main ] Started Peerplays node on a chain with 42990495 blocks.                    main.cpp:180
2021-08-28T12:34:58.463        th_a:?unnamed?                 main ] Chain ID is 6b6b5f0ce7a36d323768e534f3edb41c6d6332a541a95725b98e28d140850134                       main.cpp:181
2021-08-28T12:34:59.006   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.006   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.006   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.006   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.006   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.007   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.007   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.014   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.014   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.014   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.014   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.014   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.015   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.015   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.018   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.018   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.018   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.018   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.018   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.019   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.019   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.026   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.026   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.026   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.026   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.026   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.027   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.027   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.030   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.030   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.030   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.030   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398

... This goes on for 4 pages ...

2021-08-28T12:34:59.126   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.126   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.126   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.127   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:35:00.184 th_a:invoke handle_block         handle_block ] Got block: #42990688 time: 2021-08-28T12:35:00 latency: 184 ms from: houdini-witness  irreversible: 42990679 (-9)                       application.cpp:495
2021-08-28T12:35:00.685   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.1 Now: 2021-08-28T12:35:01                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:35:03.165 th_a:invoke handle_block         handle_block ] Got block: #42990689 time: 2021-08-28T12:35:03 latency: 165 ms from: eifos-witness  irreversible: 42990679 (-10)                        application.cpp:495
2021-08-28T12:35:03.665   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:35:04                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:35:06.354 th_a:invoke handle_block         handle_block ] Got block: #42990690 time: 2021-08-28T12:35:06 latency: 354 ms from: spacecrypt-witness  irreversible: 42990679 (-11)                   application.cpp:495
2021-08-28T12:35:06.855   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.7 Now: 2021-08-28T12:35:07                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:35:06.855   th_a:SON Processing create_son_down_prop ] Sending son down proposal for 1.33.0 from 1.33.7                   peerplays_sidechain_plugin.cpp:558
2021-08-28T12:35:06.855   th_a:SON Processing create_son_down_prop ] Sending son down proposal for 1.33.1 from 1.33.7                   peerplays_sidechain_plugin.cpp:558
2021-08-28T12:35:06.856   th_a:SON Processing create_son_down_prop ] Sending son down proposal for 1.33.2 from 1.33.7                   peerplays_sidechain_plugin.cpp:558
2021-08-28T12:35:06.856   th_a:SON Processing create_son_down_prop ] Sending son down proposal for 1.33.3 from 1.33.7                   peerplays_sidechain_plugin.cpp:558
2021-08-28T12:35:06.856   th_a:SON Processing create_son_down_prop ] Sending son down proposal for 1.33.4 from 1.33.7                   peerplays_sidechain_plugin.cpp:558
2021-08-28T12:35:06.857   th_a:SON Processing create_son_down_prop ] Sending son down proposal for 1.33.9 from 1.33.7                   peerplays_sidechain_plugin.cpp:558
(END)

p2p.log

2021-08-28T12:34:57.396 p2p:message read_loop on_connection_accept ] I don't know if I'm firewalled.  Sending a firewall check message to peer 88.198.13.39:9777                        node.cpp:2121
2021-08-28T12:34:58.182 p2p:message read_loop            read_loop ] disconnected 11 eof_exception: End Of File
End of file
    {"message":"End of file"}
    asio  asio.cpp:39 operator()

    {"len":16}
    p2p  stcp_socket.cpp:117 readsome                   message_oriented_connection.cpp:218
2021-08-28T12:34:58.373 p2p:message read_loop on_check_firewall_re ] Firewall check we initiated has returned with result: connection_successful, endpoint = 18.116.153.207:9777                        node.cpp:3774
2021-08-28T12:35:02.996 p2p:terminate_inactive_connections_loop terminate_inactive_c ] Forcibly disconnecting from handshaking peer 173.249.23.108:9777 due to inactivity of at least 5 seconds                 node.cpp:1345
2021-08-28T12:35:02.996 p2p:terminate_inactive_connections_loop terminate_inactive_c ] Peer's negotiating status: connecting, bytes sent: 0, bytes received: 0                  node.cpp:1347
2021-08-28T12:35:02.997   p2p:connect_to_task           connect_to ] fatal: error connecting to peer 173.249.23.108:9777: 0 exception: unspecified
Operation canceled
    {"message":"Operation canceled"}
    asio  asio.cpp:60 error_handler                     peer_connection.cpp:264
2021-08-28T12:35:02.998 p2p:delayed_peer_deletion_task              destroy ] Unexpected exception from peer_connection's accept_or_connect_task : ["ERROR_WHILE_CONVERTING_VALUE_TO_STRING"]                   peer_connection.cpp:167
(END)

The RPC log is blank.

Hi @serkixenos & @sierra19XX, I tried using a fresh server. This time I used x86_64 architecture instead of ARM to rule out the issue being caused by ARM. Here's the server I used this time: ### Server details * Amazon AWS EC2 instance (m5n.2xlarge) * 8 CPUs, 32 GB RAM, 200 GB storage, 25 Gigabit network * x86_64 architecture * static public IP * Inbound ports for P2P, RPC, HTTP, HTTPS, and the Bitcoin node are open. * Outbound ports are open for all outbound traffic. Everything else is the same as I set it up last time except that Bitcoin Core 0.21.1 was built for x86_64 this time. Unfortunately, the program still halts after all the blocks are synced and the SON tries to make a proposal. Here are the logs... ### Logs #### default.log ```text 2021-08-28T12:34:56.943 th_a:?unnamed? reset_p2p_node ] Adding seed node 77.105.15.199:9777 application.cpp:150 2021-08-28T12:34:56.944 th_a:?unnamed? reset_p2p_node ] Adding seed node 173.249.23.108:9777 application.cpp:150 2021-08-28T12:34:56.945 th_a:?unnamed? reset_p2p_node ] Adding seed node 193.31.27.181:7777 application.cpp:150 2021-08-28T12:34:56.992 th_a:?unnamed? reset_p2p_node ] Adding seed node 88.198.13.39:9777 application.cpp:150 2021-08-28T12:34:56.994 th_a:?unnamed? reset_p2p_node ] Configured p2p node to listen on 0.0.0.0:9777 application.cpp:193 2021-08-28T12:34:56.996 th_a:?unnamed? reset_websocket_serv ] Configured websocket rpc to listen on 0.0.0.0:8090 application.cpp:271 2021-08-28T12:34:56.997 th_a:?unnamed? plugin_startup ] bookie plugin: plugin_startup() bookie_plugin.cpp:506 2021-08-28T12:34:56.997 th_a:?unnamed? plugin_startup ] peerplays sidechain plugin: plugin_startup() begin peerplays_sidechain_plugin.cpp:682 2021-08-28T12:34:56.997 th_a:?unnamed? plugin_startup ] Starting 1 SON instances peerplays_sidechain_plugin.cpp:212 2021-08-28T12:34:57.000 th_a:?unnamed? sidechain_net_handle ] Bitcoin Public Key: 02f2be7c0784fdc2d379a3925e6408925c70f357e909c28ff6fb93e7f567b6af7f sidechain_net_handler_bitcoin.cpp:932 2021-08-28T12:34:57.035 th_a:?unnamed? loadwallet ] Bitcoin RPC call loadwallet with body {"jsonrpc": "1.0", "id":"loadwallet", "method": "loadwallet", "params": ["son-wallet"] } failed with reply '{"result":null,"error":{"code":-4,"message":"Wallet file verification failed. Refusing to load database. Data file '/home/ubuntu/.bitcoin/son-wallet/wallet.dat' is already loaded."},"id":"loadwallet"} ' sidechain_net_handler_bitcoin.cpp:673 2021-08-28T12:34:58.165 th_a:invoke handle_block handle_block ] Got block: #42990000 time: 2021-08-28T11:56:48 latency: 2290165 ms from: hiltos-witness irreversible: 42989992 (-8) application.cpp:495 2021-08-28T12:34:58.180 th_a:invoke handle_block process_dividend_ass ] In process_dividend_assets time 2021-08-28T12:00:00 db_maint.cpp:1745 2021-08-28T12:34:58.217 th_a:invoke handle_block schedule_pending_div ] payout_asset_type: 1.3.0 current_balance: 1 previous_balance: 0 db_maint.cpp:1500 2021-08-28T12:34:58.217 th_a:invoke handle_block schedule_pending_div ] total_fee_per_asset_in_payout_asset: 0 dividend_data.options: ["ERROR_WHILE_CONVERTING_VALUE_TO_STRING"] db_maint.cpp:1540 2021-08-28T12:34:58.218 th_a:invoke handle_block schedule_pending_div ] payout_asset_type: 1.3.1 current_balance: 5 previous_balance: 0 db_maint.cpp:1500 2021-08-28T12:34:58.218 th_a:invoke handle_block schedule_pending_div ] total_fee_per_asset_in_payout_asset: 0 dividend_data.options: ["ERROR_WHILE_CONVERTING_VALUE_TO_STRING"] db_maint.cpp:1540 2021-08-28T12:34:58.231 th_a:invoke handle_block update_active_sons ] Active SONs set NOT CHANGED db_maint.cpp:751 2021-08-28T12:34:58.461 th_a:?unnamed? plugin_startup ] Bitcoin sidechain handler running peerplays_sidechain_plugin.cpp:223 2021-08-28T12:34:58.461 th_a:?unnamed? sidechain_net_handle ] Peerplays Public Key: PPY6MkvgJqbTLE5e2j4ySFXDhy4G6Tx957qYDYWMGC5QVDNAyAWvJ sidechain_net_handler_peerplays.cpp:33 2021-08-28T12:34:58.461 th_a:?unnamed? plugin_startup ] Peerplays sidechain handler running peerplays_sidechain_plugin.cpp:233 2021-08-28T12:34:58.461 th_a:?unnamed? plugin_startup ] peerplays sidechain plugin: plugin_startup() end peerplays_sidechain_plugin.cpp:684 2021-08-28T12:34:58.461 th_a:?unnamed? plugin_startup ] witness plugin: plugin_startup() begin witness.cpp:126 2021-08-28T12:34:58.463 th_a:?unnamed? plugin_startup ] No witnesses configured! Please add witness IDs and private keys to configuration. witness.cpp:141 2021-08-28T12:34:58.463 th_a:?unnamed? plugin_startup ] witness plugin: plugin_startup() end witness.cpp:142 2021-08-28T12:34:58.463 th_a:?unnamed? main ] Started Peerplays node on a chain with 42990495 blocks. main.cpp:180 2021-08-28T12:34:58.463 th_a:?unnamed? main ] Chain ID is 6b6b5f0ce7a36d323768e534f3edb41c6d6332a541a95725b98e28d140850134 main.cpp:181 2021-08-28T12:34:59.006 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.006 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.006 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.006 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.006 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.007 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.007 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.014 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.014 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.014 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.014 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.014 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.015 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.015 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.018 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.018 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.018 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.018 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.018 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.019 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.019 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.026 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.026 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.026 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.026 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.026 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.027 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.027 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.030 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.030 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.030 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.030 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 ... This goes on for 4 pages ... 2021-08-28T12:34:59.126 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.126 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.126 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.127 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:35:00.184 th_a:invoke handle_block handle_block ] Got block: #42990688 time: 2021-08-28T12:35:00 latency: 184 ms from: houdini-witness irreversible: 42990679 (-9) application.cpp:495 2021-08-28T12:35:00.685 th_a:SON Processing son_processing ] Scheduled SON: 1.33.1 Now: 2021-08-28T12:35:01 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:35:03.165 th_a:invoke handle_block handle_block ] Got block: #42990689 time: 2021-08-28T12:35:03 latency: 165 ms from: eifos-witness irreversible: 42990679 (-10) application.cpp:495 2021-08-28T12:35:03.665 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:35:04 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:35:06.354 th_a:invoke handle_block handle_block ] Got block: #42990690 time: 2021-08-28T12:35:06 latency: 354 ms from: spacecrypt-witness irreversible: 42990679 (-11) application.cpp:495 2021-08-28T12:35:06.855 th_a:SON Processing son_processing ] Scheduled SON: 1.33.7 Now: 2021-08-28T12:35:07 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:35:06.855 th_a:SON Processing create_son_down_prop ] Sending son down proposal for 1.33.0 from 1.33.7 peerplays_sidechain_plugin.cpp:558 2021-08-28T12:35:06.855 th_a:SON Processing create_son_down_prop ] Sending son down proposal for 1.33.1 from 1.33.7 peerplays_sidechain_plugin.cpp:558 2021-08-28T12:35:06.856 th_a:SON Processing create_son_down_prop ] Sending son down proposal for 1.33.2 from 1.33.7 peerplays_sidechain_plugin.cpp:558 2021-08-28T12:35:06.856 th_a:SON Processing create_son_down_prop ] Sending son down proposal for 1.33.3 from 1.33.7 peerplays_sidechain_plugin.cpp:558 2021-08-28T12:35:06.856 th_a:SON Processing create_son_down_prop ] Sending son down proposal for 1.33.4 from 1.33.7 peerplays_sidechain_plugin.cpp:558 2021-08-28T12:35:06.857 th_a:SON Processing create_son_down_prop ] Sending son down proposal for 1.33.9 from 1.33.7 peerplays_sidechain_plugin.cpp:558 (END) ``` #### p2p.log ```text 2021-08-28T12:34:57.396 p2p:message read_loop on_connection_accept ] I don't know if I'm firewalled. Sending a firewall check message to peer 88.198.13.39:9777 node.cpp:2121 2021-08-28T12:34:58.182 p2p:message read_loop read_loop ] disconnected 11 eof_exception: End Of File End of file {"message":"End of file"} asio asio.cpp:39 operator() {"len":16} p2p stcp_socket.cpp:117 readsome message_oriented_connection.cpp:218 2021-08-28T12:34:58.373 p2p:message read_loop on_check_firewall_re ] Firewall check we initiated has returned with result: connection_successful, endpoint = 18.116.153.207:9777 node.cpp:3774 2021-08-28T12:35:02.996 p2p:terminate_inactive_connections_loop terminate_inactive_c ] Forcibly disconnecting from handshaking peer 173.249.23.108:9777 due to inactivity of at least 5 seconds node.cpp:1345 2021-08-28T12:35:02.996 p2p:terminate_inactive_connections_loop terminate_inactive_c ] Peer's negotiating status: connecting, bytes sent: 0, bytes received: 0 node.cpp:1347 2021-08-28T12:35:02.997 p2p:connect_to_task connect_to ] fatal: error connecting to peer 173.249.23.108:9777: 0 exception: unspecified Operation canceled {"message":"Operation canceled"} asio asio.cpp:60 error_handler peer_connection.cpp:264 2021-08-28T12:35:02.998 p2p:delayed_peer_deletion_task destroy ] Unexpected exception from peer_connection's accept_or_connect_task : ["ERROR_WHILE_CONVERTING_VALUE_TO_STRING"] peer_connection.cpp:167 (END) ``` The RPC log is blank.

serkixenos commented

2021-09-02 13:22:39 +00:00

(Migrated from gitlab.com)

Can you share your config file, remove private keys, but leave public ones.

Something is very wrong here, output like this is never supposed to happen.

2021-08-28T12:34:59.006   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.006   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.006   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.006   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.006   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.007   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.007   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.014   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.014   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.014   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.014   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.014   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.015   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.015   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.018   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.018   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.018   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.018   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.018   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.019   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.019   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.026   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.026   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.026   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.026   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.026   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.027   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.027   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.030   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.030   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.030   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398
2021-08-28T12:34:59.030   th_a:SON Processing       son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59                     peerplays_sidechain_plugin.cpp:398

Can you share your config file, remove private keys, but leave public ones. Something is very wrong here, output like this is never supposed to happen. ``` 2021-08-28T12:34:59.006 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.006 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.006 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.006 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.006 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.007 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.007 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.014 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.014 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.014 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.014 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.014 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.015 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.015 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.018 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.018 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.018 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.018 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.018 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.019 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.019 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.026 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.026 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.026 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.026 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.026 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.027 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.027 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.030 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.030 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.030 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 2021-08-28T12:34:59.030 th_a:SON Processing son_processing ] Scheduled SON: 1.33.3 Now: 2021-08-28T12:34:59 peerplays_sidechain_plugin.cpp:398 ```

Hiltos1 commented

2021-09-02 13:53:52 +00:00

(Migrated from gitlab.com)

Here's the contents of my config.ini file for the SON (attached).

SON_config.txt

Here's the contents of my `config.ini` file for the SON (attached). [SON_config.txt](/uploads/1847b91e7d3f7dbf12c4063d0adc7387/SON_config.txt)

serkixenos commented

2021-09-09 11:34:07 +00:00

(Migrated from gitlab.com)

I identified the reason for many Scheduled SON messages, its not critical, but it should not happen, I am working on a fix. Now, I am concerned about SON down proposals. This means that you have invalid data in SON statistic objects, or invalid local time.

Can you share the following data from your database, for each SON (I believe there are 10 of them, so ids are from 1.33.0 to 1.33.9)

# Get son
locked >>> get_object 1.33.0
get_object 1.33.0
{
  "id": "1.33.0",
  "son_account": "1.2.14355",
  "vote_id": "3:101",
  "total_votes": 100000000,
  "url": "https://beta.eifos.org",
  "deposit": "1.13.226",
  "signing_key": "PPY7epqV6FYhyMi8tkt4A7BAp6A9L5Z1zS3bwdWTVxUbeiREPYBef",
  "pay_vb": "1.13.225",
  "statistics": "2.25.0",
  "status": "active",
  "sidechain_public_keys": [[
      "bitcoin",
      "0242e491c8955b66f2d1e5b683ea78cb77f874d23e685365155395500997929daf"
    ]
  ]
}

# Get SON statistic object (get id from statistics field)
locked >>> get_object 2.25.0
get_object 2.25.0
[{
    "id": "2.25.0",
    "owner": "1.33.0",
    "total_txs_signed": 0,
    "txs_signed": 0,
    "total_voted_time": 22629600,
    "total_downtime": 0,
    "current_interval_downtime": 0,
    "last_down_timestamp": "1970-01-01T00:00:00",
    "last_active_timestamp": "1970-01-01T00:00:00",
    "deregistered_timestamp": "1970-01-01T00:00:00",
    "total_sidechain_txs_reported": 0,
    "sidechain_txs_reported": 0
  }
]

We need to see what values you have in three timestamp fields.

I identified the reason for many Scheduled SON messages, its not critical, but it should not happen, I am working on a fix. Now, I am concerned about SON down proposals. This means that you have invalid data in SON statistic objects, or invalid local time. Can you share the following data from your database, for each SON (I believe there are 10 of them, so ids are from 1.33.0 to 1.33.9) ``` # Get son locked >>> get_object 1.33.0 get_object 1.33.0 { "id": "1.33.0", "son_account": "1.2.14355", "vote_id": "3:101", "total_votes": 100000000, "url": "https://beta.eifos.org", "deposit": "1.13.226", "signing_key": "PPY7epqV6FYhyMi8tkt4A7BAp6A9L5Z1zS3bwdWTVxUbeiREPYBef", "pay_vb": "1.13.225", "statistics": "2.25.0", "status": "active", "sidechain_public_keys": [[ "bitcoin", "0242e491c8955b66f2d1e5b683ea78cb77f874d23e685365155395500997929daf" ] ] } # Get SON statistic object (get id from statistics field) locked >>> get_object 2.25.0 get_object 2.25.0 [{ "id": "2.25.0", "owner": "1.33.0", "total_txs_signed": 0, "txs_signed": 0, "total_voted_time": 22629600, "total_downtime": 0, "current_interval_downtime": 0, "last_down_timestamp": "1970-01-01T00:00:00", "last_active_timestamp": "1970-01-01T00:00:00", "deregistered_timestamp": "1970-01-01T00:00:00", "total_sidechain_txs_reported": 0, "sidechain_txs_reported": 0 } ] ``` We need to see what values you have in three timestamp fields.

serkixenos commented

2021-09-09 11:50:21 +00:00

(Migrated from gitlab.com)

Now, this is the data from Mainnet... Apparently, only ONE SON is sending heartbeats, all others have last_active_timestamp unchanged.

locked >>> get_object 2.25.0
get_object 2.25.0
[{
    "id": "2.25.0",
    "owner": "1.33.0",
    "total_txs_signed": 0,
    "txs_signed": 0,
    "total_voted_time": 22629600,
    "total_downtime": 0,
    "current_interval_downtime": 0,
    "last_down_timestamp": "1970-01-01T00:00:00",
    "last_active_timestamp": "1970-01-01T00:00:00",
    "deregistered_timestamp": "1970-01-01T00:00:00",
    "total_sidechain_txs_reported": 0,
    "sidechain_txs_reported": 0
  }
]
locked >>> get_object 2.25.0
get_object 2.25.0
[{
    "id": "2.25.0",
    "owner": "1.33.0",
    "total_txs_signed": 0,
    "txs_signed": 0,
    "total_voted_time": 22629600,
    "total_downtime": 0,
    "current_interval_downtime": 0,
    "last_down_timestamp": "1970-01-01T00:00:00",
    "last_active_timestamp": "1970-01-01T00:00:00",
    "deregistered_timestamp": "1970-01-01T00:00:00",
    "total_sidechain_txs_reported": 0,
    "sidechain_txs_reported": 0
  }
]
locked >>> get_object 2.25.1
get_object 2.25.1
[{
    "id": "2.25.1",
    "owner": "1.33.1",
    "total_txs_signed": 0,
    "txs_signed": 0,
    "total_voted_time": 19832400,
    "total_downtime": 0,
    "current_interval_downtime": 0,
    "last_down_timestamp": "1970-01-01T00:00:00",
    "last_active_timestamp": "1970-01-01T00:00:00",
    "deregistered_timestamp": "1970-01-01T00:00:00",
    "total_sidechain_txs_reported": 0,
    "sidechain_txs_reported": 0
  }
]
locked >>> get_object 2.25.2
get_object 2.25.2
[{
    "id": "2.25.2",
    "owner": "1.33.2",
    "total_txs_signed": 0,
    "txs_signed": 0,
    "total_voted_time": 19832400,
    "total_downtime": 0,
    "current_interval_downtime": 0,
    "last_down_timestamp": "1970-01-01T00:00:00",
    "last_active_timestamp": "1970-01-01T00:00:00",
    "deregistered_timestamp": "1970-01-01T00:00:00",
    "total_sidechain_txs_reported": 0,
    "sidechain_txs_reported": 0
  }
]
locked >>> get_object 2.25.3
get_object 2.25.3
[{
    "id": "2.25.3",
    "owner": "1.33.3",
    "total_txs_signed": 0,
    "txs_signed": 0,
    "total_voted_time": 19832400,
    "total_downtime": 0,
    "current_interval_downtime": 0,
    "last_down_timestamp": "1970-01-01T00:00:00",
    "last_active_timestamp": "1970-01-01T00:00:00",
    "deregistered_timestamp": "1970-01-01T00:00:00",
    "total_sidechain_txs_reported": 0,
    "sidechain_txs_reported": 0
  }
]
locked >>> get_object 2.25.4
get_object 2.25.4
[{
    "id": "2.25.4",
    "owner": "1.33.4",
    "total_txs_signed": 0,
    "txs_signed": 0,
    "total_voted_time": 19832400,
    "total_downtime": 0,
    "current_interval_downtime": 0,
    "last_down_timestamp": "1970-01-01T00:00:00",
    "last_active_timestamp": "1970-01-01T00:00:00",
    "deregistered_timestamp": "1970-01-01T00:00:00",
    "total_sidechain_txs_reported": 0,
    "sidechain_txs_reported": 0
  }
]
locked >>> get_object 2.25.5
get_object 2.25.5
[{
    "id": "2.25.5",
    "owner": "1.33.5",
    "total_txs_signed": 0,
    "txs_signed": 0,
    "total_voted_time": 16632000,
    "total_downtime": 0,
    "current_interval_downtime": 0,
    "last_down_timestamp": "1970-01-01T00:00:00",
    "last_active_timestamp": "1970-01-01T00:00:00",
    "deregistered_timestamp": "1970-01-01T00:00:00",
    "total_sidechain_txs_reported": 0,
    "sidechain_txs_reported": 0
  }
]
locked >>> get_object 2.25.6
get_object 2.25.6
[{
    "id": "2.25.6",
    "owner": "1.33.6",
    "total_txs_signed": 0,
    "txs_signed": 0,
    "total_voted_time": 12272400,
    "total_downtime": 0,
    "current_interval_downtime": 0,
    "last_down_timestamp": "1970-01-01T00:00:00",
    "last_active_timestamp": "1970-01-01T00:00:00",
    "deregistered_timestamp": "1970-01-01T00:00:00",
    "total_sidechain_txs_reported": 0,
    "sidechain_txs_reported": 0
  }
]
locked >>> get_object 2.25.7
get_object 2.25.7
[{
    "id": "2.25.7",
    "owner": "1.33.7",
    "total_txs_signed": 0,
    "txs_signed": 0,
    "total_voted_time": 4719600,
    "total_downtime": 0,
    "current_interval_downtime": 0,
    "last_down_timestamp": "1970-01-01T00:00:00",
    "last_active_timestamp": "2021-07-17T20:42:14",
    "deregistered_timestamp": "1970-01-01T00:00:00",
    "total_sidechain_txs_reported": 0,
    "sidechain_txs_reported": 0
  }
]
locked >>> get_object 2.25.8
get_object 2.25.8
[{
    "id": "2.25.8",
    "owner": "1.33.8",
    "total_txs_signed": 0,
    "txs_signed": 0,
    "total_voted_time": 0,
    "total_downtime": 0,
    "current_interval_downtime": 0,
    "last_down_timestamp": "1970-01-01T00:00:00",
    "last_active_timestamp": "1970-01-01T00:00:00",
    "deregistered_timestamp": "1970-01-01T00:00:00",
    "total_sidechain_txs_reported": 0,
    "sidechain_txs_reported": 0
  }
]
locked >>> get_object 2.25.9
get_object 2.25.9
[{
    "id": "2.25.9",
    "owner": "1.33.9",
    "total_txs_signed": 0,
    "txs_signed": 0,
    "total_voted_time": 2800800,
    "total_downtime": 0,
    "current_interval_downtime": 0,
    "last_down_timestamp": "1970-01-01T00:00:00",
    "last_active_timestamp": "1970-01-01T00:00:00",
    "deregistered_timestamp": "1970-01-01T00:00:00",
    "total_sidechain_txs_reported": 0,
    "sidechain_txs_reported": 0
  }
]
locked >>> get_object 2.25.10

Now, this is the data from Mainnet... Apparently, only ONE SON is sending heartbeats, all others have last_active_timestamp unchanged. ``` locked >>> get_object 2.25.0 get_object 2.25.0 [{ "id": "2.25.0", "owner": "1.33.0", "total_txs_signed": 0, "txs_signed": 0, "total_voted_time": 22629600, "total_downtime": 0, "current_interval_downtime": 0, "last_down_timestamp": "1970-01-01T00:00:00", "last_active_timestamp": "1970-01-01T00:00:00", "deregistered_timestamp": "1970-01-01T00:00:00", "total_sidechain_txs_reported": 0, "sidechain_txs_reported": 0 } ] locked >>> get_object 2.25.0 get_object 2.25.0 [{ "id": "2.25.0", "owner": "1.33.0", "total_txs_signed": 0, "txs_signed": 0, "total_voted_time": 22629600, "total_downtime": 0, "current_interval_downtime": 0, "last_down_timestamp": "1970-01-01T00:00:00", "last_active_timestamp": "1970-01-01T00:00:00", "deregistered_timestamp": "1970-01-01T00:00:00", "total_sidechain_txs_reported": 0, "sidechain_txs_reported": 0 } ] locked >>> get_object 2.25.1 get_object 2.25.1 [{ "id": "2.25.1", "owner": "1.33.1", "total_txs_signed": 0, "txs_signed": 0, "total_voted_time": 19832400, "total_downtime": 0, "current_interval_downtime": 0, "last_down_timestamp": "1970-01-01T00:00:00", "last_active_timestamp": "1970-01-01T00:00:00", "deregistered_timestamp": "1970-01-01T00:00:00", "total_sidechain_txs_reported": 0, "sidechain_txs_reported": 0 } ] locked >>> get_object 2.25.2 get_object 2.25.2 [{ "id": "2.25.2", "owner": "1.33.2", "total_txs_signed": 0, "txs_signed": 0, "total_voted_time": 19832400, "total_downtime": 0, "current_interval_downtime": 0, "last_down_timestamp": "1970-01-01T00:00:00", "last_active_timestamp": "1970-01-01T00:00:00", "deregistered_timestamp": "1970-01-01T00:00:00", "total_sidechain_txs_reported": 0, "sidechain_txs_reported": 0 } ] locked >>> get_object 2.25.3 get_object 2.25.3 [{ "id": "2.25.3", "owner": "1.33.3", "total_txs_signed": 0, "txs_signed": 0, "total_voted_time": 19832400, "total_downtime": 0, "current_interval_downtime": 0, "last_down_timestamp": "1970-01-01T00:00:00", "last_active_timestamp": "1970-01-01T00:00:00", "deregistered_timestamp": "1970-01-01T00:00:00", "total_sidechain_txs_reported": 0, "sidechain_txs_reported": 0 } ] locked >>> get_object 2.25.4 get_object 2.25.4 [{ "id": "2.25.4", "owner": "1.33.4", "total_txs_signed": 0, "txs_signed": 0, "total_voted_time": 19832400, "total_downtime": 0, "current_interval_downtime": 0, "last_down_timestamp": "1970-01-01T00:00:00", "last_active_timestamp": "1970-01-01T00:00:00", "deregistered_timestamp": "1970-01-01T00:00:00", "total_sidechain_txs_reported": 0, "sidechain_txs_reported": 0 } ] locked >>> get_object 2.25.5 get_object 2.25.5 [{ "id": "2.25.5", "owner": "1.33.5", "total_txs_signed": 0, "txs_signed": 0, "total_voted_time": 16632000, "total_downtime": 0, "current_interval_downtime": 0, "last_down_timestamp": "1970-01-01T00:00:00", "last_active_timestamp": "1970-01-01T00:00:00", "deregistered_timestamp": "1970-01-01T00:00:00", "total_sidechain_txs_reported": 0, "sidechain_txs_reported": 0 } ] locked >>> get_object 2.25.6 get_object 2.25.6 [{ "id": "2.25.6", "owner": "1.33.6", "total_txs_signed": 0, "txs_signed": 0, "total_voted_time": 12272400, "total_downtime": 0, "current_interval_downtime": 0, "last_down_timestamp": "1970-01-01T00:00:00", "last_active_timestamp": "1970-01-01T00:00:00", "deregistered_timestamp": "1970-01-01T00:00:00", "total_sidechain_txs_reported": 0, "sidechain_txs_reported": 0 } ] locked >>> get_object 2.25.7 get_object 2.25.7 [{ "id": "2.25.7", "owner": "1.33.7", "total_txs_signed": 0, "txs_signed": 0, "total_voted_time": 4719600, "total_downtime": 0, "current_interval_downtime": 0, "last_down_timestamp": "1970-01-01T00:00:00", "last_active_timestamp": "2021-07-17T20:42:14", "deregistered_timestamp": "1970-01-01T00:00:00", "total_sidechain_txs_reported": 0, "sidechain_txs_reported": 0 } ] locked >>> get_object 2.25.8 get_object 2.25.8 [{ "id": "2.25.8", "owner": "1.33.8", "total_txs_signed": 0, "txs_signed": 0, "total_voted_time": 0, "total_downtime": 0, "current_interval_downtime": 0, "last_down_timestamp": "1970-01-01T00:00:00", "last_active_timestamp": "1970-01-01T00:00:00", "deregistered_timestamp": "1970-01-01T00:00:00", "total_sidechain_txs_reported": 0, "sidechain_txs_reported": 0 } ] locked >>> get_object 2.25.9 get_object 2.25.9 [{ "id": "2.25.9", "owner": "1.33.9", "total_txs_signed": 0, "txs_signed": 0, "total_voted_time": 2800800, "total_downtime": 0, "current_interval_downtime": 0, "last_down_timestamp": "1970-01-01T00:00:00", "last_active_timestamp": "1970-01-01T00:00:00", "deregistered_timestamp": "1970-01-01T00:00:00", "total_sidechain_txs_reported": 0, "sidechain_txs_reported": 0 } ] locked >>> get_object 2.25.10 ```

serkixenos commented

2021-09-09 11:51:17 +00:00

(Migrated from gitlab.com)

Also, we have 3 inactive SONs.

locked >>> get_object 1.33.0
get_object 1.33.0
[{
    "id": "1.33.0",
    "son_account": "1.2.14355",
    "vote_id": "3:101",
    "total_votes": 100000000,
    "url": "https://beta.eifos.org",
    "deposit": "1.13.226",
    "signing_key": "PPY7epqV6FYhyMi8tkt4A7BAp6A9L5Z1zS3bwdWTVxUbeiREPYBef",
    "pay_vb": "1.13.225",
    "statistics": "2.25.0",
    "status": "active",
    "sidechain_public_keys": [[
        "bitcoin",
        "0242e491c8955b66f2d1e5b683ea78cb77f874d23e685365155395500997929daf"
      ]
    ]
  }
]
locked >>> get_object 1.33.1
get_object 1.33.1
[{
    "id": "1.33.1",
    "son_account": "1.2.14376",
    "vote_id": "3:104",
    "total_votes": 100000000,
    "url": "https://www.thespicegirls.com/",
    "deposit": "1.13.232",
    "signing_key": "PPY7ic8CA6d4vr4JERQAV4TDKy5JHjusoxT17u7SDUctimHVqHVKg",
    "pay_vb": "1.13.233",
    "statistics": "2.25.1",
    "status": "active",
    "sidechain_public_keys": [[
        "bitcoin",
        "023e342a8b8f38c193d714891a92bf987dc3955d623772c35be6c181b7bfb3cd27"
      ]
    ]
  }
]
locked >>> get_object 1.33.2
get_object 1.33.2
[{
    "id": "1.33.2",
    "son_account": "1.2.14377",
    "vote_id": "3:105",
    "total_votes": 100000000,
    "url": "https://www.thespicegirls.com/",
    "deposit": "1.13.234",
    "signing_key": "PPY7EjSR9mnhYUHVs56gXDSCBECpjtzQJCib2By6o21Bojc2FXwNX",
    "pay_vb": "1.13.235",
    "statistics": "2.25.2",
    "status": "active",
    "sidechain_public_keys": [[
        "bitcoin",
        "03f1a33fdcd8493aeb35f65fe2ef1d5bcf85dba915e52ccfc93c35b24354c4976e"
      ]
    ]
  }
]
locked >>> get_object 1.33.3
get_object 1.33.3
[{
    "id": "1.33.3",
    "son_account": "1.2.14378",
    "vote_id": "3:106",
    "total_votes": 100000000,
    "url": "https://www.thespicegirls.com/",
    "deposit": "1.13.236",
    "signing_key": "PPY8Uof6wArVqe4RVpF3AhvA9ahNhvHvnFr1g1caY1qDsEoZkvcbR",
    "pay_vb": "1.13.237",
    "statistics": "2.25.3",
    "status": "active",
    "sidechain_public_keys": [[
        "bitcoin",
        "037a82f389c6fbe18581b4e83a7dc58716d1af19a16ada72f2f3e411df3202d6eb"
      ]
    ]
  }
]
locked >>> get_object 1.33.4
get_object 1.33.4
[{
    "id": "1.33.4",
    "son_account": "1.2.14379",
    "vote_id": "3:107",
    "total_votes": 100000000,
    "url": "https://www.thespicegirls.com/",
    "deposit": "1.13.238",
    "signing_key": "PPY7w85b5vefWkrauUhojR7Fm8S5eNrxfC8XLNetC4cnKgydgE7k1",
    "pay_vb": "1.13.239",
    "statistics": "2.25.4",
    "status": "active",
    "sidechain_public_keys": [[
        "bitcoin",
        "021610f888ab219c53b958e1d46555b10e12477fecba6e051f8d82e902b6e5c0eb"
      ]
    ]
  }
]
locked >>> get_object 1.33.5
get_object 1.33.5
[{
    "id": "1.33.5",
    "son_account": "1.2.14364",
    "vote_id": "3:108",
    "total_votes": 0,
    "url": "https://peerplaysblockchain.info/",
    "deposit": "1.13.242",
    "signing_key": "PPY5t6ew3mmPaES3cyN7bJTE4T2TjLZDfY1E2nT4ZjbzM2QEf3xha",
    "pay_vb": "1.13.241",
    "statistics": "2.25.5",
    "status": "inactive",
    "sidechain_public_keys": [[
        "bitcoin",
        "02a088e1f88805609b3fbf51d484ab95b4e2c7ca4995815b5fa485775aacd1b50d"
      ]
    ]
  }
]
locked >>> get_object 1.33.6
get_object 1.33.6
[{
    "id": "1.33.6",
    "son_account": "1.2.14407",
    "vote_id": "3:110",
    "total_votes": 0,
    "url": "",
    "deposit": "1.13.252",
    "signing_key": "PPY8jMvs9bNghhmaXTnNpXygbn7UPULkRDt6PFTFdq5pz1jpjH5P7",
    "pay_vb": "1.13.253",
    "statistics": "2.25.6",
    "status": "inactive",
    "sidechain_public_keys": [[
        "bitcoin",
        "02c640a878225f8a48c741cd60aa886d3f7b88054ae2004060fd91783aabb7a199"
      ]
    ]
  }
]
locked >>> get_object 1.33.7
get_object 1.33.7
[{
    "id": "1.33.7",
    "son_account": "1.2.14436",
    "vote_id": "3:115",
    "total_votes": 2200000000,
    "url": "https://son.hiltos.network",
    "deposit": "1.13.290",
    "signing_key": "PPY6MkvgJqbTLE5e2j4ySFXDhy4G6Tx957qYDYWMGC5QVDNAyAWvJ",
    "pay_vb": "1.13.291",
    "statistics": "2.25.7",
    "status": "active",
    "sidechain_public_keys": [[
        "bitcoin",
        "02f2be7c0784fdc2d379a3925e6408925c70f357e909c28ff6fb93e7f567b6af7f"
      ]
    ]
  }
]
locked >>> get_object 1.33.8
get_object 1.33.8
[{
    "id": "1.33.8",
    "son_account": "1.2.14466",
    "vote_id": "3:116",
    "total_votes": 0,
    "url": "https://imagens.i9shops.com.br/about.html",
    "deposit": "1.13.293",
    "signing_key": "PPY5erE11fGLhWT6ypZaPyRjWyZffABqsxWhLwvAmN5AwyTPaNRay",
    "pay_vb": "1.13.294",
    "statistics": "2.25.8",
    "status": "inactive",
    "sidechain_public_keys": [[
        "bitcoin",
        "02975eb99b7168ad2c58693fae71edef5711dc6d4bd9c926d229460b17d9343529"
      ]
    ]
  }
]
locked >>> get_object 1.33.9
get_object 1.33.9
[{
    "id": "1.33.9",
    "son_account": "1.2.14467",
    "vote_id": "3:117",
    "total_votes": 100000000,
    "url": "",
    "deposit": "1.13.295",
    "signing_key": "PPY8kULPF18gL2oFW9rTmuRNq544wkBjQXYizmvKujZT838S7amSs",
    "pay_vb": "1.13.296",
    "statistics": "2.25.9",
    "status": "active",
    "sidechain_public_keys": [[
        "bitcoin",
        "03124d27e7d13b030136dc84bf4d15b209b1b887aad5fb14542aea4ffff907cf7d"
      ]
    ]
  }
]

Also, we have 3 inactive SONs. ``` locked >>> get_object 1.33.0 get_object 1.33.0 [{ "id": "1.33.0", "son_account": "1.2.14355", "vote_id": "3:101", "total_votes": 100000000, "url": "https://beta.eifos.org", "deposit": "1.13.226", "signing_key": "PPY7epqV6FYhyMi8tkt4A7BAp6A9L5Z1zS3bwdWTVxUbeiREPYBef", "pay_vb": "1.13.225", "statistics": "2.25.0", "status": "active", "sidechain_public_keys": [[ "bitcoin", "0242e491c8955b66f2d1e5b683ea78cb77f874d23e685365155395500997929daf" ] ] } ] locked >>> get_object 1.33.1 get_object 1.33.1 [{ "id": "1.33.1", "son_account": "1.2.14376", "vote_id": "3:104", "total_votes": 100000000, "url": "https://www.thespicegirls.com/", "deposit": "1.13.232", "signing_key": "PPY7ic8CA6d4vr4JERQAV4TDKy5JHjusoxT17u7SDUctimHVqHVKg", "pay_vb": "1.13.233", "statistics": "2.25.1", "status": "active", "sidechain_public_keys": [[ "bitcoin", "023e342a8b8f38c193d714891a92bf987dc3955d623772c35be6c181b7bfb3cd27" ] ] } ] locked >>> get_object 1.33.2 get_object 1.33.2 [{ "id": "1.33.2", "son_account": "1.2.14377", "vote_id": "3:105", "total_votes": 100000000, "url": "https://www.thespicegirls.com/", "deposit": "1.13.234", "signing_key": "PPY7EjSR9mnhYUHVs56gXDSCBECpjtzQJCib2By6o21Bojc2FXwNX", "pay_vb": "1.13.235", "statistics": "2.25.2", "status": "active", "sidechain_public_keys": [[ "bitcoin", "03f1a33fdcd8493aeb35f65fe2ef1d5bcf85dba915e52ccfc93c35b24354c4976e" ] ] } ] locked >>> get_object 1.33.3 get_object 1.33.3 [{ "id": "1.33.3", "son_account": "1.2.14378", "vote_id": "3:106", "total_votes": 100000000, "url": "https://www.thespicegirls.com/", "deposit": "1.13.236", "signing_key": "PPY8Uof6wArVqe4RVpF3AhvA9ahNhvHvnFr1g1caY1qDsEoZkvcbR", "pay_vb": "1.13.237", "statistics": "2.25.3", "status": "active", "sidechain_public_keys": [[ "bitcoin", "037a82f389c6fbe18581b4e83a7dc58716d1af19a16ada72f2f3e411df3202d6eb" ] ] } ] locked >>> get_object 1.33.4 get_object 1.33.4 [{ "id": "1.33.4", "son_account": "1.2.14379", "vote_id": "3:107", "total_votes": 100000000, "url": "https://www.thespicegirls.com/", "deposit": "1.13.238", "signing_key": "PPY7w85b5vefWkrauUhojR7Fm8S5eNrxfC8XLNetC4cnKgydgE7k1", "pay_vb": "1.13.239", "statistics": "2.25.4", "status": "active", "sidechain_public_keys": [[ "bitcoin", "021610f888ab219c53b958e1d46555b10e12477fecba6e051f8d82e902b6e5c0eb" ] ] } ] locked >>> get_object 1.33.5 get_object 1.33.5 [{ "id": "1.33.5", "son_account": "1.2.14364", "vote_id": "3:108", "total_votes": 0, "url": "https://peerplaysblockchain.info/", "deposit": "1.13.242", "signing_key": "PPY5t6ew3mmPaES3cyN7bJTE4T2TjLZDfY1E2nT4ZjbzM2QEf3xha", "pay_vb": "1.13.241", "statistics": "2.25.5", "status": "inactive", "sidechain_public_keys": [[ "bitcoin", "02a088e1f88805609b3fbf51d484ab95b4e2c7ca4995815b5fa485775aacd1b50d" ] ] } ] locked >>> get_object 1.33.6 get_object 1.33.6 [{ "id": "1.33.6", "son_account": "1.2.14407", "vote_id": "3:110", "total_votes": 0, "url": "", "deposit": "1.13.252", "signing_key": "PPY8jMvs9bNghhmaXTnNpXygbn7UPULkRDt6PFTFdq5pz1jpjH5P7", "pay_vb": "1.13.253", "statistics": "2.25.6", "status": "inactive", "sidechain_public_keys": [[ "bitcoin", "02c640a878225f8a48c741cd60aa886d3f7b88054ae2004060fd91783aabb7a199" ] ] } ] locked >>> get_object 1.33.7 get_object 1.33.7 [{ "id": "1.33.7", "son_account": "1.2.14436", "vote_id": "3:115", "total_votes": 2200000000, "url": "https://son.hiltos.network", "deposit": "1.13.290", "signing_key": "PPY6MkvgJqbTLE5e2j4ySFXDhy4G6Tx957qYDYWMGC5QVDNAyAWvJ", "pay_vb": "1.13.291", "statistics": "2.25.7", "status": "active", "sidechain_public_keys": [[ "bitcoin", "02f2be7c0784fdc2d379a3925e6408925c70f357e909c28ff6fb93e7f567b6af7f" ] ] } ] locked >>> get_object 1.33.8 get_object 1.33.8 [{ "id": "1.33.8", "son_account": "1.2.14466", "vote_id": "3:116", "total_votes": 0, "url": "https://imagens.i9shops.com.br/about.html", "deposit": "1.13.293", "signing_key": "PPY5erE11fGLhWT6ypZaPyRjWyZffABqsxWhLwvAmN5AwyTPaNRay", "pay_vb": "1.13.294", "statistics": "2.25.8", "status": "inactive", "sidechain_public_keys": [[ "bitcoin", "02975eb99b7168ad2c58693fae71edef5711dc6d4bd9c926d229460b17d9343529" ] ] } ] locked >>> get_object 1.33.9 get_object 1.33.9 [{ "id": "1.33.9", "son_account": "1.2.14467", "vote_id": "3:117", "total_votes": 100000000, "url": "", "deposit": "1.13.295", "signing_key": "PPY8kULPF18gL2oFW9rTmuRNq544wkBjQXYizmvKujZT838S7amSs", "pay_vb": "1.13.296", "statistics": "2.25.9", "status": "active", "sidechain_public_keys": [[ "bitcoin", "03124d27e7d13b030136dc84bf4d15b209b1b887aad5fb14542aea4ffff907cf7d" ] ] } ] ```

serkixenos commented

2021-09-09 19:23:27 +00:00

(Migrated from gitlab.com)

Apparently, this is caused by more than few misconfigured SONs on mainnet. Putting it on Hold, until we get feedback from SON operators.

serkixenos commented

2021-10-14 12:28:05 +00:00

(Migrated from gitlab.com)

assigned to @prandnum and unassigned @sierra19XX and @serkixenos

serkixenos commented

2021-10-14 12:28:08 +00:00

(Migrated from gitlab.com)

Might be resolved with:
9b3c84c2bd

@prandnum Please retest

Might be resolved with: https://gitlab.com/PBSA/peerplays/-/commit/9b3c84c2bdd0f251140b08cd0a0683e1abafdd5e @prandnum Please retest

prandnum commented

2021-10-14 19:51:15 +00:00

(Migrated from gitlab.com)

@serkixenos I am not the originator of this ticket, hence not aware of the scenario to be verified. please provide the verification steps for this bug.

serkixenos commented

2021-10-15 14:35:27 +00:00

(Migrated from gitlab.com)

If the problem is (not) what we believe it is (what coredump showed us), this will be visible even in a testnet environment, as this problem is not mainnet/testnet specific.

Just keep the full environment running, and from time to time cast some votes to the SONs to initiate primary wallet recreation.

We can postpone final confirmation on this one until the change reaches Mainnet.

If the problem is (not) what we believe it is (what coredump showed us), this will be visible even in a testnet environment, as this problem is not mainnet/testnet specific. Just keep the full environment running, and from time to time cast some votes to the SONs to initiate primary wallet recreation. We can postpone final confirmation on this one until the change reaches Mainnet.

serkixenos commented

2021-12-01 17:55:50 +00:00

(Migrated from gitlab.com)

mentioned in issue #51

serkixenos commented

2021-12-09 13:57:26 +00:00

(Migrated from gitlab.com)

mentioned in issue #223

hbelakon commented

2021-12-09 19:02:32 +00:00

(Migrated from gitlab.com)

@serkixenos Shall we de-scope this ticket form our December release... Our changes will not reach Mainnet before the end of December- so we can not bother about this ticket in this milestone , right? I am de-scoping it. Please correct me if I am wrong @serkixenos

serkixenos commented

2021-12-30 03:35:17 +00:00

(Migrated from gitlab.com)

SON network is up, nodes are sending heartbeats and there is no crashing:

locked >>> get_global_properties 
get_global_properties 
{
  "id": "2.0.0",
  "parameters": {
    "current_fees": {
      "parameters": [[
          0,{
            "fee": 1000,
            "price_per_kbyte": 1000
          }
        ],[
          1,{
            "fee": 50
          }
        ],[
          2,{
            "fee": 0
          }
        ],[
          3,{
            "fee": "500000000000"
          }
        ],[
          4,{}
        ],[
          5,{
            "basic_fee": 500,
            "premium_fee": 250000,
            "price_per_kbyte": 1000
          }
        ],[
          6,{
            "fee": 100,
            "price_per_kbyte": 1000
          }
        ],[
          7,{
            "fee": 1000
          }
        ],[
          8,{
            "membership_annual_fee": "500000000000",
            "membership_lifetime_fee": 500000
          }
        ],[
          9,{
            "fee": 200000
          }
        ],[
          10,{
            "symbol3": "500000000000",
            "symbol4": "500000000000",
            "long_symbol": 5000000,
            "price_per_kbyte": 1000
          }
        ],[
          11,{
            "fee": 100000,
            "price_per_kbyte": 100
          }
        ],[
          12,{
            "fee": "500000000000"
          }
        ],[
          13,{
            "fee": "500000000000"
          }
        ],[
          14,{
            "fee": 1000,
            "price_per_kbyte": 1000
          }
        ],[
          15,{
            "fee": 1000
          }
        ],[
          16,{
            "fee": 1000
          }
        ],[
          17,{
            "fee": "500000000000"
          }
        ],[
          18,{
            "fee": "500000000000"
          }
        ],[
          19,{
            "fee": "500000000000"
          }
        ],[
          20,{
            "fee": 800000
          }
        ],[
          21,{
            "fee": 50000
          }
        ],[
          22,{
            "fee": 1000,
            "price_per_kbyte": 1000
          }
        ],[
          23,{
            "fee": 1000,
            "price_per_kbyte": 1000
          }
        ],[
          24,{
            "fee": 0
          }
        ],[
          25,{
            "fee": 3000
          }
        ],[
          26,{
            "fee": 1000
          }
        ],[
          27,{
            "fee": 1000,
            "price_per_kbyte": 1000
          }
        ],[
          28,{
            "fee": 0
          }
        ],[
          29,{
            "fee": 100000
          }
        ],[
          30,{
            "fee": 50000
          }
        ],[
          31,{
            "fee": 1000
          }
        ],[
          32,{
            "fee": 100000
          }
        ],[
          33,{
            "fee": 1000
          }
        ],[
          34,{
            "fee": "500000000000"
          }
        ],[
          35,{
            "fee": 1000,
            "price_per_kbyte": 1000
          }
        ],[
          36,{
            "fee": 1000
          }
        ],[
          37,{}
        ],[
          38,{
            "fee": 20000,
            "price_per_kbyte": 1000
          }
        ],[
          39,{
            "fee": 1000,
            "price_per_output": 1000
          }
        ],[
          40,{
            "fee": 1000,
            "price_per_output": 1000
          }
        ],[
          41,{
            "fee": 1000
          }
        ],[
          42,{}
        ],[
          43,{
            "fee": 3000
          }
        ],[
          44,{}
        ],[
          45,{
            "fee": 1000
          }
        ],[
          46,{
            "fee": 5000
          }
        ],[
          47,{
            "fee": 0
          }
        ],[
          48,{
            "fee": 100000
          }
        ],[
          49,{
            "distribution_base_fee": 0,
            "distribution_fee_per_holder": 0
          }
        ],[
          52,{
            "fee": 1000
          }
        ],[
          53,{
            "fee": 1000
          }
        ],[
          54,{
            "fee": 1000
          }
        ],[
          55,{
            "fee": 1000
          }
        ],[
          56,{
            "fee": 1
          }
        ],[
          57,{
            "fee": 1
          }
        ],[
          58,{
            "fee": 1000
          }
        ],[
          59,{
            "fee": 1000
          }
        ],[
          60,{
            "fee": 1
          }
        ],[
          61,{
            "fee": 1
          }
        ],[
          62,{
            "fee": 1
          }
        ],[
          63,{
            "fee": 1
          }
        ],[
          66,{
            "fee": 0
          }
        ],[
          68,{
            "fee": 0
          }
        ],[
          70,{
            "fee": 1
          }
        ],[
          71,{
            "fee": 1
          }
        ],[
          72,{
            "fee": 1
          }
        ],[
          73,{
            "fee": 1000
          }
        ],[
          74,{
            "fee": 1000
          }
        ]
      ],
      "scale": 10000
    },
    "block_interval": 3,
    "maintenance_interval": 3600,
    "maintenance_skip_slots": 3,
    "committee_proposal_review_period": 3600,
    "maximum_transaction_size": 99999,
    "maximum_block_size": 2000000,
    "maximum_time_until_expiration": 86400,
    "maximum_proposal_lifetime": 2419200,
    "maximum_asset_whitelist_authorities": 10,
    "maximum_asset_feed_publishers": 10,
    "maximum_witness_count": 101,
    "maximum_committee_count": 33,
    "maximum_authority_membership": 11,
    "reserve_percent_of_fee": 0,
    "network_percent_of_fee": 10000,
    "lifetime_referrer_percent_of_fee": 0,
    "cashback_vesting_period_seconds": 9999999,
    "cashback_vesting_threshold": 100000,
    "count_non_member_votes": true,
    "allow_non_member_whitelists": true,
    "witness_pay_per_block": 725,
    "worker_budget_per_day": 0,
    "max_predicate_opcode": 1,
    "fee_liquidation_threshold": "500000000000",
    "accounts_per_fee_scale": 1000,
    "account_fee_scale_bitshifts": 0,
    "max_authority_depth": 2,
    "witness_schedule_algorithm": 0,
    "min_round_delay": 10,
    "max_round_delay": 300,
    "min_time_per_commit_move": 15,
    "max_time_per_commit_move": 15,
    "min_time_per_reveal_move": 9,
    "max_time_per_reveal_move": 9,
    "rake_fee_percentage": 300,
    "maximum_registration_deadline": 2592000,
    "maximum_players_in_tournament": 256,
    "maximum_tournament_whitelist_length": 1000,
    "maximum_tournament_start_time_in_future": 5184000,
    "maximum_tournament_start_delay": 259200,
    "maximum_tournament_number_of_wins": 25,
    "extensions": {
      "betting_rake_fee_percentage": 100,
      "live_betting_delay_time": 0,
      "sweeps_distribution_percentage": 200,
      "sweeps_distribution_asset": "1.3.0",
      "sweeps_vesting_accumulator_account": "1.2.0",
      "gpos_period": 15552000,
      "gpos_subperiod": 2592000,
      "gpos_period_start": 1628632800,
      "gpos_vesting_lockin_period": 2592000,
      "rbac_max_permissions_per_account": 5,
      "rbac_max_account_authority_lifetime": 15552000,
      "rbac_max_authorities_per_permission": 15,
      "account_roles_max_per_account": 20,
      "account_roles_max_lifetime": 31536000,
      "son_vesting_amount": 5000000,
      "son_vesting_period": 172800,
      "son_pay_max": 20000000,
      "son_pay_time": 86400,
      "son_deregister_time": 43200,
      "son_heartbeat_frequency": 180,
      "son_down_time": 360,
      "son_bitcoin_min_tx_confirmations": 1,
      "son_account": "1.2.14358",
      "btc_asset": "1.3.22",
      "maximum_son_count": 7,
      "hbd_asset": "1.3.23",
      "hive_asset": "1.3.24"
    }
  },
  "next_available_vote_id": 126,
  "active_committee_members": [
    "1.5.15",
    "1.5.11",
    "1.5.13",
    "1.5.10",
    "1.5.7",
    "1.5.8",
    "1.5.0",
    "1.5.1",
    "1.5.2"
  ],
  "active_witnesses": [
    "1.6.31",
    "1.6.36",
    "1.6.38",
    "1.6.39",
    "1.6.72",
    "1.6.81",
    "1.6.86",
    "1.6.87",
    "1.6.89",
    "1.6.90",
    "1.6.92"
  ],
  "active_sons": [{
      "son_id": "1.33.7",
      "weight": 1,
      "signing_key": "PPY6MkvgJqbTLE5e2j4ySFXDhy4G6Tx957qYDYWMGC5QVDNAyAWvJ",
      "sidechain_public_keys": [[
          "bitcoin",
          "02f2be7c0784fdc2d379a3925e6408925c70f357e909c28ff6fb93e7f567b6af7f"
        ],[
          "hive",
          "hiltos-son"
        ]
      ]
    },{
      "son_id": "1.33.0",
      "weight": 1,
      "signing_key": "PPY7epqV6FYhyMi8tkt4A7BAp6A9L5Z1zS3bwdWTVxUbeiREPYBef",
      "sidechain_public_keys": [[
          "bitcoin",
          "0242e491c8955b66f2d1e5b683ea78cb77f874d23e685365155395500997929daf"
        ],[
          "hive",
          "coerced-son"
        ]
      ]
    },{
      "son_id": "1.33.11",
      "weight": 1,
      "signing_key": "PPY5rZvu8ij6QxPUCD5GsCV211WXGJNjb5oQREArxEQgBC5f7XYkB",
      "sidechain_public_keys": [[
          "bitcoin",
          "03a6e5b2b38d4d562f4c9fee983edb7555976c6450223203090da4765402f09197"
        ],[
          "hive",
          "brewster-son"
        ]
      ]
    },{
      "son_id": "1.33.12",
      "weight": 1,
      "signing_key": "PPY6yXbiCkTBJ2ekk1pgiHNR6TMp8WgRe81F6FFamfYx5QQgTmXAF",
      "sidechain_public_keys": [[
          "bitcoin",
          "03c1ffea0de5b23496bd7c1430e24be37a25487e9f1828a452fdcd95fcadf95264"
        ],[
          "hive",
          "muskelkater-son"
        ]
      ]
    },{
      "son_id": "1.33.13",
      "weight": 1,
      "signing_key": "PPY6uDJpKatiPfNtRH31XKRMziXUYsFBZvXvmCtGmVTbxoiQPZuAH",
      "sidechain_public_keys": [[
          "bitcoin",
          "02ce50b8836bb7b816015cad14736eb69d2e37de4fb417772f8d59ccf811179c92"
        ],[
          "hive",
          "huszar-son"
        ]
      ]
    },{
      "son_id": "1.33.14",
      "weight": 1,
      "signing_key": "PPY6jdmSpbZAqYpyHx5EK8qEUetnpjNgUs9WrAW3DXpFj1KAeFXRA",
      "sidechain_public_keys": [[
          "bitcoin",
          "03df4cfbe4aeb07bd1b9b633bf8053acef2e348d2de971ca513aa6f32fac8bfa93"
        ],[
          "hive",
          "yonder-son"
        ]
      ]
    },{
      "son_id": "1.33.15",
      "weight": 1,
      "signing_key": "PPY7iNUw6vRvG3coUehNQsrxsNRakhTYQBRgb14wDB3pnekUg2mEZ",
      "sidechain_public_keys": [[
          "bitcoin",
          "03bcebb9e68c63d08385f47f6d01db88eaa321da60d3b71f7465e0d2e0c640dee1"
        ],[
          "hive",
          "will-cook"
        ]
      ]
    }
  ]
}
locked >>> get_object 2.25.7
get_object 2.25.7
[{
    "id": "2.25.7",
    "owner": "1.33.7",
    "total_txs_signed": [[
        "hive",
        2
      ]
    ],
    "txs_signed": [[
        "hive",
        1
      ]
    ],
    "total_voted_time": 13924800,
    "total_downtime": 1240,
    "current_interval_downtime": 0,
    "last_down_timestamp": "2021-12-29T16:00:00",
    "last_active_timestamp": "2021-12-30T03:29:08",
    "deregistered_timestamp": "1970-01-01T00:00:00",
    "total_sidechain_txs_reported": [[
        "hive",
        1
      ]
    ],
    "sidechain_txs_reported": [[
        "hive",
        0
      ]
    ]
  }
]
locked >>> get_object 2.25.0
get_object 2.25.0
[{
    "id": "2.25.0",
    "owner": "1.33.0",
    "total_txs_signed": [],
    "txs_signed": [],
    "total_voted_time": 31075200,
    "total_downtime": 4874,
    "current_interval_downtime": 0,
    "last_down_timestamp": "2021-12-30T00:00:00",
    "last_active_timestamp": "2021-12-30T03:30:38",
    "deregistered_timestamp": "1970-01-01T00:00:00",
    "total_sidechain_txs_reported": [],
    "sidechain_txs_reported": []
  }
]
locked >>> get_object 2.25.11
get_object 2.25.11
[{
    "id": "2.25.11",
    "owner": "1.33.11",
    "total_txs_signed": [[
        "hive",
        2
      ]
    ],
    "txs_signed": [[
        "hive",
        1
      ]
    ],
    "total_voted_time": 770400,
    "total_downtime": 0,
    "current_interval_downtime": 0,
    "last_down_timestamp": "1970-01-01T00:00:00",
    "last_active_timestamp": "2021-12-30T03:29:08",
    "deregistered_timestamp": "1970-01-01T00:00:00",
    "total_sidechain_txs_reported": [[
        "hive",
        1
      ]
    ],
    "sidechain_txs_reported": [[
        "hive",
        0
      ]
    ]
  }
]
locked >>> get_object 2.25.12
get_object 2.25.12
^[[A[{
    "id": "2.25.12",
    "owner": "1.33.12",
    "total_txs_signed": [[
        "hive",
        2
      ]
    ],
    "txs_signed": [[
        "hive",
        1
      ]
    ],
    "total_voted_time": 770400,
    "total_downtime": 1242,
    "current_interval_downtime": 0,
    "last_down_timestamp": "2021-12-29T16:00:00",
    "last_active_timestamp": "2021-12-30T03:29:08",
    "deregistered_timestamp": "1970-01-01T00:00:00",
    "total_sidechain_txs_reported": [[
        "hive",
        1
      ]
    ],
    "sidechain_txs_reported": [[
        "hive",
        0
      ]
    ]
  }
]
locked >>> get_object 2.25.13
get_object 2.25.13
^[[A[{
    "id": "2.25.13",
    "owner": "1.33.13",
    "total_txs_signed": [[
        "hive",
        1
      ]
    ],
    "txs_signed": [[
        "hive",
        0
      ]
    ],
    "total_voted_time": 770400,
    "total_downtime": 1247,
    "current_interval_downtime": 0,
    "last_down_timestamp": "2021-12-29T16:00:00",
    "last_active_timestamp": "2021-12-30T03:29:08",
    "deregistered_timestamp": "1970-01-01T00:00:00",
    "total_sidechain_txs_reported": [[
        "hive",
        1
      ]
    ],
    "sidechain_txs_reported": [[
        "hive",
        0
      ]
    ]
  }
]
locked >>> get_object 2.25.14
get_object 2.25.14
[{
    "id": "2.25.14",
    "owner": "1.33.14",
    "total_txs_signed": [[
        "hive",
        1
      ]
    ],
    "txs_signed": [[
        "hive",
        1
      ]
    ],
    "total_voted_time": 770400,
    "total_downtime": 1249,
    "current_interval_downtime": 0,
    "last_down_timestamp": "2021-12-29T16:00:00",
    "last_active_timestamp": "2021-12-30T03:29:08",
    "deregistered_timestamp": "1970-01-01T00:00:00",
    "total_sidechain_txs_reported": [[
        "hive",
        1
      ]
    ],
    "sidechain_txs_reported": [[
        "hive",
        0
      ]
    ]
  }
]
locked >>> get_object 2.25.15
get_object 2.25.15
[{
    "id": "2.25.15",
    "owner": "1.33.15",
    "total_txs_signed": [[
        "hive",
        2
      ]
    ],
    "txs_signed": [[
        "hive",
        1
      ]
    ],
    "total_voted_time": 770400,
    "total_downtime": 1429,
    "current_interval_downtime": 0,
    "last_down_timestamp": "2021-12-29T16:00:00",
    "last_active_timestamp": "2021-12-30T03:29:08",
    "deregistered_timestamp": "1970-01-01T00:00:00",
    "total_sidechain_txs_reported": [[
        "hive",
        1
      ]
    ],
    "sidechain_txs_reported": [[
        "hive",
        0
      ]
    ]
  }
]
locked >>>

SON network is up, nodes are sending heartbeats and there is no crashing: ``` locked >>> get_global_properties get_global_properties { "id": "2.0.0", "parameters": { "current_fees": { "parameters": [[ 0,{ "fee": 1000, "price_per_kbyte": 1000 } ],[ 1,{ "fee": 50 } ],[ 2,{ "fee": 0 } ],[ 3,{ "fee": "500000000000" } ],[ 4,{} ],[ 5,{ "basic_fee": 500, "premium_fee": 250000, "price_per_kbyte": 1000 } ],[ 6,{ "fee": 100, "price_per_kbyte": 1000 } ],[ 7,{ "fee": 1000 } ],[ 8,{ "membership_annual_fee": "500000000000", "membership_lifetime_fee": 500000 } ],[ 9,{ "fee": 200000 } ],[ 10,{ "symbol3": "500000000000", "symbol4": "500000000000", "long_symbol": 5000000, "price_per_kbyte": 1000 } ],[ 11,{ "fee": 100000, "price_per_kbyte": 100 } ],[ 12,{ "fee": "500000000000" } ],[ 13,{ "fee": "500000000000" } ],[ 14,{ "fee": 1000, "price_per_kbyte": 1000 } ],[ 15,{ "fee": 1000 } ],[ 16,{ "fee": 1000 } ],[ 17,{ "fee": "500000000000" } ],[ 18,{ "fee": "500000000000" } ],[ 19,{ "fee": "500000000000" } ],[ 20,{ "fee": 800000 } ],[ 21,{ "fee": 50000 } ],[ 22,{ "fee": 1000, "price_per_kbyte": 1000 } ],[ 23,{ "fee": 1000, "price_per_kbyte": 1000 } ],[ 24,{ "fee": 0 } ],[ 25,{ "fee": 3000 } ],[ 26,{ "fee": 1000 } ],[ 27,{ "fee": 1000, "price_per_kbyte": 1000 } ],[ 28,{ "fee": 0 } ],[ 29,{ "fee": 100000 } ],[ 30,{ "fee": 50000 } ],[ 31,{ "fee": 1000 } ],[ 32,{ "fee": 100000 } ],[ 33,{ "fee": 1000 } ],[ 34,{ "fee": "500000000000" } ],[ 35,{ "fee": 1000, "price_per_kbyte": 1000 } ],[ 36,{ "fee": 1000 } ],[ 37,{} ],[ 38,{ "fee": 20000, "price_per_kbyte": 1000 } ],[ 39,{ "fee": 1000, "price_per_output": 1000 } ],[ 40,{ "fee": 1000, "price_per_output": 1000 } ],[ 41,{ "fee": 1000 } ],[ 42,{} ],[ 43,{ "fee": 3000 } ],[ 44,{} ],[ 45,{ "fee": 1000 } ],[ 46,{ "fee": 5000 } ],[ 47,{ "fee": 0 } ],[ 48,{ "fee": 100000 } ],[ 49,{ "distribution_base_fee": 0, "distribution_fee_per_holder": 0 } ],[ 52,{ "fee": 1000 } ],[ 53,{ "fee": 1000 } ],[ 54,{ "fee": 1000 } ],[ 55,{ "fee": 1000 } ],[ 56,{ "fee": 1 } ],[ 57,{ "fee": 1 } ],[ 58,{ "fee": 1000 } ],[ 59,{ "fee": 1000 } ],[ 60,{ "fee": 1 } ],[ 61,{ "fee": 1 } ],[ 62,{ "fee": 1 } ],[ 63,{ "fee": 1 } ],[ 66,{ "fee": 0 } ],[ 68,{ "fee": 0 } ],[ 70,{ "fee": 1 } ],[ 71,{ "fee": 1 } ],[ 72,{ "fee": 1 } ],[ 73,{ "fee": 1000 } ],[ 74,{ "fee": 1000 } ] ], "scale": 10000 }, "block_interval": 3, "maintenance_interval": 3600, "maintenance_skip_slots": 3, "committee_proposal_review_period": 3600, "maximum_transaction_size": 99999, "maximum_block_size": 2000000, "maximum_time_until_expiration": 86400, "maximum_proposal_lifetime": 2419200, "maximum_asset_whitelist_authorities": 10, "maximum_asset_feed_publishers": 10, "maximum_witness_count": 101, "maximum_committee_count": 33, "maximum_authority_membership": 11, "reserve_percent_of_fee": 0, "network_percent_of_fee": 10000, "lifetime_referrer_percent_of_fee": 0, "cashback_vesting_period_seconds": 9999999, "cashback_vesting_threshold": 100000, "count_non_member_votes": true, "allow_non_member_whitelists": true, "witness_pay_per_block": 725, "worker_budget_per_day": 0, "max_predicate_opcode": 1, "fee_liquidation_threshold": "500000000000", "accounts_per_fee_scale": 1000, "account_fee_scale_bitshifts": 0, "max_authority_depth": 2, "witness_schedule_algorithm": 0, "min_round_delay": 10, "max_round_delay": 300, "min_time_per_commit_move": 15, "max_time_per_commit_move": 15, "min_time_per_reveal_move": 9, "max_time_per_reveal_move": 9, "rake_fee_percentage": 300, "maximum_registration_deadline": 2592000, "maximum_players_in_tournament": 256, "maximum_tournament_whitelist_length": 1000, "maximum_tournament_start_time_in_future": 5184000, "maximum_tournament_start_delay": 259200, "maximum_tournament_number_of_wins": 25, "extensions": { "betting_rake_fee_percentage": 100, "live_betting_delay_time": 0, "sweeps_distribution_percentage": 200, "sweeps_distribution_asset": "1.3.0", "sweeps_vesting_accumulator_account": "1.2.0", "gpos_period": 15552000, "gpos_subperiod": 2592000, "gpos_period_start": 1628632800, "gpos_vesting_lockin_period": 2592000, "rbac_max_permissions_per_account": 5, "rbac_max_account_authority_lifetime": 15552000, "rbac_max_authorities_per_permission": 15, "account_roles_max_per_account": 20, "account_roles_max_lifetime": 31536000, "son_vesting_amount": 5000000, "son_vesting_period": 172800, "son_pay_max": 20000000, "son_pay_time": 86400, "son_deregister_time": 43200, "son_heartbeat_frequency": 180, "son_down_time": 360, "son_bitcoin_min_tx_confirmations": 1, "son_account": "1.2.14358", "btc_asset": "1.3.22", "maximum_son_count": 7, "hbd_asset": "1.3.23", "hive_asset": "1.3.24" } }, "next_available_vote_id": 126, "active_committee_members": [ "1.5.15", "1.5.11", "1.5.13", "1.5.10", "1.5.7", "1.5.8", "1.5.0", "1.5.1", "1.5.2" ], "active_witnesses": [ "1.6.31", "1.6.36", "1.6.38", "1.6.39", "1.6.72", "1.6.81", "1.6.86", "1.6.87", "1.6.89", "1.6.90", "1.6.92" ], "active_sons": [{ "son_id": "1.33.7", "weight": 1, "signing_key": "PPY6MkvgJqbTLE5e2j4ySFXDhy4G6Tx957qYDYWMGC5QVDNAyAWvJ", "sidechain_public_keys": [[ "bitcoin", "02f2be7c0784fdc2d379a3925e6408925c70f357e909c28ff6fb93e7f567b6af7f" ],[ "hive", "hiltos-son" ] ] },{ "son_id": "1.33.0", "weight": 1, "signing_key": "PPY7epqV6FYhyMi8tkt4A7BAp6A9L5Z1zS3bwdWTVxUbeiREPYBef", "sidechain_public_keys": [[ "bitcoin", "0242e491c8955b66f2d1e5b683ea78cb77f874d23e685365155395500997929daf" ],[ "hive", "coerced-son" ] ] },{ "son_id": "1.33.11", "weight": 1, "signing_key": "PPY5rZvu8ij6QxPUCD5GsCV211WXGJNjb5oQREArxEQgBC5f7XYkB", "sidechain_public_keys": [[ "bitcoin", "03a6e5b2b38d4d562f4c9fee983edb7555976c6450223203090da4765402f09197" ],[ "hive", "brewster-son" ] ] },{ "son_id": "1.33.12", "weight": 1, "signing_key": "PPY6yXbiCkTBJ2ekk1pgiHNR6TMp8WgRe81F6FFamfYx5QQgTmXAF", "sidechain_public_keys": [[ "bitcoin", "03c1ffea0de5b23496bd7c1430e24be37a25487e9f1828a452fdcd95fcadf95264" ],[ "hive", "muskelkater-son" ] ] },{ "son_id": "1.33.13", "weight": 1, "signing_key": "PPY6uDJpKatiPfNtRH31XKRMziXUYsFBZvXvmCtGmVTbxoiQPZuAH", "sidechain_public_keys": [[ "bitcoin", "02ce50b8836bb7b816015cad14736eb69d2e37de4fb417772f8d59ccf811179c92" ],[ "hive", "huszar-son" ] ] },{ "son_id": "1.33.14", "weight": 1, "signing_key": "PPY6jdmSpbZAqYpyHx5EK8qEUetnpjNgUs9WrAW3DXpFj1KAeFXRA", "sidechain_public_keys": [[ "bitcoin", "03df4cfbe4aeb07bd1b9b633bf8053acef2e348d2de971ca513aa6f32fac8bfa93" ],[ "hive", "yonder-son" ] ] },{ "son_id": "1.33.15", "weight": 1, "signing_key": "PPY7iNUw6vRvG3coUehNQsrxsNRakhTYQBRgb14wDB3pnekUg2mEZ", "sidechain_public_keys": [[ "bitcoin", "03bcebb9e68c63d08385f47f6d01db88eaa321da60d3b71f7465e0d2e0c640dee1" ],[ "hive", "will-cook" ] ] } ] } locked >>> get_object 2.25.7 get_object 2.25.7 [{ "id": "2.25.7", "owner": "1.33.7", "total_txs_signed": [[ "hive", 2 ] ], "txs_signed": [[ "hive", 1 ] ], "total_voted_time": 13924800, "total_downtime": 1240, "current_interval_downtime": 0, "last_down_timestamp": "2021-12-29T16:00:00", "last_active_timestamp": "2021-12-30T03:29:08", "deregistered_timestamp": "1970-01-01T00:00:00", "total_sidechain_txs_reported": [[ "hive", 1 ] ], "sidechain_txs_reported": [[ "hive", 0 ] ] } ] locked >>> get_object 2.25.0 get_object 2.25.0 [{ "id": "2.25.0", "owner": "1.33.0", "total_txs_signed": [], "txs_signed": [], "total_voted_time": 31075200, "total_downtime": 4874, "current_interval_downtime": 0, "last_down_timestamp": "2021-12-30T00:00:00", "last_active_timestamp": "2021-12-30T03:30:38", "deregistered_timestamp": "1970-01-01T00:00:00", "total_sidechain_txs_reported": [], "sidechain_txs_reported": [] } ] locked >>> get_object 2.25.11 get_object 2.25.11 [{ "id": "2.25.11", "owner": "1.33.11", "total_txs_signed": [[ "hive", 2 ] ], "txs_signed": [[ "hive", 1 ] ], "total_voted_time": 770400, "total_downtime": 0, "current_interval_downtime": 0, "last_down_timestamp": "1970-01-01T00:00:00", "last_active_timestamp": "2021-12-30T03:29:08", "deregistered_timestamp": "1970-01-01T00:00:00", "total_sidechain_txs_reported": [[ "hive", 1 ] ], "sidechain_txs_reported": [[ "hive", 0 ] ] } ] locked >>> get_object 2.25.12 get_object 2.25.12 ^[[A[{ "id": "2.25.12", "owner": "1.33.12", "total_txs_signed": [[ "hive", 2 ] ], "txs_signed": [[ "hive", 1 ] ], "total_voted_time": 770400, "total_downtime": 1242, "current_interval_downtime": 0, "last_down_timestamp": "2021-12-29T16:00:00", "last_active_timestamp": "2021-12-30T03:29:08", "deregistered_timestamp": "1970-01-01T00:00:00", "total_sidechain_txs_reported": [[ "hive", 1 ] ], "sidechain_txs_reported": [[ "hive", 0 ] ] } ] locked >>> get_object 2.25.13 get_object 2.25.13 ^[[A[{ "id": "2.25.13", "owner": "1.33.13", "total_txs_signed": [[ "hive", 1 ] ], "txs_signed": [[ "hive", 0 ] ], "total_voted_time": 770400, "total_downtime": 1247, "current_interval_downtime": 0, "last_down_timestamp": "2021-12-29T16:00:00", "last_active_timestamp": "2021-12-30T03:29:08", "deregistered_timestamp": "1970-01-01T00:00:00", "total_sidechain_txs_reported": [[ "hive", 1 ] ], "sidechain_txs_reported": [[ "hive", 0 ] ] } ] locked >>> get_object 2.25.14 get_object 2.25.14 [{ "id": "2.25.14", "owner": "1.33.14", "total_txs_signed": [[ "hive", 1 ] ], "txs_signed": [[ "hive", 1 ] ], "total_voted_time": 770400, "total_downtime": 1249, "current_interval_downtime": 0, "last_down_timestamp": "2021-12-29T16:00:00", "last_active_timestamp": "2021-12-30T03:29:08", "deregistered_timestamp": "1970-01-01T00:00:00", "total_sidechain_txs_reported": [[ "hive", 1 ] ], "sidechain_txs_reported": [[ "hive", 0 ] ] } ] locked >>> get_object 2.25.15 get_object 2.25.15 [{ "id": "2.25.15", "owner": "1.33.15", "total_txs_signed": [[ "hive", 2 ] ], "txs_signed": [[ "hive", 1 ] ], "total_voted_time": 770400, "total_downtime": 1429, "current_interval_downtime": 0, "last_down_timestamp": "2021-12-29T16:00:00", "last_active_timestamp": "2021-12-30T03:29:08", "deregistered_timestamp": "1970-01-01T00:00:00", "total_sidechain_txs_reported": [[ "hive", 1 ] ], "sidechain_txs_reported": [[ "hive", 0 ] ] } ] locked >>> ```

serkixenos (Migrated from gitlab.com) closed this issue

2022-01-17 21:00:24 +00:00

serkixenos commented

2022-03-18 20:20:35 +00:00

(Migrated from gitlab.com)

mentioned in issue peerplays-dex#529

witness_node program is crashing on active SON node (Mainnet) #168

Description:

Relevant info:

Server details

Software details

Configuration details

Logs

Expected results:

Actual results:

Server details

Logs

default.log

p2p.log