after stopping witness node peerplays02 other witness nodes crashed.

prandnum commented

2023-01-13 15:17:42 +00:00

(Migrated from gitlab.com)

Steps:

After stopping witness node peerplays02 other witness nodes crashed.

kiran@warpspeed:~/PBSA/06012023/peerplays-utils/peerplays-qa-environment$ docker ps -a
CONTAINER ID   IMAGE                            COMMAND                  CREATED             STATUS                                    PORTS                                                                                                                                                   NAMES
f7ba51b8a3fd   peerplays05                      "./witness_node -d .…"   45 minutes ago      Up 45 minutes                             22/tcp, 0.0.0.0:8094->8090/tcp, :::8094->8090/tcp, 0.0.0.0:9781->9777/tcp, :::9781->9777/tcp                                                            peerplays-qa-environment_peerplays05_1
a413e77f303b   peerplays10                      "./witness_node -d .…"   45 minutes ago      Up 8 seconds                              22/tcp, 0.0.0.0:8099->8090/tcp, :::8099->8090/tcp, 0.0.0.0:9786->9777/tcp, :::9786->9777/tcp                                                            peerplays-qa-environment_peerplays10_1
16b87407ac40   peerplays14                      "./witness_node -d .…"   45 minutes ago      Up 7 seconds                              22/tcp, 0.0.0.0:8103->8090/tcp, :::8103->8090/tcp, 0.0.0.0:9790->9777/tcp, :::9790->9777/tcp                                                            peerplays-qa-environment_peerplays14_1
fc00fe7ec784   peerplays06                      "./witness_node -d .…"   45 minutes ago      Up 8 seconds                              22/tcp, 0.0.0.0:8095->8090/tcp, :::8095->8090/tcp, 0.0.0.0:9782->9777/tcp, :::9782->9777/tcp                                                            peerplays-qa-environment_peerplays06_1
f366d30cf234   peerplays12                      "./witness_node -d .…"   45 minutes ago      Up 19 seconds                             22/tcp, 0.0.0.0:8101->8090/tcp, :::8101->8090/tcp, 0.0.0.0:9788->9777/tcp, :::9788->9777/tcp                                                            peerplays-qa-environment_peerplays12_1
5eb751b3e493   peerplays07                      "./witness_node -d .…"   45 minutes ago      Up 45 minutes                             22/tcp, 0.0.0.0:8096->8090/tcp, :::8096->8090/tcp, 0.0.0.0:9783->9777/tcp, :::9783->9777/tcp                                                            peerplays-qa-environment_peerplays07_1
c736743f669b   peerplays04                      "./witness_node -d .…"   45 minutes ago      Up 45 minutes                             22/tcp, 0.0.0.0:8093->8090/tcp, :::8093->8090/tcp, 0.0.0.0:9780->9777/tcp, :::9780->9777/tcp                                                            peerplays-qa-environment_peerplays04_1
a781598646cd   peerplays03                      "./witness_node -d .…"   45 minutes ago      Up 45 minutes                             22/tcp, 0.0.0.0:8092->8090/tcp, :::8092->8090/tcp, 0.0.0.0:9779->9777/tcp, :::9779->9777/tcp                                                            peerplays-qa-environment_peerplays03_1
26424f96836e   peerplays08                      "./witness_node -d .…"   45 minutes ago      Restarting (139) Less than a second ago                                                                                                                                                           peerplays-qa-environment_peerplays08_1
45b8579384e4   peerplays01                      "./witness_node -d .…"   45 minutes ago      Up 45 minutes                             0.0.0.0:8090->8090/tcp, :::8090->8090/tcp, 22/tcp, 0.0.0.0:9777->9777/tcp, :::9777->9777/tcp                                                            peerplays-qa-environment_peerplays01_1
6c11ee69e16b   peerplays11                      "./witness_node -d .…"   45 minutes ago      Up 20 seconds                             22/tcp, 0.0.0.0:8100->8090/tcp, :::8100->8090/tcp, 0.0.0.0:9787->9777/tcp, :::9787->9777/tcp                                                            peerplays-qa-environment_peerplays11_1
70d9996e2e0d   peerplays15                      "./witness_node -d .…"   45 minutes ago      Up 45 minutes                             22/tcp, 0.0.0.0:8104->8090/tcp, :::8104->8090/tcp, 0.0.0.0:9791->9777/tcp, :::9791->9777/tcp                                                            peerplays-qa-environment_peerplays15_1
2c084dd3c8cf   peerplays09                      "./witness_node -d .…"   45 minutes ago      Up 45 minutes                             22/tcp, 0.0.0.0:8098->8090/tcp, :::8098->8090/tcp, 0.0.0.0:9785->9777/tcp, :::9785->9777/tcp                                                            peerplays-qa-environment_peerplays09_1
0d3b2dbcdb29   peerplays16                      "./witness_node -d .…"   45 minutes ago      Up 45 minutes                             22/tcp, 0.0.0.0:8105->8090/tcp, :::8105->8090/tcp, 0.0.0.0:9792->9777/tcp, :::9792->9777/tcp                                                            peerplays-qa-environment_peerplays16_1
18d4edab2023   peerplays02                      "./witness_node -d .…"   45 minutes ago      Exited (137) 32 minutes ago                                                                                                                                                                       peerplays-qa-environment_peerplays02_1
e2c6788d5c73   peerplays13                      "./witness_node -d .…"   45 minutes ago      Up 45 minutes                             22/tcp, 0.0.0.0:8102->8090/tcp, :::8102->8090/tcp, 0.0.0.0:9789->9777/tcp, :::9789->9777/tcp                                                            peerplays-qa-environment_peerplays13_1
e694f0b5083c   peerplays-base                   "bash"                   45 minutes ago      Exited (0) 45 minutes ago                                                                                                                                                                         peerplays-qa-environment_peerplays-base_1
40f563d58db9   hive-for-peerplays               "./hived -d ./hived_…"   59 minutes ago      Up 59 minutes                             22/tcp, 0.0.0.0:28090-28091->28090-28091/tcp, :::28090-28091->28090-28091/tcp                                                                           peerplays-qa-environment_hive-for-peerplays_1
7eeb00febe10   ethereum-for-peerplays           "./geth --config ./n…"   About an hour ago   Up About an hour                          0.0.0.0:6060->6060/tcp, :::6060->6060/tcp, 22/tcp, 0.0.0.0:8545->8545/tcp, :::8545->8545/tcp, 8546/tcp, 0.0.0.0:30303->30303/tcp, :::30303->30303/tcp   peerplays-qa-environment_ethereum-for-peerplays_1
467391e4c22f   libbitcoinserver-for-peerplays   "/bin/sh runserver.sh"   About an hour ago   Up About an hour                          0.0.0.0:9091->9091/tcp, :::9091->9091/tcp, 22/tcp, 0.0.0.0:9093->9093/tcp, :::9093->9093/tcp                                                            peerplays-qa-environment_libbitcoinserver-for-peerplays_1
604f2e13e799   bitcoin-for-peerplays            "./bitcoind -datadir…"   About an hour ago   Up About an hour                          0.0.0.0:8332->8332/tcp, :::8332->8332/tcp, 0.0.0.0:11111->11111/tcp, :::11111->11111/tcp, 22/tcp, 0.0.0.0:18444->18444/tcp, :::18444->18444/tcp         peerplays-qa-environment_bitcoin-for-peerplays_1
0a99747528d5   ubuntu-for-peerplays             "bash"                   About an hour ago   Exited (0) 45 minutes ago                                                                                                                                                                         peerplays-qa-environment_ubuntu-for-peerplays_1
ad0ee593076b   e28a50f651f9                     "bash"                   4 days ago          Exited (0) 4 days ago                                                                                                                                                                             clever_hodgkin
2b1ca83f2f7c   ubuntu:18.04                     "bash"                   4 days ago          Exited (0) 4 days ago                                                                                                                                                                             inspiring_tharp
kiran@warpspeed:~/PBSA/06012023/peerplays-utils/peerplays-qa-environment$

kiran@warpspeed:/home$ sudo coredumpctl debug 4999
           PID: 4999 (witness_node)
           UID: 0 (root)
           GID: 0 (root)
        Signal: 11 (SEGV)
     Timestamp: Fri 2023-01-13 00:17:36 +0530 (9min ago)
  Command Line: ./witness_node -d ./witness_node_data_dir --replay-blockchain
    Executable: /home/peerplays/src/peerplays/build/programs/witness_node/witness_node
 Control Group: /docker/16b87407ac4019e43c2c16987ae405a3af7d52c9256235aded38ac02c6acf5ff
         Slice: -.slice
       Boot ID: 4ced3cb6f68d4f38b5ad5574dd873a6e
    Machine ID: a388625228614d18b5373ad1be39a93f
      Hostname: warpspeed
       Storage: /var/lib/systemd/coredump/core.witness_node.0.4ced3cb6f68d4f38b5ad5574dd873a6e.4999.1673549256000000000000.lz4
       Message: Process 4999 (witness_node) of user 0 dumped core.
                
                Stack trace of thread 15:
                #0  0x00007f4f8c6ba941 __GI_abort (libc.so.6 + 0x22941)
                #1  0x00007f4f8c6ba729 __assert_fail_base (libc.so.6 + 0x22729)
                #2  0x00007f4f8c6cbfd6 __GI___assert_fail (libc.so.6 + 0x33fd6)
                #3  0x000055cea30ef588 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x57cb588)
                #4  0x000055cea310cb13 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x57e8b13)
                #5  0x000055cea3259c26 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x5935c26)
                #6  0x000055cea326071b n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x593c71b)
                #7  0x000055cea3260031 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x593c031)
                #8  0x000055cea3265a0e n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x5941a0e)
                #9  0x000055cea45b7138 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6c93138)
                #10 0x000055cea45b70b4 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6c930b4)
                #11 0x000055cea45a8c62 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6c84c62)
                #12 0x000055cea45a9011 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6c85011)
                #13 0x000055cea45a85e1 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6c845e1)
                #14 0x000055cea481d96f n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6ef996f)

GNU gdb (Ubuntu 9.2-0ubuntu1~20.04.1) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/peerplays/src/peerplays/build/programs/witness_node/witness_node...
[New LWP 15]
[New LWP 9]
[New LWP 12]
[New LWP 13]
[New LWP 7]
[New LWP 19]
[New LWP 16]
[New LWP 2442]
[New LWP 18]
[New LWP 10]
[New LWP 17]
[New LWP 2441]
[New LWP 2443]
[New LWP 8]
[New LWP 1]
[New LWP 11]
[New LWP 14]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `./witness_node -d ./witness_node_data_dir --replay-blockchain'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  __GI_abort () at abort.c:107
107	abort.c: No such file or directory.
[Current thread is 1 (Thread 0x7f4f727fc700 (LWP 15))]
(gdb) bt
#0  __GI_abort () at abort.c:107
#1  0x00007f4f8c6ba729 in __assert_fail_base (fmt=0x7f4f8c850588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x55cea4cb8458 "_closing_connections.find(peer_to_delete) == _closing_connections.end()", file=0x55cea4cb69e8 "/home/peerplays/src/peerplays/libraries/net/node.cpp", line=1654, 
    function=<optimized out>) at assert.c:92
#2  0x00007f4f8c6cbfd6 in __GI___assert_fail (assertion=0x55cea4cb8458 "_closing_connections.find(peer_to_delete) == _closing_connections.end()", file=0x55cea4cb69e8 "/home/peerplays/src/peerplays/libraries/net/node.cpp", line=1654, 
    function=0x55cea4cb8360 "void graphene::net::detail::node_impl::schedule_peer_for_deletion(const peer_connection_ptr&)") at assert.c:101
#3  0x000055cea30ef588 in graphene::net::detail::node_impl::schedule_peer_for_deletion (this=0x55cea6cae5a0, peer_to_delete=std::shared_ptr<class graphene::net::peer_connection> (use count 4, weak count 2) = {...}) at /home/peerplays/src/peerplays/libraries/net/node.cpp:1654
#4  0x000055cea310cb13 in graphene::net::detail::node_impl::on_connection_closed (this=0x55cea6cae5a0, originating_peer=0x7f4f5c07ded0) at /home/peerplays/src/peerplays/libraries/net/node.cpp:3106
#5  0x000055cea3259c26 in graphene::net::peer_connection::on_connection_closed (this=0x7f4f5c07ded0, originating_connection=0x7f4f5c07df00) at /home/peerplays/src/peerplays/libraries/net/peer_connection.cpp:283
#6  0x000055cea326071b in graphene::net::detail::message_oriented_connection_impl::read_loop (this=0x7f4f5c07e2c0) at /home/peerplays/src/peerplays/libraries/net/message_oriented_connection.cpp:241
#7  0x000055cea3260031 in graphene::net::detail::message_oriented_connection_impl::<lambda()>::operator()(void) const (__closure=0x7f4f5c159c08) at /home/peerplays/src/peerplays/libraries/net/message_oriented_connection.cpp:134
#8  0x000055cea3265a0e in fc::detail::void_functor_run<graphene::net::detail::message_oriented_connection_impl::connect_to(const fc::ip::endpoint&)::<lambda()> >::run(void *, void *) (functor=0x7f4f5c159c08, prom=0x7f4f5c159c00) at /home/peerplays/src/peerplays/libraries/fc/include/fc/thread/task.hpp:83
#9  0x000055cea45b7138 in fc::task_base::run_impl (this=0x7f4f5c159c10) at /home/peerplays/src/peerplays/libraries/fc/src/thread/task.cpp:43
#10 0x000055cea45b70b4 in fc::task_base::run (this=0x7f4f5c159c10) at /home/peerplays/src/peerplays/libraries/fc/src/thread/task.cpp:32
#11 0x000055cea45a8c62 in fc::thread_d::run_next_task (this=0x7f4f5c000b60) at /home/peerplays/src/peerplays/libraries/fc/src/thread/thread_d.hpp:520
#12 0x000055cea45a9011 in fc::thread_d::process_tasks (this=0x7f4f5c000b60) at /home/peerplays/src/peerplays/libraries/fc/src/thread/thread_d.hpp:568
#13 0x000055cea45a85e1 in fc::thread_d::start_process_tasks (my=...) at /home/peerplays/src/peerplays/libraries/fc/src/thread/thread_d.hpp:499
#14 0x000055cea481d96f in make_fcontext ()
#15 0x0000000000000000 in ?? ()
(gdb)

kiran@warpspeed:/home$ sudo coredumpctl debug 1935140
           PID: 1935140 (witness_node)
           UID: 0 (root)
           GID: 0 (root)
        Signal: 11 (SEGV)
     Timestamp: Fri 2023-01-13 20:10:39 +0530 (26min ago)
  Command Line: ./witness_node -d ./witness_node_data_dir --replay-blockchain
    Executable: /home/peerplays/src/peerplays/build/programs/witness_node/witness_node
 Control Group: /docker/2c084dd3c8cfb2d4b01898de6e02665b9594bccd75aabd1d080ff4db2ee81847
         Slice: -.slice
       Boot ID: 4ced3cb6f68d4f38b5ad5574dd873a6e
    Machine ID: a388625228614d18b5373ad1be39a93f
      Hostname: warpspeed
       Storage: /var/lib/systemd/coredump/core.witness_node.0.4ced3cb6f68d4f38b5ad5574dd873a6e.1935140.1673620839000000000000.lz4
       Message: Process 1935140 (witness_node) of user 0 dumped core.
                
                Stack trace of thread 15:
                #0  0x00007fe937fb3941 __GI_abort (libc.so.6 + 0x22941)
                #1  0x00007fe937fb3729 __assert_fail_base (libc.so.6 + 0x22729)
                #2  0x00007fe937fc4fd6 __GI___assert_fail (libc.so.6 + 0x33fd6)
                #3  0x00005634786d4588 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x57cb588)
                #4  0x00005634786f1b13 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x57e8b13)
                #5  0x000056347883ec26 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x5935c26)
                #6  0x000056347884571b n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x593c71b)
                #7  0x0000563478845031 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x593c031)
                #8  0x000056347884aa0e n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x5941a0e)
                #9  0x0000563479b9c138 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6c93138)
                #10 0x0000563479b9c0b4 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6c930b4)
                #11 0x0000563479b8dc62 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6c84c62)
                #12 0x0000563479b8e011 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6c85011)
                #13 0x0000563479b8d5e1 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6c845e1)
                #14 0x0000563479e0296f n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6ef996f)

GNU gdb (Ubuntu 9.2-0ubuntu1~20.04.1) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/peerplays/src/peerplays/build/programs/witness_node/witness_node...
[New LWP 15]
[New LWP 10]
[New LWP 8]
[New LWP 9]
[New LWP 11]
[New LWP 14]
[New LWP 16]
[New LWP 1]
[New LWP 7]
[New LWP 13]
[New LWP 19]
[New LWP 12]
[New LWP 17]
[New LWP 18]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `./witness_node -d ./witness_node_data_dir --replay-blockchain'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  __GI_abort () at abort.c:107
107	abort.c: No such file or directory.
[Current thread is 1 (Thread 0x7fe91dffb700 (LWP 15))]
(gdb) bt
#0  __GI_abort () at abort.c:107
#1  0x00007fe937fb3729 in __assert_fail_base (fmt=0x7fe938149588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x56347a29d458 "_closing_connections.find(peer_to_delete) == _closing_connections.end()", file=0x56347a29b9e8 "/home/peerplays/src/peerplays/libraries/net/node.cpp", line=1654, 
    function=<optimized out>) at assert.c:92
#2  0x00007fe937fc4fd6 in __GI___assert_fail (assertion=0x56347a29d458 "_closing_connections.find(peer_to_delete) == _closing_connections.end()", file=0x56347a29b9e8 "/home/peerplays/src/peerplays/libraries/net/node.cpp", line=1654, 
    function=0x56347a29d360 "void graphene::net::detail::node_impl::schedule_peer_for_deletion(const peer_connection_ptr&)") at assert.c:101
#3  0x00005634786d4588 in graphene::net::detail::node_impl::schedule_peer_for_deletion (this=0x56347ed06780, peer_to_delete=std::shared_ptr<class graphene::net::peer_connection> (use count 4, weak count 2) = {...}) at /home/peerplays/src/peerplays/libraries/net/node.cpp:1654
#4  0x00005634786f1b13 in graphene::net::detail::node_impl::on_connection_closed (this=0x56347ed06780, originating_peer=0x7fe908133700) at /home/peerplays/src/peerplays/libraries/net/node.cpp:3106
#5  0x000056347883ec26 in graphene::net::peer_connection::on_connection_closed (this=0x7fe908133700, originating_connection=0x7fe908133730) at /home/peerplays/src/peerplays/libraries/net/peer_connection.cpp:283
#6  0x000056347884571b in graphene::net::detail::message_oriented_connection_impl::read_loop (this=0x7fe908133af0) at /home/peerplays/src/peerplays/libraries/net/message_oriented_connection.cpp:241
#7  0x0000563478845031 in graphene::net::detail::message_oriented_connection_impl::<lambda()>::operator()(void) const (__closure=0x7fe90814cad8) at /home/peerplays/src/peerplays/libraries/net/message_oriented_connection.cpp:134
#8  0x000056347884aa0e in fc::detail::void_functor_run<graphene::net::detail::message_oriented_connection_impl::connect_to(const fc::ip::endpoint&)::<lambda()> >::run(void *, void *) (functor=0x7fe90814cad8, prom=0x7fe90814cad0) at /home/peerplays/src/peerplays/libraries/fc/include/fc/thread/task.hpp:83
#9  0x0000563479b9c138 in fc::task_base::run_impl (this=0x7fe90814cae0) at /home/peerplays/src/peerplays/libraries/fc/src/thread/task.cpp:43
#10 0x0000563479b9c0b4 in fc::task_base::run (this=0x7fe90814cae0) at /home/peerplays/src/peerplays/libraries/fc/src/thread/task.cpp:32
#11 0x0000563479b8dc62 in fc::thread_d::run_next_task (this=0x7fe908000b60) at /home/peerplays/src/peerplays/libraries/fc/src/thread/thread_d.hpp:520
#12 0x0000563479b8e011 in fc::thread_d::process_tasks (this=0x7fe908000b60) at /home/peerplays/src/peerplays/libraries/fc/src/thread/thread_d.hpp:568
#13 0x0000563479b8d5e1 in fc::thread_d::start_process_tasks (my=...) at /home/peerplays/src/peerplays/libraries/fc/src/thread/thread_d.hpp:499
#14 0x0000563479e0296f in make_fcontext ()
#15 0x0000000000000000 in ?? ()
(gdb)

Logs at: https://gitlab.com/PBSA/documentation/working-documents/-/blob/develop/501/crash.tar.gz

Steps: 1. After stopping witness node peerplays02 other witness nodes crashed. ``` kiran@warpspeed:~/PBSA/06012023/peerplays-utils/peerplays-qa-environment$ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES f7ba51b8a3fd peerplays05 "./witness_node -d .…" 45 minutes ago Up 45 minutes 22/tcp, 0.0.0.0:8094->8090/tcp, :::8094->8090/tcp, 0.0.0.0:9781->9777/tcp, :::9781->9777/tcp peerplays-qa-environment_peerplays05_1 a413e77f303b peerplays10 "./witness_node -d .…" 45 minutes ago Up 8 seconds 22/tcp, 0.0.0.0:8099->8090/tcp, :::8099->8090/tcp, 0.0.0.0:9786->9777/tcp, :::9786->9777/tcp peerplays-qa-environment_peerplays10_1 16b87407ac40 peerplays14 "./witness_node -d .…" 45 minutes ago Up 7 seconds 22/tcp, 0.0.0.0:8103->8090/tcp, :::8103->8090/tcp, 0.0.0.0:9790->9777/tcp, :::9790->9777/tcp peerplays-qa-environment_peerplays14_1 fc00fe7ec784 peerplays06 "./witness_node -d .…" 45 minutes ago Up 8 seconds 22/tcp, 0.0.0.0:8095->8090/tcp, :::8095->8090/tcp, 0.0.0.0:9782->9777/tcp, :::9782->9777/tcp peerplays-qa-environment_peerplays06_1 f366d30cf234 peerplays12 "./witness_node -d .…" 45 minutes ago Up 19 seconds 22/tcp, 0.0.0.0:8101->8090/tcp, :::8101->8090/tcp, 0.0.0.0:9788->9777/tcp, :::9788->9777/tcp peerplays-qa-environment_peerplays12_1 5eb751b3e493 peerplays07 "./witness_node -d .…" 45 minutes ago Up 45 minutes 22/tcp, 0.0.0.0:8096->8090/tcp, :::8096->8090/tcp, 0.0.0.0:9783->9777/tcp, :::9783->9777/tcp peerplays-qa-environment_peerplays07_1 c736743f669b peerplays04 "./witness_node -d .…" 45 minutes ago Up 45 minutes 22/tcp, 0.0.0.0:8093->8090/tcp, :::8093->8090/tcp, 0.0.0.0:9780->9777/tcp, :::9780->9777/tcp peerplays-qa-environment_peerplays04_1 a781598646cd peerplays03 "./witness_node -d .…" 45 minutes ago Up 45 minutes 22/tcp, 0.0.0.0:8092->8090/tcp, :::8092->8090/tcp, 0.0.0.0:9779->9777/tcp, :::9779->9777/tcp peerplays-qa-environment_peerplays03_1 26424f96836e peerplays08 "./witness_node -d .…" 45 minutes ago Restarting (139) Less than a second ago peerplays-qa-environment_peerplays08_1 45b8579384e4 peerplays01 "./witness_node -d .…" 45 minutes ago Up 45 minutes 0.0.0.0:8090->8090/tcp, :::8090->8090/tcp, 22/tcp, 0.0.0.0:9777->9777/tcp, :::9777->9777/tcp peerplays-qa-environment_peerplays01_1 6c11ee69e16b peerplays11 "./witness_node -d .…" 45 minutes ago Up 20 seconds 22/tcp, 0.0.0.0:8100->8090/tcp, :::8100->8090/tcp, 0.0.0.0:9787->9777/tcp, :::9787->9777/tcp peerplays-qa-environment_peerplays11_1 70d9996e2e0d peerplays15 "./witness_node -d .…" 45 minutes ago Up 45 minutes 22/tcp, 0.0.0.0:8104->8090/tcp, :::8104->8090/tcp, 0.0.0.0:9791->9777/tcp, :::9791->9777/tcp peerplays-qa-environment_peerplays15_1 2c084dd3c8cf peerplays09 "./witness_node -d .…" 45 minutes ago Up 45 minutes 22/tcp, 0.0.0.0:8098->8090/tcp, :::8098->8090/tcp, 0.0.0.0:9785->9777/tcp, :::9785->9777/tcp peerplays-qa-environment_peerplays09_1 0d3b2dbcdb29 peerplays16 "./witness_node -d .…" 45 minutes ago Up 45 minutes 22/tcp, 0.0.0.0:8105->8090/tcp, :::8105->8090/tcp, 0.0.0.0:9792->9777/tcp, :::9792->9777/tcp peerplays-qa-environment_peerplays16_1 18d4edab2023 peerplays02 "./witness_node -d .…" 45 minutes ago Exited (137) 32 minutes ago peerplays-qa-environment_peerplays02_1 e2c6788d5c73 peerplays13 "./witness_node -d .…" 45 minutes ago Up 45 minutes 22/tcp, 0.0.0.0:8102->8090/tcp, :::8102->8090/tcp, 0.0.0.0:9789->9777/tcp, :::9789->9777/tcp peerplays-qa-environment_peerplays13_1 e694f0b5083c peerplays-base "bash" 45 minutes ago Exited (0) 45 minutes ago peerplays-qa-environment_peerplays-base_1 40f563d58db9 hive-for-peerplays "./hived -d ./hived_…" 59 minutes ago Up 59 minutes 22/tcp, 0.0.0.0:28090-28091->28090-28091/tcp, :::28090-28091->28090-28091/tcp peerplays-qa-environment_hive-for-peerplays_1 7eeb00febe10 ethereum-for-peerplays "./geth --config ./n…" About an hour ago Up About an hour 0.0.0.0:6060->6060/tcp, :::6060->6060/tcp, 22/tcp, 0.0.0.0:8545->8545/tcp, :::8545->8545/tcp, 8546/tcp, 0.0.0.0:30303->30303/tcp, :::30303->30303/tcp peerplays-qa-environment_ethereum-for-peerplays_1 467391e4c22f libbitcoinserver-for-peerplays "/bin/sh runserver.sh" About an hour ago Up About an hour 0.0.0.0:9091->9091/tcp, :::9091->9091/tcp, 22/tcp, 0.0.0.0:9093->9093/tcp, :::9093->9093/tcp peerplays-qa-environment_libbitcoinserver-for-peerplays_1 604f2e13e799 bitcoin-for-peerplays "./bitcoind -datadir…" About an hour ago Up About an hour 0.0.0.0:8332->8332/tcp, :::8332->8332/tcp, 0.0.0.0:11111->11111/tcp, :::11111->11111/tcp, 22/tcp, 0.0.0.0:18444->18444/tcp, :::18444->18444/tcp peerplays-qa-environment_bitcoin-for-peerplays_1 0a99747528d5 ubuntu-for-peerplays "bash" About an hour ago Exited (0) 45 minutes ago peerplays-qa-environment_ubuntu-for-peerplays_1 ad0ee593076b e28a50f651f9 "bash" 4 days ago Exited (0) 4 days ago clever_hodgkin 2b1ca83f2f7c ubuntu:18.04 "bash" 4 days ago Exited (0) 4 days ago inspiring_tharp kiran@warpspeed:~/PBSA/06012023/peerplays-utils/peerplays-qa-environment$ ``` ``` kiran@warpspeed:/home$ sudo coredumpctl debug 4999 PID: 4999 (witness_node) UID: 0 (root) GID: 0 (root) Signal: 11 (SEGV) Timestamp: Fri 2023-01-13 00:17:36 +0530 (9min ago) Command Line: ./witness_node -d ./witness_node_data_dir --replay-blockchain Executable: /home/peerplays/src/peerplays/build/programs/witness_node/witness_node Control Group: /docker/16b87407ac4019e43c2c16987ae405a3af7d52c9256235aded38ac02c6acf5ff Slice: -.slice Boot ID: 4ced3cb6f68d4f38b5ad5574dd873a6e Machine ID: a388625228614d18b5373ad1be39a93f Hostname: warpspeed Storage: /var/lib/systemd/coredump/core.witness_node.0.4ced3cb6f68d4f38b5ad5574dd873a6e.4999.1673549256000000000000.lz4 Message: Process 4999 (witness_node) of user 0 dumped core. Stack trace of thread 15: #0 0x00007f4f8c6ba941 __GI_abort (libc.so.6 + 0x22941) #1 0x00007f4f8c6ba729 __assert_fail_base (libc.so.6 + 0x22729) #2 0x00007f4f8c6cbfd6 __GI___assert_fail (libc.so.6 + 0x33fd6) #3 0x000055cea30ef588 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x57cb588) #4 0x000055cea310cb13 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x57e8b13) #5 0x000055cea3259c26 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x5935c26) #6 0x000055cea326071b n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x593c71b) #7 0x000055cea3260031 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x593c031) #8 0x000055cea3265a0e n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x5941a0e) #9 0x000055cea45b7138 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6c93138) #10 0x000055cea45b70b4 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6c930b4) #11 0x000055cea45a8c62 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6c84c62) #12 0x000055cea45a9011 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6c85011) #13 0x000055cea45a85e1 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6c845e1) #14 0x000055cea481d96f n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6ef996f) GNU gdb (Ubuntu 9.2-0ubuntu1~20.04.1) 9.2 Copyright (C) 2020 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /home/peerplays/src/peerplays/build/programs/witness_node/witness_node... [New LWP 15] [New LWP 9] [New LWP 12] [New LWP 13] [New LWP 7] [New LWP 19] [New LWP 16] [New LWP 2442] [New LWP 18] [New LWP 10] [New LWP 17] [New LWP 2441] [New LWP 2443] [New LWP 8] [New LWP 1] [New LWP 11] [New LWP 14] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `./witness_node -d ./witness_node_data_dir --replay-blockchain'. Program terminated with signal SIGSEGV, Segmentation fault. #0 __GI_abort () at abort.c:107 107 abort.c: No such file or directory. [Current thread is 1 (Thread 0x7f4f727fc700 (LWP 15))] (gdb) bt #0 __GI_abort () at abort.c:107 #1 0x00007f4f8c6ba729 in __assert_fail_base (fmt=0x7f4f8c850588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x55cea4cb8458 "_closing_connections.find(peer_to_delete) == _closing_connections.end()", file=0x55cea4cb69e8 "/home/peerplays/src/peerplays/libraries/net/node.cpp", line=1654, function=<optimized out>) at assert.c:92 #2 0x00007f4f8c6cbfd6 in __GI___assert_fail (assertion=0x55cea4cb8458 "_closing_connections.find(peer_to_delete) == _closing_connections.end()", file=0x55cea4cb69e8 "/home/peerplays/src/peerplays/libraries/net/node.cpp", line=1654, function=0x55cea4cb8360 "void graphene::net::detail::node_impl::schedule_peer_for_deletion(const peer_connection_ptr&)") at assert.c:101 #3 0x000055cea30ef588 in graphene::net::detail::node_impl::schedule_peer_for_deletion (this=0x55cea6cae5a0, peer_to_delete=std::shared_ptr<class graphene::net::peer_connection> (use count 4, weak count 2) = {...}) at /home/peerplays/src/peerplays/libraries/net/node.cpp:1654 #4 0x000055cea310cb13 in graphene::net::detail::node_impl::on_connection_closed (this=0x55cea6cae5a0, originating_peer=0x7f4f5c07ded0) at /home/peerplays/src/peerplays/libraries/net/node.cpp:3106 #5 0x000055cea3259c26 in graphene::net::peer_connection::on_connection_closed (this=0x7f4f5c07ded0, originating_connection=0x7f4f5c07df00) at /home/peerplays/src/peerplays/libraries/net/peer_connection.cpp:283 #6 0x000055cea326071b in graphene::net::detail::message_oriented_connection_impl::read_loop (this=0x7f4f5c07e2c0) at /home/peerplays/src/peerplays/libraries/net/message_oriented_connection.cpp:241 #7 0x000055cea3260031 in graphene::net::detail::message_oriented_connection_impl::<lambda()>::operator()(void) const (__closure=0x7f4f5c159c08) at /home/peerplays/src/peerplays/libraries/net/message_oriented_connection.cpp:134 #8 0x000055cea3265a0e in fc::detail::void_functor_run<graphene::net::detail::message_oriented_connection_impl::connect_to(const fc::ip::endpoint&)::<lambda()> >::run(void *, void *) (functor=0x7f4f5c159c08, prom=0x7f4f5c159c00) at /home/peerplays/src/peerplays/libraries/fc/include/fc/thread/task.hpp:83 #9 0x000055cea45b7138 in fc::task_base::run_impl (this=0x7f4f5c159c10) at /home/peerplays/src/peerplays/libraries/fc/src/thread/task.cpp:43 #10 0x000055cea45b70b4 in fc::task_base::run (this=0x7f4f5c159c10) at /home/peerplays/src/peerplays/libraries/fc/src/thread/task.cpp:32 #11 0x000055cea45a8c62 in fc::thread_d::run_next_task (this=0x7f4f5c000b60) at /home/peerplays/src/peerplays/libraries/fc/src/thread/thread_d.hpp:520 #12 0x000055cea45a9011 in fc::thread_d::process_tasks (this=0x7f4f5c000b60) at /home/peerplays/src/peerplays/libraries/fc/src/thread/thread_d.hpp:568 #13 0x000055cea45a85e1 in fc::thread_d::start_process_tasks (my=...) at /home/peerplays/src/peerplays/libraries/fc/src/thread/thread_d.hpp:499 #14 0x000055cea481d96f in make_fcontext () #15 0x0000000000000000 in ?? () (gdb) ``` ``` kiran@warpspeed:/home$ sudo coredumpctl debug 1935140 PID: 1935140 (witness_node) UID: 0 (root) GID: 0 (root) Signal: 11 (SEGV) Timestamp: Fri 2023-01-13 20:10:39 +0530 (26min ago) Command Line: ./witness_node -d ./witness_node_data_dir --replay-blockchain Executable: /home/peerplays/src/peerplays/build/programs/witness_node/witness_node Control Group: /docker/2c084dd3c8cfb2d4b01898de6e02665b9594bccd75aabd1d080ff4db2ee81847 Slice: -.slice Boot ID: 4ced3cb6f68d4f38b5ad5574dd873a6e Machine ID: a388625228614d18b5373ad1be39a93f Hostname: warpspeed Storage: /var/lib/systemd/coredump/core.witness_node.0.4ced3cb6f68d4f38b5ad5574dd873a6e.1935140.1673620839000000000000.lz4 Message: Process 1935140 (witness_node) of user 0 dumped core. Stack trace of thread 15: #0 0x00007fe937fb3941 __GI_abort (libc.so.6 + 0x22941) #1 0x00007fe937fb3729 __assert_fail_base (libc.so.6 + 0x22729) #2 0x00007fe937fc4fd6 __GI___assert_fail (libc.so.6 + 0x33fd6) #3 0x00005634786d4588 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x57cb588) #4 0x00005634786f1b13 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x57e8b13) #5 0x000056347883ec26 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x5935c26) #6 0x000056347884571b n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x593c71b) #7 0x0000563478845031 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x593c031) #8 0x000056347884aa0e n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x5941a0e) #9 0x0000563479b9c138 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6c93138) #10 0x0000563479b9c0b4 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6c930b4) #11 0x0000563479b8dc62 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6c84c62) #12 0x0000563479b8e011 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6c85011) #13 0x0000563479b8d5e1 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6c845e1) #14 0x0000563479e0296f n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6ef996f) GNU gdb (Ubuntu 9.2-0ubuntu1~20.04.1) 9.2 Copyright (C) 2020 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /home/peerplays/src/peerplays/build/programs/witness_node/witness_node... [New LWP 15] [New LWP 10] [New LWP 8] [New LWP 9] [New LWP 11] [New LWP 14] [New LWP 16] [New LWP 1] [New LWP 7] [New LWP 13] [New LWP 19] [New LWP 12] [New LWP 17] [New LWP 18] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `./witness_node -d ./witness_node_data_dir --replay-blockchain'. Program terminated with signal SIGSEGV, Segmentation fault. #0 __GI_abort () at abort.c:107 107 abort.c: No such file or directory. [Current thread is 1 (Thread 0x7fe91dffb700 (LWP 15))] (gdb) bt #0 __GI_abort () at abort.c:107 #1 0x00007fe937fb3729 in __assert_fail_base (fmt=0x7fe938149588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x56347a29d458 "_closing_connections.find(peer_to_delete) == _closing_connections.end()", file=0x56347a29b9e8 "/home/peerplays/src/peerplays/libraries/net/node.cpp", line=1654, function=<optimized out>) at assert.c:92 #2 0x00007fe937fc4fd6 in __GI___assert_fail (assertion=0x56347a29d458 "_closing_connections.find(peer_to_delete) == _closing_connections.end()", file=0x56347a29b9e8 "/home/peerplays/src/peerplays/libraries/net/node.cpp", line=1654, function=0x56347a29d360 "void graphene::net::detail::node_impl::schedule_peer_for_deletion(const peer_connection_ptr&)") at assert.c:101 #3 0x00005634786d4588 in graphene::net::detail::node_impl::schedule_peer_for_deletion (this=0x56347ed06780, peer_to_delete=std::shared_ptr<class graphene::net::peer_connection> (use count 4, weak count 2) = {...}) at /home/peerplays/src/peerplays/libraries/net/node.cpp:1654 #4 0x00005634786f1b13 in graphene::net::detail::node_impl::on_connection_closed (this=0x56347ed06780, originating_peer=0x7fe908133700) at /home/peerplays/src/peerplays/libraries/net/node.cpp:3106 #5 0x000056347883ec26 in graphene::net::peer_connection::on_connection_closed (this=0x7fe908133700, originating_connection=0x7fe908133730) at /home/peerplays/src/peerplays/libraries/net/peer_connection.cpp:283 #6 0x000056347884571b in graphene::net::detail::message_oriented_connection_impl::read_loop (this=0x7fe908133af0) at /home/peerplays/src/peerplays/libraries/net/message_oriented_connection.cpp:241 #7 0x0000563478845031 in graphene::net::detail::message_oriented_connection_impl::<lambda()>::operator()(void) const (__closure=0x7fe90814cad8) at /home/peerplays/src/peerplays/libraries/net/message_oriented_connection.cpp:134 #8 0x000056347884aa0e in fc::detail::void_functor_run<graphene::net::detail::message_oriented_connection_impl::connect_to(const fc::ip::endpoint&)::<lambda()> >::run(void *, void *) (functor=0x7fe90814cad8, prom=0x7fe90814cad0) at /home/peerplays/src/peerplays/libraries/fc/include/fc/thread/task.hpp:83 #9 0x0000563479b9c138 in fc::task_base::run_impl (this=0x7fe90814cae0) at /home/peerplays/src/peerplays/libraries/fc/src/thread/task.cpp:43 #10 0x0000563479b9c0b4 in fc::task_base::run (this=0x7fe90814cae0) at /home/peerplays/src/peerplays/libraries/fc/src/thread/task.cpp:32 #11 0x0000563479b8dc62 in fc::thread_d::run_next_task (this=0x7fe908000b60) at /home/peerplays/src/peerplays/libraries/fc/src/thread/thread_d.hpp:520 #12 0x0000563479b8e011 in fc::thread_d::process_tasks (this=0x7fe908000b60) at /home/peerplays/src/peerplays/libraries/fc/src/thread/thread_d.hpp:568 #13 0x0000563479b8d5e1 in fc::thread_d::start_process_tasks (my=...) at /home/peerplays/src/peerplays/libraries/fc/src/thread/thread_d.hpp:499 #14 0x0000563479e0296f in make_fcontext () #15 0x0000000000000000 in ?? () (gdb) ``` Logs at: https://gitlab.com/PBSA/documentation/working-documents/-/blob/develop/501/crash.tar.gz

prandnum commented

2023-01-13 15:20:50 +00:00

(Migrated from gitlab.com)

changed the description

vampik commented

2023-02-02 16:27:55 +00:00

(Migrated from gitlab.com)

assigned to @vampik

vampik commented

2023-02-13 08:29:43 +00:00

(Migrated from gitlab.com)

mentioned in issue #512

vampik commented

2023-02-23 15:56:00 +00:00

(Migrated from gitlab.com)

mentioned in commit 2788281062

mentioned in commit 2788281062205a0bc9904b31aeae4ef30f7e7643

vampik commented

2023-02-27 07:23:45 +00:00

(Migrated from gitlab.com)

mentioned in merge request !212

vampik commented

2023-02-27 07:24:11 +00:00

(Migrated from gitlab.com)

assigned to @serkixenos

vampik commented

2023-02-27 13:34:42 +00:00

(Migrated from gitlab.com)

mentioned in commit 5867a8ae27

mentioned in commit 5867a8ae27710192aebb4a1ebe79b3746242ba5b

vampik commented

2023-02-27 13:36:21 +00:00

(Migrated from gitlab.com)

assigned to @prandnum

vampik commented

2023-02-27 13:37:29 +00:00

(Migrated from gitlab.com)

@prandnum We implement thread-safe pool of connections.
It should fix appearing of this assert.

@prandnum We implement thread-safe pool of connections. It should fix appearing of this assert.

prandnum commented

2023-02-28 13:35:56 +00:00

(Migrated from gitlab.com)

@vampik As you have put the state in testing, is this feature already implemented? If it is not implemented and there is a feature ticket, then better to close giving a link to that feature ticket.

CC: @serkixenos @bobinson

@vampik As you have put the state in testing, is this feature already implemented? If it is not implemented and there is a feature ticket, then better to close giving a link to that feature ticket. CC: @serkixenos @bobinson

vampik commented

2023-03-01 06:04:39 +00:00

(Migrated from gitlab.com)

There is no separate ticket for this feature.

prandnum commented

2023-03-06 15:09:42 +00:00

(Migrated from gitlab.com)

Tried the following

stopped one witness at a time(3 in total)
There was no crash
Brought up witness one at a time and when the third witness was brought up, after few mins(5mins) witness node crashed(peerplays05)

coredump details given below

kiran@warpspeed:~/PBSA/28022023/peerplays-utils/peerplays-qa-environment/e2e-tests$ !1912
docker exec -it peerplays-qa-environment_peerplays01_1 /bin/bash
root@e709bfa5ec77:~/peerplays-network# 
root@e709bfa5ec77:~/peerplays-network# 
root@e709bfa5ec77:~/peerplays-network# cd ../src/peerplays/
root@e709bfa5ec77:~/src/peerplays# 
root@e709bfa5ec77:~/src/peerplays# 
root@e709bfa5ec77:~/src/peerplays# git log -3
commit 16ba10ffab86c8992d24ece8d23b4b0e26cb5696 (HEAD -> develop, origin/develop)
Merge: 5867a8ae 79974280
Author: Vlad Dobromyslov <vdobrik@peerplays.com>
Date:   Wed Mar 1 06:01:52 2023 +0000

    Merge branch 'feature/SON-connection-pool' into 'develop'
    
    SON connection pool
    
    See merge request PBSA/peerplays!181

commit 79974280c0a627e82fd6d07b55ed4b01ac533f1f
Author: timur <12267899-timur.5@users.noreply.gitlab.com>
Date:   Wed Mar 1 06:01:52 2023 +0000

    SON connection pool

commit 5867a8ae27710192aebb4a1ebe79b3746242ba5b
Merge: 741534c4 27882810
Author: Vlad Dobromyslov <vdobrik@peerplays.com>
Date:   Mon Feb 27 13:34:41 2023 +0000

    Merge branch 'bug/501-connection-pool' into 'develop'
    
    #501 - concurrent_unordered_set for connection
    
    See merge request PBSA/peerplays!212
root@e709bfa5ec77:~/src/peerplays#

kiran@warpspeed:/home$ sudo coredumpctl  debug 1118416
           PID: 1118416 (witness_node)
           UID: 0 (root)
           GID: 0 (root)
        Signal: 11 (SEGV)
     Timestamp: Sat 2023-03-04 01:11:12 +0530 (7min ago)
  Command Line: ./witness_node -d ./witness_node_data_dir --replay-blockchain
    Executable: /home/peerplays/src/peerplays/build/programs/witness_node/witness_node
 Control Group: /docker/9e02a9dc16dcd530d8389449ed181b093cde95f9249c58e10abfd7e37ec351b3
         Slice: -.slice
       Boot ID: f9ea01af03534b149efc0d39fb9d10e3
    Machine ID: a388625228614d18b5373ad1be39a93f
      Hostname: warpspeed
       Storage: /var/lib/systemd/coredump/core.witness_node.0.f9ea01af03534b149efc0d39fb9d10e3.1118416.1677872472000000000000.lz4
       Message: Process 1118416 (witness_node) of user 0 dumped core.
                
                Stack trace of thread 15:
                #0  0x00007f23c1d05941 __GI_abort (libc.so.6 + 0x22941)
                #1  0x00007f23c1d05729 __assert_fail_base (libc.so.6 + 0x22729)
                #2  0x00007f23c1d16fd6 __GI___assert_fail (libc.so.6 + 0x33fd6)
                #3  0x0000558cd4bd3275 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x58d4275)
                #4  0x0000558cd4bf0c07 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x58f1c07)
                #5  0x0000558cd4d3e56a n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x5a3f56a)
                #6  0x0000558cd4d4505f n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x5a4605f)
                #7  0x0000558cd4d44975 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x5a45975)
                #8  0x0000558cd4d4a352 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x5a4b352)
                #9  0x0000558cd60a3a72 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6da4a72)
                #10 0x0000558cd60a39ee n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6da49ee)
                #11 0x0000558cd609559c n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6d9659c)
                #12 0x0000558cd609594b n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6d9694b)
                #13 0x0000558cd6094f1b n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6d95f1b)
                #14 0x0000558cd630a4bf n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x700b4bf)

GNU gdb (Ubuntu 9.2-0ubuntu1~20.04.1) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/peerplays/src/peerplays/build/programs/witness_node/witness_node...
[New LWP 15]
[New LWP 7]
[New LWP 12]
[New LWP 8]
[New LWP 9]
[New LWP 11]
[New LWP 10]
[New LWP 13]
[New LWP 14]
[New LWP 16]
[New LWP 20]
[New LWP 19]
[New LWP 24]
[New LWP 17]
[New LWP 23]
[New LWP 21]
[New LWP 18]
[New LWP 22]
[New LWP 1]

warning: .dynamic section for "/usr/local/lib/libbitcoin-protocol.so.0" is not at the expected address (wrong library or version mismatch?)

warning: .dynamic section for "/usr/local/lib/libbitcoin-client.so.0" is not at the expected address (wrong library or version mismatch?)

warning: .dynamic section for "/usr/local/lib/libbitcoin-explorer.so.0" is not at the expected address (wrong library or version mismatch?)

warning: .dynamic section for "/usr/local//lib/libsecp256k1.so.0" is not at the expected address (wrong library or version mismatch?)

warning: .dynamic section for "/usr/local//lib/libbitcoin-network.so.0" is not at the expected address (wrong library or version mismatch?)
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `./witness_node -d ./witness_node_data_dir --replay-blockchain'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  __GI_abort () at abort.c:107
107	abort.c: No such file or directory.
[Current thread is 1 (Thread 0x7f23ad7fa700 (LWP 15))]
(gdb) bt
#0  __GI_abort () at abort.c:107
#1  0x00007f23c1d05729 in __assert_fail_base (fmt=0x7f23c1e9b588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x558cd67d3d58 "_closing_connections.find(peer_to_delete) == _closing_connections.end()", 
    file=0x558cd67d22e8 "/home/peerplays/src/peerplays/libraries/net/node.cpp", line=1803, function=<optimized out>) at assert.c:92
#2  0x00007f23c1d16fd6 in __GI___assert_fail (assertion=0x558cd67d3d58 "_closing_connections.find(peer_to_delete) == _closing_connections.end()", file=0x558cd67d22e8 "/home/peerplays/src/peerplays/libraries/net/node.cpp", 
    line=1803, function=0x558cd67d3c60 "void graphene::net::detail::node_impl::schedule_peer_for_deletion(const peer_connection_ptr&)") at assert.c:101
#3  0x0000558cd4bd3275 in graphene::net::detail::node_impl::schedule_peer_for_deletion (this=0x558cd9ae5530, peer_to_delete=std::shared_ptr<class graphene::net::peer_connection> (use count 3, weak count 2) = {...})
    at /home/peerplays/src/peerplays/libraries/net/node.cpp:1803
#4  0x0000558cd4bf0c07 in graphene::net::detail::node_impl::on_connection_closed (this=0x558cd9ae5530, originating_peer=0x7f23901309e0) at /home/peerplays/src/peerplays/libraries/net/node.cpp:3284
#5  0x0000558cd4d3e56a in graphene::net::peer_connection::on_connection_closed (this=0x7f23901309e0, originating_connection=0x7f2390130a10) at /home/peerplays/src/peerplays/libraries/net/peer_connection.cpp:283
#6  0x0000558cd4d4505f in graphene::net::detail::message_oriented_connection_impl::read_loop (this=0x7f23900a6c40) at /home/peerplays/src/peerplays/libraries/net/message_oriented_connection.cpp:241
#7  0x0000558cd4d44975 in graphene::net::detail::message_oriented_connection_impl::<lambda()>::operator()(void) const (__closure=0x7f2390079058) at /home/peerplays/src/peerplays/libraries/net/message_oriented_connection.cpp:134
#8  0x0000558cd4d4a352 in fc::detail::void_functor_run<graphene::net::detail::message_oriented_connection_impl::connect_to(const fc::ip::endpoint&)::<lambda()> >::run(void *, void *) (functor=0x7f2390079058, prom=0x7f2390079050)
    at /home/peerplays/src/peerplays/libraries/fc/include/fc/thread/task.hpp:83
#9  0x0000558cd60a3a72 in fc::task_base::run_impl (this=0x7f2390079060) at /home/peerplays/src/peerplays/libraries/fc/src/thread/task.cpp:43
#10 0x0000558cd60a39ee in fc::task_base::run (this=0x7f2390079060) at /home/peerplays/src/peerplays/libraries/fc/src/thread/task.cpp:32
#11 0x0000558cd609559c in fc::thread_d::run_next_task (this=0x7f2390000b60) at /home/peerplays/src/peerplays/libraries/fc/src/thread/thread_d.hpp:520
#12 0x0000558cd609594b in fc::thread_d::process_tasks (this=0x7f2390000b60) at /home/peerplays/src/peerplays/libraries/fc/src/thread/thread_d.hpp:568
#13 0x0000558cd6094f1b in fc::thread_d::start_process_tasks (my=...) at /home/peerplays/src/peerplays/libraries/fc/src/thread/thread_d.hpp:499
#14 0x0000558cd630a4bf in make_fcontext ()
#15 0x0000000000000000 in ?? ()
(gdb) quit
kiran@warpspeed:/home$

Tried the following 1) stopped one witness at a time(3 in total) 2) There was no crash 3) Brought up witness one at a time and when the third witness was brought up, after few mins(5mins) witness node crashed(peerplays05) coredump details given below ``` kiran@warpspeed:~/PBSA/28022023/peerplays-utils/peerplays-qa-environment/e2e-tests$ !1912 docker exec -it peerplays-qa-environment_peerplays01_1 /bin/bash root@e709bfa5ec77:~/peerplays-network# root@e709bfa5ec77:~/peerplays-network# root@e709bfa5ec77:~/peerplays-network# cd ../src/peerplays/ root@e709bfa5ec77:~/src/peerplays# root@e709bfa5ec77:~/src/peerplays# root@e709bfa5ec77:~/src/peerplays# git log -3 commit 16ba10ffab86c8992d24ece8d23b4b0e26cb5696 (HEAD -> develop, origin/develop) Merge: 5867a8ae 79974280 Author: Vlad Dobromyslov <vdobrik@peerplays.com> Date: Wed Mar 1 06:01:52 2023 +0000 Merge branch 'feature/SON-connection-pool' into 'develop' SON connection pool See merge request PBSA/peerplays!181 commit 79974280c0a627e82fd6d07b55ed4b01ac533f1f Author: timur <12267899-timur.5@users.noreply.gitlab.com> Date: Wed Mar 1 06:01:52 2023 +0000 SON connection pool commit 5867a8ae27710192aebb4a1ebe79b3746242ba5b Merge: 741534c4 27882810 Author: Vlad Dobromyslov <vdobrik@peerplays.com> Date: Mon Feb 27 13:34:41 2023 +0000 Merge branch 'bug/501-connection-pool' into 'develop' #501 - concurrent_unordered_set for connection See merge request PBSA/peerplays!212 root@e709bfa5ec77:~/src/peerplays# ``` ``` kiran@warpspeed:/home$ sudo coredumpctl debug 1118416 PID: 1118416 (witness_node) UID: 0 (root) GID: 0 (root) Signal: 11 (SEGV) Timestamp: Sat 2023-03-04 01:11:12 +0530 (7min ago) Command Line: ./witness_node -d ./witness_node_data_dir --replay-blockchain Executable: /home/peerplays/src/peerplays/build/programs/witness_node/witness_node Control Group: /docker/9e02a9dc16dcd530d8389449ed181b093cde95f9249c58e10abfd7e37ec351b3 Slice: -.slice Boot ID: f9ea01af03534b149efc0d39fb9d10e3 Machine ID: a388625228614d18b5373ad1be39a93f Hostname: warpspeed Storage: /var/lib/systemd/coredump/core.witness_node.0.f9ea01af03534b149efc0d39fb9d10e3.1118416.1677872472000000000000.lz4 Message: Process 1118416 (witness_node) of user 0 dumped core. Stack trace of thread 15: #0 0x00007f23c1d05941 __GI_abort (libc.so.6 + 0x22941) #1 0x00007f23c1d05729 __assert_fail_base (libc.so.6 + 0x22729) #2 0x00007f23c1d16fd6 __GI___assert_fail (libc.so.6 + 0x33fd6) #3 0x0000558cd4bd3275 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x58d4275) #4 0x0000558cd4bf0c07 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x58f1c07) #5 0x0000558cd4d3e56a n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x5a3f56a) #6 0x0000558cd4d4505f n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x5a4605f) #7 0x0000558cd4d44975 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x5a45975) #8 0x0000558cd4d4a352 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x5a4b352) #9 0x0000558cd60a3a72 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6da4a72) #10 0x0000558cd60a39ee n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6da49ee) #11 0x0000558cd609559c n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6d9659c) #12 0x0000558cd609594b n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6d9694b) #13 0x0000558cd6094f1b n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6d95f1b) #14 0x0000558cd630a4bf n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x700b4bf) GNU gdb (Ubuntu 9.2-0ubuntu1~20.04.1) 9.2 Copyright (C) 2020 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /home/peerplays/src/peerplays/build/programs/witness_node/witness_node... [New LWP 15] [New LWP 7] [New LWP 12] [New LWP 8] [New LWP 9] [New LWP 11] [New LWP 10] [New LWP 13] [New LWP 14] [New LWP 16] [New LWP 20] [New LWP 19] [New LWP 24] [New LWP 17] [New LWP 23] [New LWP 21] [New LWP 18] [New LWP 22] [New LWP 1] warning: .dynamic section for "/usr/local/lib/libbitcoin-protocol.so.0" is not at the expected address (wrong library or version mismatch?) warning: .dynamic section for "/usr/local/lib/libbitcoin-client.so.0" is not at the expected address (wrong library or version mismatch?) warning: .dynamic section for "/usr/local/lib/libbitcoin-explorer.so.0" is not at the expected address (wrong library or version mismatch?) warning: .dynamic section for "/usr/local//lib/libsecp256k1.so.0" is not at the expected address (wrong library or version mismatch?) warning: .dynamic section for "/usr/local//lib/libbitcoin-network.so.0" is not at the expected address (wrong library or version mismatch?) [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `./witness_node -d ./witness_node_data_dir --replay-blockchain'. Program terminated with signal SIGSEGV, Segmentation fault. #0 __GI_abort () at abort.c:107 107 abort.c: No such file or directory. [Current thread is 1 (Thread 0x7f23ad7fa700 (LWP 15))] (gdb) bt #0 __GI_abort () at abort.c:107 #1 0x00007f23c1d05729 in __assert_fail_base (fmt=0x7f23c1e9b588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x558cd67d3d58 "_closing_connections.find(peer_to_delete) == _closing_connections.end()", file=0x558cd67d22e8 "/home/peerplays/src/peerplays/libraries/net/node.cpp", line=1803, function=<optimized out>) at assert.c:92 #2 0x00007f23c1d16fd6 in __GI___assert_fail (assertion=0x558cd67d3d58 "_closing_connections.find(peer_to_delete) == _closing_connections.end()", file=0x558cd67d22e8 "/home/peerplays/src/peerplays/libraries/net/node.cpp", line=1803, function=0x558cd67d3c60 "void graphene::net::detail::node_impl::schedule_peer_for_deletion(const peer_connection_ptr&)") at assert.c:101 #3 0x0000558cd4bd3275 in graphene::net::detail::node_impl::schedule_peer_for_deletion (this=0x558cd9ae5530, peer_to_delete=std::shared_ptr<class graphene::net::peer_connection> (use count 3, weak count 2) = {...}) at /home/peerplays/src/peerplays/libraries/net/node.cpp:1803 #4 0x0000558cd4bf0c07 in graphene::net::detail::node_impl::on_connection_closed (this=0x558cd9ae5530, originating_peer=0x7f23901309e0) at /home/peerplays/src/peerplays/libraries/net/node.cpp:3284 #5 0x0000558cd4d3e56a in graphene::net::peer_connection::on_connection_closed (this=0x7f23901309e0, originating_connection=0x7f2390130a10) at /home/peerplays/src/peerplays/libraries/net/peer_connection.cpp:283 #6 0x0000558cd4d4505f in graphene::net::detail::message_oriented_connection_impl::read_loop (this=0x7f23900a6c40) at /home/peerplays/src/peerplays/libraries/net/message_oriented_connection.cpp:241 #7 0x0000558cd4d44975 in graphene::net::detail::message_oriented_connection_impl::<lambda()>::operator()(void) const (__closure=0x7f2390079058) at /home/peerplays/src/peerplays/libraries/net/message_oriented_connection.cpp:134 #8 0x0000558cd4d4a352 in fc::detail::void_functor_run<graphene::net::detail::message_oriented_connection_impl::connect_to(const fc::ip::endpoint&)::<lambda()> >::run(void *, void *) (functor=0x7f2390079058, prom=0x7f2390079050) at /home/peerplays/src/peerplays/libraries/fc/include/fc/thread/task.hpp:83 #9 0x0000558cd60a3a72 in fc::task_base::run_impl (this=0x7f2390079060) at /home/peerplays/src/peerplays/libraries/fc/src/thread/task.cpp:43 #10 0x0000558cd60a39ee in fc::task_base::run (this=0x7f2390079060) at /home/peerplays/src/peerplays/libraries/fc/src/thread/task.cpp:32 #11 0x0000558cd609559c in fc::thread_d::run_next_task (this=0x7f2390000b60) at /home/peerplays/src/peerplays/libraries/fc/src/thread/thread_d.hpp:520 #12 0x0000558cd609594b in fc::thread_d::process_tasks (this=0x7f2390000b60) at /home/peerplays/src/peerplays/libraries/fc/src/thread/thread_d.hpp:568 #13 0x0000558cd6094f1b in fc::thread_d::start_process_tasks (my=...) at /home/peerplays/src/peerplays/libraries/fc/src/thread/thread_d.hpp:499 #14 0x0000558cd630a4bf in make_fcontext () #15 0x0000000000000000 in ?? () (gdb) quit kiran@warpspeed:/home$ ```

prandnum commented

2023-03-06 15:10:40 +00:00

(Migrated from gitlab.com)

@vampik looks similar, what do you think?

prandnum commented

2023-03-07 18:13:11 +00:00

(Migrated from gitlab.com)

@vampik did you get a chance to take a look?

vampik commented

2023-03-08 05:38:08 +00:00

(Migrated from gitlab.com)

Not yet. I will let you know, when I will have a chance.

vampik commented

2023-03-09 12:57:26 +00:00

(Migrated from gitlab.com)

Yes, looks similar

vampik commented

2023-03-15 05:45:35 +00:00

(Migrated from gitlab.com)

mentioned in commit 5203869a9d

mentioned in commit 5203869a9d876b8f0d9549fb19b361fc485dadb0

vampik commented

2023-03-15 05:46:26 +00:00

(Migrated from gitlab.com)

@prandnum
Can you try the build from this branch:
https://gitlab.com/PBSA/peerplays/-/commits/bug/501-on_connection_closed

@prandnum Can you try the build from this branch: https://gitlab.com/PBSA/peerplays/-/commits/bug/501-on_connection_closed

prandnum commented

2023-03-19 10:24:15 +00:00

(Migrated from gitlab.com)

@vampik same steps and one witness node crashed.

kiran@warpspeed:/home$ sudo coredumpctl debug 4182006
           PID: 4182006 (witness_node)
           UID: 0 (root)
           GID: 0 (root)
        Signal: 11 (SEGV)
     Timestamp: Sat 2023-03-18 11:49:38 +0530 (1 day 4h ago)
  Command Line: ./witness_node -d ./witness_node_data_dir --replay-blockchain
    Executable: /home/peerplays/src/peerplays/build/programs/witness_node/witness_node
 Control Group: /docker/305dcb1d73ba446f8b0db24edd3c92914ce517eef02bae8eb08fa23ac4ba178c
         Slice: -.slice
       Boot ID: f9ea01af03534b149efc0d39fb9d10e3
    Machine ID: a388625228614d18b5373ad1be39a93f
      Hostname: warpspeed
       Storage: /var/lib/systemd/coredump/core.witness_node.0.f9ea01af03534b149efc0d39fb9d10e3.4182006.1679120378000000000000.lz4
       Message: Process 4182006 (witness_node) of user 0 dumped core.
                
                Stack trace of thread 15:
                #0  0x00007f8e65512941 __GI_abort (libc.so.6 + 0x22941)
                #1  0x00007f8e65512729 __assert_fail_base (libc.so.6 + 0x22729)
                #2  0x00007f8e65523fd6 __GI___assert_fail (libc.so.6 + 0x33fd6)
                #3  0x000056178784e000 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6d98000)
                #4  0x0000561787846bb3 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6d90bb3)
                #5  0x000056178785f459 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6da9459)
                #6  0x000056178640597c n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x594f97c)
                #7  0x000056178637cf4f n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x58c6f4f)
                #8  0x00005617863cdded n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x5917ded)
                #9  0x00005617863fd08e n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x594708e)
                #10 0x000056178785d3e8 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6da73e8)
                #11 0x000056178785d364 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6da7364)
                #12 0x000056178784ef12 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6d98f12)
                #13 0x000056178784f2c1 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6d992c1)
                #14 0x000056178784e891 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6d98891)
                #15 0x0000561787ac3e2f n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x700de2f)

GNU gdb (Ubuntu 9.2-0ubuntu1~20.04.1) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/peerplays/src/peerplays/build/programs/witness_node/witness_node...
[New LWP 15]
[New LWP 10]
[New LWP 7]
[New LWP 1]
[New LWP 13]
[New LWP 8]
[New LWP 16]
[New LWP 19]
[New LWP 9]
[New LWP 22]
[New LWP 12]
[New LWP 17]
[New LWP 20]
[New LWP 18]
[New LWP 1343]
[New LWP 1342]
[New LWP 24]
[New LWP 21]
[New LWP 1344]
[New LWP 23]
[New LWP 14]
[New LWP 11]

warning: .dynamic section for "/usr/local/lib/libbitcoin-protocol.so.0" is not at the expected address (wrong library or version mismatch?)

warning: .dynamic section for "/usr/local/lib/libbitcoin-client.so.0" is not at the expected address (wrong library or version mismatch?)

warning: .dynamic section for "/usr/local/lib/libbitcoin-explorer.so.0" is not at the expected address (wrong library or version mismatch?)

warning: .dynamic section for "/usr/local//lib/libsecp256k1.so.0" is not at the expected address (wrong library or version mismatch?)

warning: .dynamic section for "/usr/local//lib/libbitcoin-network.so.0" is not at the expected address (wrong library or version mismatch?)
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `./witness_node -d ./witness_node_data_dir --replay-blockchain'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  __GI_abort () at abort.c:107
107	abort.c: No such file or directory.
[Current thread is 1 (Thread 0x7f8e58ff9700 (LWP 15))]
(gdb) bt
#0  __GI_abort () at abort.c:107
#1  0x00007f8e65512729 in __assert_fail_base (fmt=0x7f8e656a8588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x5617882c1310 "non_preemptable_scope_count == 0", file=0x5617882c1168 "/home/peerplays/src/peerplays/libraries/fc/src/thread/thread_d.hpp", line=369, function=<optimized out>) at assert.c:92
#2  0x00007f8e65523fd6 in __GI___assert_fail (assertion=0x5617882c1310 "non_preemptable_scope_count == 0", file=0x5617882c1168 "/home/peerplays/src/peerplays/libraries/fc/src/thread/thread_d.hpp", line=369, function=0x5617882c12e0 "bool fc::thread_d::start_next_fiber(bool)") at assert.c:101
#3  0x000056178784e000 in fc::thread_d::start_next_fiber (this=0x7f8e34000b60, reschedule=false) at /home/peerplays/src/peerplays/libraries/fc/src/thread/thread_d.hpp:369
#4  0x0000561787846bb3 in fc::thread::yield (this=0x56178a127550, reschedule=false) at /home/peerplays/src/peerplays/libraries/fc/src/thread/thread.cpp:268
#5  0x000056178785f459 in fc::mutex::lock (this=0x56178a182640) at /home/peerplays/src/peerplays/libraries/fc/src/thread/mutex.cpp:177
#6  0x000056178640597c in fc::scoped_lock<fc::mutex>::scoped_lock (this=0x7f8e585f6a28, l=...) at /home/peerplays/src/peerplays/libraries/fc/include/fc/thread/scoped_lock.hpp:7
#7  0x000056178637cf4f in graphene::net::detail::node_impl::fetch_sync_items_loop (this=0x56178a182270) at /home/peerplays/src/peerplays/libraries/net/node.cpp:1185
#8  0x00005617863cdded in graphene::net::detail::node_impl::<lambda()>::operator()(void) const (__closure=0x7f8e340677f8) at /home/peerplays/src/peerplays/libraries/net/node.cpp:4958
#9  0x00005617863fd08e in fc::detail::void_functor_run<graphene::net::detail::node_impl::connect_to_p2p_network()::<lambda()> >::run(void *, void *) (functor=0x7f8e340677f8, prom=0x7f8e340677f0) at /home/peerplays/src/peerplays/libraries/fc/include/fc/thread/task.hpp:83
#10 0x000056178785d3e8 in fc::task_base::run_impl (this=0x7f8e34067800) at /home/peerplays/src/peerplays/libraries/fc/src/thread/task.cpp:43
#11 0x000056178785d364 in fc::task_base::run (this=0x7f8e34067800) at /home/peerplays/src/peerplays/libraries/fc/src/thread/task.cpp:32
#12 0x000056178784ef12 in fc::thread_d::run_next_task (this=0x7f8e34000b60) at /home/peerplays/src/peerplays/libraries/fc/src/thread/thread_d.hpp:520
#13 0x000056178784f2c1 in fc::thread_d::process_tasks (this=0x7f8e34000b60) at /home/peerplays/src/peerplays/libraries/fc/src/thread/thread_d.hpp:568
#14 0x000056178784e891 in fc::thread_d::start_process_tasks (my=...) at /home/peerplays/src/peerplays/libraries/fc/src/thread/thread_d.hpp:499
#15 0x0000561787ac3e2f in make_fcontext ()
#16 0x0000000000000000 in ?? ()
(gdb)

@vampik same steps and one witness node crashed. ``` kiran@warpspeed:/home$ sudo coredumpctl debug 4182006 PID: 4182006 (witness_node) UID: 0 (root) GID: 0 (root) Signal: 11 (SEGV) Timestamp: Sat 2023-03-18 11:49:38 +0530 (1 day 4h ago) Command Line: ./witness_node -d ./witness_node_data_dir --replay-blockchain Executable: /home/peerplays/src/peerplays/build/programs/witness_node/witness_node Control Group: /docker/305dcb1d73ba446f8b0db24edd3c92914ce517eef02bae8eb08fa23ac4ba178c Slice: -.slice Boot ID: f9ea01af03534b149efc0d39fb9d10e3 Machine ID: a388625228614d18b5373ad1be39a93f Hostname: warpspeed Storage: /var/lib/systemd/coredump/core.witness_node.0.f9ea01af03534b149efc0d39fb9d10e3.4182006.1679120378000000000000.lz4 Message: Process 4182006 (witness_node) of user 0 dumped core. Stack trace of thread 15: #0 0x00007f8e65512941 __GI_abort (libc.so.6 + 0x22941) #1 0x00007f8e65512729 __assert_fail_base (libc.so.6 + 0x22729) #2 0x00007f8e65523fd6 __GI___assert_fail (libc.so.6 + 0x33fd6) #3 0x000056178784e000 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6d98000) #4 0x0000561787846bb3 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6d90bb3) #5 0x000056178785f459 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6da9459) #6 0x000056178640597c n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x594f97c) #7 0x000056178637cf4f n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x58c6f4f) #8 0x00005617863cdded n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x5917ded) #9 0x00005617863fd08e n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x594708e) #10 0x000056178785d3e8 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6da73e8) #11 0x000056178785d364 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6da7364) #12 0x000056178784ef12 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6d98f12) #13 0x000056178784f2c1 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6d992c1) #14 0x000056178784e891 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6d98891) #15 0x0000561787ac3e2f n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x700de2f) GNU gdb (Ubuntu 9.2-0ubuntu1~20.04.1) 9.2 Copyright (C) 2020 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /home/peerplays/src/peerplays/build/programs/witness_node/witness_node... [New LWP 15] [New LWP 10] [New LWP 7] [New LWP 1] [New LWP 13] [New LWP 8] [New LWP 16] [New LWP 19] [New LWP 9] [New LWP 22] [New LWP 12] [New LWP 17] [New LWP 20] [New LWP 18] [New LWP 1343] [New LWP 1342] [New LWP 24] [New LWP 21] [New LWP 1344] [New LWP 23] [New LWP 14] [New LWP 11] warning: .dynamic section for "/usr/local/lib/libbitcoin-protocol.so.0" is not at the expected address (wrong library or version mismatch?) warning: .dynamic section for "/usr/local/lib/libbitcoin-client.so.0" is not at the expected address (wrong library or version mismatch?) warning: .dynamic section for "/usr/local/lib/libbitcoin-explorer.so.0" is not at the expected address (wrong library or version mismatch?) warning: .dynamic section for "/usr/local//lib/libsecp256k1.so.0" is not at the expected address (wrong library or version mismatch?) warning: .dynamic section for "/usr/local//lib/libbitcoin-network.so.0" is not at the expected address (wrong library or version mismatch?) [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `./witness_node -d ./witness_node_data_dir --replay-blockchain'. Program terminated with signal SIGSEGV, Segmentation fault. #0 __GI_abort () at abort.c:107 107 abort.c: No such file or directory. [Current thread is 1 (Thread 0x7f8e58ff9700 (LWP 15))] (gdb) bt #0 __GI_abort () at abort.c:107 #1 0x00007f8e65512729 in __assert_fail_base (fmt=0x7f8e656a8588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x5617882c1310 "non_preemptable_scope_count == 0", file=0x5617882c1168 "/home/peerplays/src/peerplays/libraries/fc/src/thread/thread_d.hpp", line=369, function=<optimized out>) at assert.c:92 #2 0x00007f8e65523fd6 in __GI___assert_fail (assertion=0x5617882c1310 "non_preemptable_scope_count == 0", file=0x5617882c1168 "/home/peerplays/src/peerplays/libraries/fc/src/thread/thread_d.hpp", line=369, function=0x5617882c12e0 "bool fc::thread_d::start_next_fiber(bool)") at assert.c:101 #3 0x000056178784e000 in fc::thread_d::start_next_fiber (this=0x7f8e34000b60, reschedule=false) at /home/peerplays/src/peerplays/libraries/fc/src/thread/thread_d.hpp:369 #4 0x0000561787846bb3 in fc::thread::yield (this=0x56178a127550, reschedule=false) at /home/peerplays/src/peerplays/libraries/fc/src/thread/thread.cpp:268 #5 0x000056178785f459 in fc::mutex::lock (this=0x56178a182640) at /home/peerplays/src/peerplays/libraries/fc/src/thread/mutex.cpp:177 #6 0x000056178640597c in fc::scoped_lock<fc::mutex>::scoped_lock (this=0x7f8e585f6a28, l=...) at /home/peerplays/src/peerplays/libraries/fc/include/fc/thread/scoped_lock.hpp:7 #7 0x000056178637cf4f in graphene::net::detail::node_impl::fetch_sync_items_loop (this=0x56178a182270) at /home/peerplays/src/peerplays/libraries/net/node.cpp:1185 #8 0x00005617863cdded in graphene::net::detail::node_impl::<lambda()>::operator()(void) const (__closure=0x7f8e340677f8) at /home/peerplays/src/peerplays/libraries/net/node.cpp:4958 #9 0x00005617863fd08e in fc::detail::void_functor_run<graphene::net::detail::node_impl::connect_to_p2p_network()::<lambda()> >::run(void *, void *) (functor=0x7f8e340677f8, prom=0x7f8e340677f0) at /home/peerplays/src/peerplays/libraries/fc/include/fc/thread/task.hpp:83 #10 0x000056178785d3e8 in fc::task_base::run_impl (this=0x7f8e34067800) at /home/peerplays/src/peerplays/libraries/fc/src/thread/task.cpp:43 #11 0x000056178785d364 in fc::task_base::run (this=0x7f8e34067800) at /home/peerplays/src/peerplays/libraries/fc/src/thread/task.cpp:32 #12 0x000056178784ef12 in fc::thread_d::run_next_task (this=0x7f8e34000b60) at /home/peerplays/src/peerplays/libraries/fc/src/thread/thread_d.hpp:520 #13 0x000056178784f2c1 in fc::thread_d::process_tasks (this=0x7f8e34000b60) at /home/peerplays/src/peerplays/libraries/fc/src/thread/thread_d.hpp:568 #14 0x000056178784e891 in fc::thread_d::start_process_tasks (my=...) at /home/peerplays/src/peerplays/libraries/fc/src/thread/thread_d.hpp:499 #15 0x0000561787ac3e2f in make_fcontext () #16 0x0000000000000000 in ?? () (gdb) ```

vampik commented

2023-03-20 11:46:31 +00:00

(Migrated from gitlab.com)

mentioned in issue #433

vampik commented

2023-03-21 09:36:26 +00:00

(Migrated from gitlab.com)

mentioned in commit 5e75e8043a

mentioned in commit 5e75e8043a43496dcc7ad689250b94488bc55ec4

vampik commented

2023-03-21 09:37:19 +00:00

(Migrated from gitlab.com)

I added some new guards, try update bug/501-on_connection_closed branch

I added some new guards, try update `bug/501-on_connection_closed` branch

vampik commented

2023-03-21 09:38:30 +00:00

(Migrated from gitlab.com)

Tried several times to start. Stop / start numerous different witnesses, can't reproduce the crash.
Example of one of iteration:

Vladislav, [20.03.2023 11:38]
CONTAINER ID   IMAGE                             COMMAND                  CREATED          STATUS                      PORTS                                                                                                                                                   NAMES
6a942c7de653   peerplays03                       "./witness_node -d .…"   26 minutes ago   Up 21 minutes               22/tcp, 0.0.0.0:8092->8090/tcp, :::8092->8090/tcp, 0.0.0.0:9779->9777/tcp, :::9779->9777/tcp                                                            peerplays-qa-environment_peerplays03_1
29dcb14dc818   peerplays07                       "./witness_node -d .…"   26 minutes ago   Up 26 minutes               22/tcp, 0.0.0.0:8096->8090/tcp, :::8096->8090/tcp, 0.0.0.0:9783->9777/tcp, :::9783->9777/tcp                                                            peerplays-qa-environment_peerplays07_1
618a52248819   peerplays11                       "./witness_node -d .…"   26 minutes ago   Up 10 minutes               22/tcp, 0.0.0.0:8100->8090/tcp, :::8100->8090/tcp, 0.0.0.0:9787->9777/tcp, :::9787->9777/tcp                                                            peerplays-qa-environment_peerplays11_1
8f6ed2db207b   peerplays04                       "./witness_node -d .…"   26 minutes ago   Up 26 minutes               22/tcp, 0.0.0.0:8093->8090/tcp, :::8093->8090/tcp, 0.0.0.0:9780->9777/tcp, :::9780->9777/tcp                                                            peerplays-qa-environment_peerplays04_1
7ecca4f38234   peerplays08                       "./witness_node -d .…"   26 minutes ago   Up 21 minutes               22/tcp, 0.0.0.0:8097->8090/tcp, :::8097->8090/tcp, 0.0.0.0:9784->9777/tcp, :::9784->9777/tcp                                                            peerplays-qa-environment_peerplays08_1
a381f42b69b2   peerplays16                       "./witness_node -d .…"   26 minutes ago   Up 26 minutes               22/tcp, 0.0.0.0:8105->8090/tcp, :::8105->8090/tcp, 0.0.0.0:9792->9777/tcp, :::9792->9777/tcp                                                            peerplays-qa-environment_peerplays16_1
911ab7e1924b   peerplays06                       "./witness_node -d .…"   26 minutes ago   Up 10 minutes               22/tcp, 0.0.0.0:8095->8090/tcp, :::8095->8090/tcp, 0.0.0.0:9782->9777/tcp, :::9782->9777/tcp                                                            peerplays-qa-environment_peerplays06_1
047b3c2426a0   peerplays12                       "./witness_node -d .…"   26 minutes ago   Up 26 minutes               22/tcp, 0.0.0.0:8101->8090/tcp, :::8101->8090/tcp, 0.0.0.0:9788->9777/tcp, :::9788->9777/tcp                                                            peerplays-qa-environment_peerplays12_1
34193a9acaa0   peerplays13                       "./witness_node -d .…"   26 minutes ago   Up 26 minutes               22/tcp, 0.0.0.0:8102->8090/tcp, :::8102->8090/tcp, 0.0.0.0:9789->9777/tcp, :::9789->9777/tcp                                                            peerplays-qa-environment_peerplays13_1
4841c1e41ddf   peerplays14                       "./witness_node -d .…"   26 minutes ago   Up 10 minutes               22/tcp, 0.0.0.0:8103->8090/tcp, :::8103->8090/tcp, 0.0.0.0:9790->9777/tcp, :::9790->9777/tcp                                                            peerplays-qa-environment_peerplays14_1
005edf4b285f   peerplays02                       "./witness_node -d .…"   26 minutes ago   Up 26 minutes               22/tcp, 0.0.0.0:8091->8090/tcp, :::8091->8090/tcp, 0.0.0.0:9778->9777/tcp, :::9778->9777/tcp                                                            peerplays-qa-environment_peerplays02_1
95a1d014d20c   peerplays09                       "./witness_node -d .…"   26 minutes ago   Up 26 minutes               22/tcp, 0.0.0.0:8098->8090/tcp, :::8098->8090/tcp, 0.0.0.0:9785->9777/tcp, :::9785->9777/tcp                                                            peerplays-qa-environment_peerplays09_1
e210e034d337   peerplays05                       "./witness_node -d .…"   26 minutes ago   Up 26 minutes               22/tcp, 0.0.0.0:8094->8090/tcp, :::8094->8090/tcp, 0.0.0.0:9781->9777/tcp, :::9781->9777/tcp                                                            peerplays-qa-environment_peerplays05_1
aecd1063f587   peerplays10                       "./witness_node -d .…"   26 minutes ago   Up 10 minutes               22/tcp, 0.0.0.0:8099->8090/tcp, :::8099->8090/tcp, 0.0.0.0:9786->9777/tcp, :::9786->9777/tcp                                                            peerplays-qa-environment_peerplays10_1
66737d0a227d   peerplays15                       "./witness_node -d .…"   26 minutes ago   Up 26 minutes               22/tcp, 0.0.0.0:8104->8090/tcp, :::8104->8090/tcp, 0.0.0.0:9791->9777/tcp, :::9791->9777/tcp                                                            peerplays-qa-environment_peerplays15_1

Tried several times to start. Stop / start numerous different witnesses, can't reproduce the crash. Example of one of iteration: ``` Vladislav, [20.03.2023 11:38] CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 6a942c7de653 peerplays03 "./witness_node -d .…" 26 minutes ago Up 21 minutes 22/tcp, 0.0.0.0:8092->8090/tcp, :::8092->8090/tcp, 0.0.0.0:9779->9777/tcp, :::9779->9777/tcp peerplays-qa-environment_peerplays03_1 29dcb14dc818 peerplays07 "./witness_node -d .…" 26 minutes ago Up 26 minutes 22/tcp, 0.0.0.0:8096->8090/tcp, :::8096->8090/tcp, 0.0.0.0:9783->9777/tcp, :::9783->9777/tcp peerplays-qa-environment_peerplays07_1 618a52248819 peerplays11 "./witness_node -d .…" 26 minutes ago Up 10 minutes 22/tcp, 0.0.0.0:8100->8090/tcp, :::8100->8090/tcp, 0.0.0.0:9787->9777/tcp, :::9787->9777/tcp peerplays-qa-environment_peerplays11_1 8f6ed2db207b peerplays04 "./witness_node -d .…" 26 minutes ago Up 26 minutes 22/tcp, 0.0.0.0:8093->8090/tcp, :::8093->8090/tcp, 0.0.0.0:9780->9777/tcp, :::9780->9777/tcp peerplays-qa-environment_peerplays04_1 7ecca4f38234 peerplays08 "./witness_node -d .…" 26 minutes ago Up 21 minutes 22/tcp, 0.0.0.0:8097->8090/tcp, :::8097->8090/tcp, 0.0.0.0:9784->9777/tcp, :::9784->9777/tcp peerplays-qa-environment_peerplays08_1 a381f42b69b2 peerplays16 "./witness_node -d .…" 26 minutes ago Up 26 minutes 22/tcp, 0.0.0.0:8105->8090/tcp, :::8105->8090/tcp, 0.0.0.0:9792->9777/tcp, :::9792->9777/tcp peerplays-qa-environment_peerplays16_1 911ab7e1924b peerplays06 "./witness_node -d .…" 26 minutes ago Up 10 minutes 22/tcp, 0.0.0.0:8095->8090/tcp, :::8095->8090/tcp, 0.0.0.0:9782->9777/tcp, :::9782->9777/tcp peerplays-qa-environment_peerplays06_1 047b3c2426a0 peerplays12 "./witness_node -d .…" 26 minutes ago Up 26 minutes 22/tcp, 0.0.0.0:8101->8090/tcp, :::8101->8090/tcp, 0.0.0.0:9788->9777/tcp, :::9788->9777/tcp peerplays-qa-environment_peerplays12_1 34193a9acaa0 peerplays13 "./witness_node -d .…" 26 minutes ago Up 26 minutes 22/tcp, 0.0.0.0:8102->8090/tcp, :::8102->8090/tcp, 0.0.0.0:9789->9777/tcp, :::9789->9777/tcp peerplays-qa-environment_peerplays13_1 4841c1e41ddf peerplays14 "./witness_node -d .…" 26 minutes ago Up 10 minutes 22/tcp, 0.0.0.0:8103->8090/tcp, :::8103->8090/tcp, 0.0.0.0:9790->9777/tcp, :::9790->9777/tcp peerplays-qa-environment_peerplays14_1 005edf4b285f peerplays02 "./witness_node -d .…" 26 minutes ago Up 26 minutes 22/tcp, 0.0.0.0:8091->8090/tcp, :::8091->8090/tcp, 0.0.0.0:9778->9777/tcp, :::9778->9777/tcp peerplays-qa-environment_peerplays02_1 95a1d014d20c peerplays09 "./witness_node -d .…" 26 minutes ago Up 26 minutes 22/tcp, 0.0.0.0:8098->8090/tcp, :::8098->8090/tcp, 0.0.0.0:9785->9777/tcp, :::9785->9777/tcp peerplays-qa-environment_peerplays09_1 e210e034d337 peerplays05 "./witness_node -d .…" 26 minutes ago Up 26 minutes 22/tcp, 0.0.0.0:8094->8090/tcp, :::8094->8090/tcp, 0.0.0.0:9781->9777/tcp, :::9781->9777/tcp peerplays-qa-environment_peerplays05_1 aecd1063f587 peerplays10 "./witness_node -d .…" 26 minutes ago Up 10 minutes 22/tcp, 0.0.0.0:8099->8090/tcp, :::8099->8090/tcp, 0.0.0.0:9786->9777/tcp, :::9786->9777/tcp peerplays-qa-environment_peerplays10_1 66737d0a227d peerplays15 "./witness_node -d .…" 26 minutes ago Up 26 minutes 22/tcp, 0.0.0.0:8104->8090/tcp, :::8104->8090/tcp, 0.0.0.0:9791->9777/tcp, :::9791->9777/tcp peerplays-qa-environment_peerplays15_1 ```

prandnum commented

2023-03-28 18:29:03 +00:00

(Migrated from gitlab.com)

@vampik Tried stop/start of peerplays02/03/04/05 with a sleep of 300s and this crash was seen when the witness node was stopped the second time.

Coredump details given below:

kiran@warpspeed:/home$ sudo coredumpctl debug 1473806
           PID: 1473806 (witness_node)
           UID: 0 (root)
           GID: 0 (root)
        Signal: 11 (SEGV)
     Timestamp: Tue 2023-03-28 22:06:18 +0530 (1h 47min ago)
  Command Line: ./witness_node -d ./witness_node_data_dir --replay-blockchain
    Executable: /home/peerplays/src/peerplays/build/programs/witness_node/witness_node
 Control Group: /docker/d04f5441a88811f089744d83e013334ab2b70753aedfd925c3a81e987109e9d4
         Slice: -.slice
       Boot ID: df3dc7b78e3a4833a1b0c1dd75a80a53
    Machine ID: a388625228614d18b5373ad1be39a93f
      Hostname: warpspeed
       Storage: /var/lib/systemd/coredump/core.witness_node.0.df3dc7b78e3a4833a1b0c1dd75a80a53.1473806.1680021378000000000000.lz4
       Message: Process 1473806 (witness_node) of user 0 dumped core.
                
                Stack trace of thread 1:
                #0  0x000055b47c770a6c n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x4fe9a6c)
                #1  0x000055b47c77da76 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x4ff6a76)
                #2  0x000055b47c77d8e5 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x4ff68e5)
                #3  0x000055b47c77d621 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x4ff6621)
                #4  0x000055b47c77f433 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x4ff8433)
                #5  0x000055b47e52d1a0 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6da61a0)
                #6  0x000055b47e52d11c n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6da611c)
                #7  0x000055b47e51ecca n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6d97cca)
                #8  0x000055b47e51f079 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6d98079)
                #9  0x000055b47e51e649 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6d97649)
                #10 0x000055b47e793bdf n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x700cbdf)

GNU gdb (Ubuntu 9.2-0ubuntu1~20.04.1) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/peerplays/src/peerplays/build/programs/witness_node/witness_node...
[New LWP 1]
[New LWP 11]
[New LWP 10]
[New LWP 7]
[New LWP 13]
[New LWP 17]
[New LWP 14]
[New LWP 18]
[New LWP 9]
[New LWP 16]
[New LWP 12]
[New LWP 8]

warning: .dynamic section for "/usr/local/lib/libbitcoin-protocol.so.0" is not at the expected address (wrong library or version mismatch?)

warning: .dynamic section for "/usr/local/lib/libbitcoin-client.so.0" is not at the expected address (wrong library or version mismatch?)

warning: .dynamic section for "/usr/local/lib/libbitcoin-explorer.so.0" is not at the expected address (wrong library or version mismatch?)

warning: .dynamic section for "/usr/local//lib/libsecp256k1.so.0" is not at the expected address (wrong library or version mismatch?)

warning: .dynamic section for "/usr/local//lib/libbitcoin-network.so.0" is not at the expected address (wrong library or version mismatch?)
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `./witness_node -d ./witness_node_data_dir --replay-blockchain'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000055b47c770a6c in graphene::peerplays_sidechain::hive_rpc_client::block_api_get_block[abi:cxx11](unsigned int) (this=0x222c7d2231323030, block_number=1491) at /home/peerplays/src/peerplays/libraries/plugins/peerplays_sidechain/sidechain_net_handler_hive.cpp:44
44	   return send_post_request("block_api.get_block", params, debug_rpc_calls);
[Current thread is 1 (Thread 0x7fb6c6fbe0c0 (LWP 1))]
(gdb) bt
#0  0x000055b47c770a6c in graphene::peerplays_sidechain::hive_rpc_client::block_api_get_block[abi:cxx11](unsigned int) (this=0x222c7d2231323030, block_number=1491) at /home/peerplays/src/peerplays/libraries/plugins/peerplays_sidechain/sidechain_net_handler_hive.cpp:44
#1  0x000055b47c77da76 in graphene::peerplays_sidechain::sidechain_net_handler_hive::handle_event (this=0x55b48044bbf0, event_data="1491") at /home/peerplays/src/peerplays/libraries/plugins/peerplays_sidechain/sidechain_net_handler_hive.cpp:933
#2  0x000055b47c77d8e5 in graphene::peerplays_sidechain::sidechain_net_handler_hive::hive_listener_loop (this=0x55b48044bbf0) at /home/peerplays/src/peerplays/libraries/plugins/peerplays_sidechain/sidechain_net_handler_hive.cpp:915
#3  0x000055b47c77d621 in graphene::peerplays_sidechain::sidechain_net_handler_hive::<lambda()>::operator()(void) const (__closure=0x55b4803f2118) at /home/peerplays/src/peerplays/libraries/plugins/peerplays_sidechain/sidechain_net_handler_hive.cpp:898
#4  0x000055b47c77f433 in fc::detail::void_functor_run<graphene::peerplays_sidechain::sidechain_net_handler_hive::schedule_hive_listener()::<lambda()> >::run(void *, void *) (functor=0x55b4803f2118, prom=0x55b4803f2110) at /home/peerplays/src/peerplays/libraries/fc/include/fc/thread/task.hpp:83
#5  0x000055b47e52d1a0 in fc::task_base::run_impl (this=0x55b4803f2120) at /home/peerplays/src/peerplays/libraries/fc/src/thread/task.cpp:43
#6  0x000055b47e52d11c in fc::task_base::run (this=0x55b4803f2120) at /home/peerplays/src/peerplays/libraries/fc/src/thread/task.cpp:32
#7  0x000055b47e51ecca in fc::thread_d::run_next_task (this=0x55b480302b00) at /home/peerplays/src/peerplays/libraries/fc/src/thread/thread_d.hpp:520
#8  0x000055b47e51f079 in fc::thread_d::process_tasks (this=0x55b480302b00) at /home/peerplays/src/peerplays/libraries/fc/src/thread/thread_d.hpp:568
#9  0x000055b47e51e649 in fc::thread_d::start_process_tasks (my=...) at /home/peerplays/src/peerplays/libraries/fc/src/thread/thread_d.hpp:499
#10 0x000055b47e793bdf in make_fcontext ()
#11 0x0000000000000000 in ?? ()
(gdb)

kiran@warpspeed:~/PBSA/501/peerplays-utils/peerplays-qa-environment$ docker exec -it f0a63bf04b51 bash
root@f0a63bf04b51:~/peerplays-network# cd ../src/peerplays/
root@f0a63bf04b51:~/src/peerplays# git log -3
commit fba1c0cb95f71f239509c539df60e51c82973660 (HEAD -> master, origin/bug/501-on_connection_closed)
Author: Vlad Dobromyslov <vdobrik@peerplays.com>
Date:   Mon Mar 20 11:02:02 2023 +0300

    Atomic bool for flags

commit 5e75e8043a43496dcc7ad689250b94488bc55ec4
Author: Vlad Dobromyslov <vdobrik@peerplays.com>
Date:   Mon Mar 20 10:55:02 2023 +0300

    #501 Fix _active_connections mutex

commit 5203869a9d876b8f0d9549fb19b361fc485dadb0
Author: Vlad Dobromyslov <vdobrik@peerplays.com>
Date:   Tue Mar 14 10:29:38 2023 +0300

    #501 on_connection_closed - clear maps of connections
root@f0a63bf04b51:~/src/peerplays#

@vampik Tried stop/start of peerplays02/03/04/05 with a sleep of 300s and this crash was seen when the witness node was stopped the second time. Coredump details given below: ``` kiran@warpspeed:/home$ sudo coredumpctl debug 1473806 PID: 1473806 (witness_node) UID: 0 (root) GID: 0 (root) Signal: 11 (SEGV) Timestamp: Tue 2023-03-28 22:06:18 +0530 (1h 47min ago) Command Line: ./witness_node -d ./witness_node_data_dir --replay-blockchain Executable: /home/peerplays/src/peerplays/build/programs/witness_node/witness_node Control Group: /docker/d04f5441a88811f089744d83e013334ab2b70753aedfd925c3a81e987109e9d4 Slice: -.slice Boot ID: df3dc7b78e3a4833a1b0c1dd75a80a53 Machine ID: a388625228614d18b5373ad1be39a93f Hostname: warpspeed Storage: /var/lib/systemd/coredump/core.witness_node.0.df3dc7b78e3a4833a1b0c1dd75a80a53.1473806.1680021378000000000000.lz4 Message: Process 1473806 (witness_node) of user 0 dumped core. Stack trace of thread 1: #0 0x000055b47c770a6c n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x4fe9a6c) #1 0x000055b47c77da76 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x4ff6a76) #2 0x000055b47c77d8e5 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x4ff68e5) #3 0x000055b47c77d621 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x4ff6621) #4 0x000055b47c77f433 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x4ff8433) #5 0x000055b47e52d1a0 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6da61a0) #6 0x000055b47e52d11c n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6da611c) #7 0x000055b47e51ecca n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6d97cca) #8 0x000055b47e51f079 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6d98079) #9 0x000055b47e51e649 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6d97649) #10 0x000055b47e793bdf n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x700cbdf) GNU gdb (Ubuntu 9.2-0ubuntu1~20.04.1) 9.2 Copyright (C) 2020 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /home/peerplays/src/peerplays/build/programs/witness_node/witness_node... [New LWP 1] [New LWP 11] [New LWP 10] [New LWP 7] [New LWP 13] [New LWP 17] [New LWP 14] [New LWP 18] [New LWP 9] [New LWP 16] [New LWP 12] [New LWP 8] warning: .dynamic section for "/usr/local/lib/libbitcoin-protocol.so.0" is not at the expected address (wrong library or version mismatch?) warning: .dynamic section for "/usr/local/lib/libbitcoin-client.so.0" is not at the expected address (wrong library or version mismatch?) warning: .dynamic section for "/usr/local/lib/libbitcoin-explorer.so.0" is not at the expected address (wrong library or version mismatch?) warning: .dynamic section for "/usr/local//lib/libsecp256k1.so.0" is not at the expected address (wrong library or version mismatch?) warning: .dynamic section for "/usr/local//lib/libbitcoin-network.so.0" is not at the expected address (wrong library or version mismatch?) [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `./witness_node -d ./witness_node_data_dir --replay-blockchain'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x000055b47c770a6c in graphene::peerplays_sidechain::hive_rpc_client::block_api_get_block[abi:cxx11](unsigned int) (this=0x222c7d2231323030, block_number=1491) at /home/peerplays/src/peerplays/libraries/plugins/peerplays_sidechain/sidechain_net_handler_hive.cpp:44 44 return send_post_request("block_api.get_block", params, debug_rpc_calls); [Current thread is 1 (Thread 0x7fb6c6fbe0c0 (LWP 1))] (gdb) bt #0 0x000055b47c770a6c in graphene::peerplays_sidechain::hive_rpc_client::block_api_get_block[abi:cxx11](unsigned int) (this=0x222c7d2231323030, block_number=1491) at /home/peerplays/src/peerplays/libraries/plugins/peerplays_sidechain/sidechain_net_handler_hive.cpp:44 #1 0x000055b47c77da76 in graphene::peerplays_sidechain::sidechain_net_handler_hive::handle_event (this=0x55b48044bbf0, event_data="1491") at /home/peerplays/src/peerplays/libraries/plugins/peerplays_sidechain/sidechain_net_handler_hive.cpp:933 #2 0x000055b47c77d8e5 in graphene::peerplays_sidechain::sidechain_net_handler_hive::hive_listener_loop (this=0x55b48044bbf0) at /home/peerplays/src/peerplays/libraries/plugins/peerplays_sidechain/sidechain_net_handler_hive.cpp:915 #3 0x000055b47c77d621 in graphene::peerplays_sidechain::sidechain_net_handler_hive::<lambda()>::operator()(void) const (__closure=0x55b4803f2118) at /home/peerplays/src/peerplays/libraries/plugins/peerplays_sidechain/sidechain_net_handler_hive.cpp:898 #4 0x000055b47c77f433 in fc::detail::void_functor_run<graphene::peerplays_sidechain::sidechain_net_handler_hive::schedule_hive_listener()::<lambda()> >::run(void *, void *) (functor=0x55b4803f2118, prom=0x55b4803f2110) at /home/peerplays/src/peerplays/libraries/fc/include/fc/thread/task.hpp:83 #5 0x000055b47e52d1a0 in fc::task_base::run_impl (this=0x55b4803f2120) at /home/peerplays/src/peerplays/libraries/fc/src/thread/task.cpp:43 #6 0x000055b47e52d11c in fc::task_base::run (this=0x55b4803f2120) at /home/peerplays/src/peerplays/libraries/fc/src/thread/task.cpp:32 #7 0x000055b47e51ecca in fc::thread_d::run_next_task (this=0x55b480302b00) at /home/peerplays/src/peerplays/libraries/fc/src/thread/thread_d.hpp:520 #8 0x000055b47e51f079 in fc::thread_d::process_tasks (this=0x55b480302b00) at /home/peerplays/src/peerplays/libraries/fc/src/thread/thread_d.hpp:568 #9 0x000055b47e51e649 in fc::thread_d::start_process_tasks (my=...) at /home/peerplays/src/peerplays/libraries/fc/src/thread/thread_d.hpp:499 #10 0x000055b47e793bdf in make_fcontext () #11 0x0000000000000000 in ?? () (gdb) ``` ``` kiran@warpspeed:~/PBSA/501/peerplays-utils/peerplays-qa-environment$ docker exec -it f0a63bf04b51 bash root@f0a63bf04b51:~/peerplays-network# cd ../src/peerplays/ root@f0a63bf04b51:~/src/peerplays# git log -3 commit fba1c0cb95f71f239509c539df60e51c82973660 (HEAD -> master, origin/bug/501-on_connection_closed) Author: Vlad Dobromyslov <vdobrik@peerplays.com> Date: Mon Mar 20 11:02:02 2023 +0300 Atomic bool for flags commit 5e75e8043a43496dcc7ad689250b94488bc55ec4 Author: Vlad Dobromyslov <vdobrik@peerplays.com> Date: Mon Mar 20 10:55:02 2023 +0300 #501 Fix _active_connections mutex commit 5203869a9d876b8f0d9549fb19b361fc485dadb0 Author: Vlad Dobromyslov <vdobrik@peerplays.com> Date: Tue Mar 14 10:29:38 2023 +0300 #501 on_connection_closed - clear maps of connections root@f0a63bf04b51:~/src/peerplays# ```

after stopping witness node peerplays02 other witness nodes crashed. #501