Investigate if transaction can cause network fork/halt

serkixenos commented

2022-10-07 18:54:21 +00:00

(Migrated from gitlab.com)

This happened on mainnet. The following transaction was broadcasted. There were two operations in the transaction sidechain_address_delete_operation and sidechain_address_add_operation. The intention was to update the sidechain address by removing it, and then adding it again.

Investigate further, is this the actual cause of the problem, focus on how these two, mutually dependant operations, act in the same transaction, what if we used one transaction for each operation, etc...

Find out this: At the moment of sidechain_address_add_operation execution, when accessing database from sidechain_address_add_operation evaluator, do we see sidechain address as deleted or not.

{
   "ref_block_num":12298,
   "ref_block_prefix":1805427571,
   "expiration":"2022-10-07T09:30:04",
   "operations":[
      [
         115,
         {
            "fee":{
               "amount":0,
               "asset_id":"1.3.0"
            },
            "payer":"1.2.14681",
            "sidechain_address_id":"1.38.45",
            "sidechain_address_account":"1.2.14681",
            "sidechain":"bitcoin"
         }
      ],
      [
         113,
         {
            "fee":{
               "amount":0,
               "asset_id":"1.3.0"
            },
            "payer":"1.2.14681",
            "sidechain_address_account":"1.2.14681",
            "sidechain":"bitcoin",
            "deposit_public_key":"022A555344284F324C19C89AEF4DC62B03A7C5BEA5E3E2559F2F5F7B5D3A051396",
            "deposit_address":"",
            "deposit_address_data":"",
            "withdraw_public_key":"0277B4714A84EFAD0D803B684829782B0EED7FAD20E5F0B0B541F9F92893E60846",
            "withdraw_address":"1MehAHhQfrWSpWKMNjyzBsdJSSiovoy4ut"
         }
      ]
   ],
   "extensions":[
      
   ],
   "signatures":[
      "1f4d682033afbf2db809052f6805892db79f97b66aca2fca66de87833fd2f081f364c9b3d860b82e6a7ed4c24298f56f416594fa4680a62ac93a58709c5a5e87a9"
   ]
}

This happened on mainnet. The following transaction was broadcasted. There were two operations in the transaction sidechain_address_delete_operation and sidechain_address_add_operation. The intention was to update the sidechain address by removing it, and then adding it again. Investigate further, is this the actual cause of the problem, focus on how these two, mutually dependant operations, act in the same transaction, what if we used one transaction for each operation, etc... Find out this: At the moment of sidechain_address_add_operation execution, when accessing database from sidechain_address_add_operation evaluator, do we see sidechain address as deleted or not. ``` { "ref_block_num":12298, "ref_block_prefix":1805427571, "expiration":"2022-10-07T09:30:04", "operations":[ [ 115, { "fee":{ "amount":0, "asset_id":"1.3.0" }, "payer":"1.2.14681", "sidechain_address_id":"1.38.45", "sidechain_address_account":"1.2.14681", "sidechain":"bitcoin" } ], [ 113, { "fee":{ "amount":0, "asset_id":"1.3.0" }, "payer":"1.2.14681", "sidechain_address_account":"1.2.14681", "sidechain":"bitcoin", "deposit_public_key":"022A555344284F324C19C89AEF4DC62B03A7C5BEA5E3E2559F2F5F7B5D3A051396", "deposit_address":"", "deposit_address_data":"", "withdraw_public_key":"0277B4714A84EFAD0D803B684829782B0EED7FAD20E5F0B0B541F9F92893E60846", "withdraw_address":"1MehAHhQfrWSpWKMNjyzBsdJSSiovoy4ut" } ] ], "extensions":[ ], "signatures":[ "1f4d682033afbf2db809052f6805892db79f97b66aca2fca66de87833fd2f081f364c9b3d860b82e6a7ed4c24298f56f416594fa4680a62ac93a58709c5a5e87a9" ] } ```

serkixenos commented

2022-10-07 18:58:41 +00:00

(Migrated from gitlab.com)

https://hastebin.com/zoqulilapu.json

serkixenos commented

2022-10-07 19:05:12 +00:00

(Migrated from gitlab.com)

changed the description

hirunda commented

2022-10-10 12:23:08 +00:00

(Migrated from gitlab.com)

assigned to @hirunda

hirunda commented

2022-10-11 11:51:22 +00:00

(Migrated from gitlab.com)

I am able to reproduce the issue (resulting in segfault) manually if I am performing fast enough the following two commands:

delete_sidechain_address account02 1 true
add_sidechain_address account02 bitcoin "02b510a452d6e80f943e4cc85af5cad6c528bda87fc92b821dd246a1a76c175b0d" "" "0212cbe2f328ba614886ca01e1ef48688196f72fbd07b31726f160292c96da20d5" "2N75PEYXU6X556UU6VYNiHMgcCBL8XmWnc1" true

I am working on resolving the issue.

I am able to reproduce the issue (resulting in segfault) manually if I am performing fast enough the following two commands: 1) `delete_sidechain_address account02 1 true ` 2) `add_sidechain_address account02 bitcoin "02b510a452d6e80f943e4cc85af5cad6c528bda87fc92b821dd246a1a76c175b0d" "" "0212cbe2f328ba614886ca01e1ef48688196f72fbd07b31726f160292c96da20d5" "2N75PEYXU6X556UU6VYNiHMgcCBL8XmWnc1" true ` I am working on resolving the issue.

bobinson commented

2022-10-11 14:03:02 +00:00

(Migrated from gitlab.com)

marked this issue as related to NEX#551

bobinson commented

2022-10-11 14:04:19 +00:00

(Migrated from gitlab.com)

mentioned in issue NEX#551

hirunda commented

2022-10-13 14:17:06 +00:00

(Migrated from gitlab.com)

Hi @wsalloum , please try to reproduce the issue before we test with the fix.

Steps to reproduce the issue:

Run witness node
open cli_wallet and unlock
Execute the following commands fast (it's important to execute the second command as fast as possible after first command :

first command: delete_sidechain_address account02 1 true

second command: add_sidechain_address account02 bitcoin "02b510a452d6e80f943e4cc85af5cad6c528bda87fc92b821dd246a1a76c175b0d" "" "0212cbe2f328ba614886ca01e1ef48688196f72fbd07b31726f160292c96da20d5" "2N75PEYXU6X556UU6VYNiHMgcCBL8XmWnc1" true

You should expect segfault in witness node.

Please try to establish consistent reproduction, so you could later test with the fix.

Hi @wsalloum , please try to reproduce the issue before we test with the fix. Steps to reproduce the issue: 1) Run witness node 2) open cli_wallet and unlock 3) Execute the following commands fast (it's important to execute the second command as fast as possible after first command : first command: `delete_sidechain_address account02 1 true ` second command: `add_sidechain_address account02 bitcoin "02b510a452d6e80f943e4cc85af5cad6c528bda87fc92b821dd246a1a76c175b0d" "" "0212cbe2f328ba614886ca01e1ef48688196f72fbd07b31726f160292c96da20d5" "2N75PEYXU6X556UU6VYNiHMgcCBL8XmWnc1" true ` You should expect segfault in witness node. Please try to establish consistent reproduction, so you could later test with the fix.

wsalloum commented

2022-10-14 08:43:42 +00:00

(Migrated from gitlab.com)

Run QA Env
delete_sidechain_address account02 1 true
add_sidechain_address account02 bitcoin "02b510a452d6e80f943e4cc85af5cad6c528bda87fc92b821dd246a1a76c175b0d" "" "0212cbe2f328ba614886ca01e1ef48688196f72fbd07b31726f160292c96da20d5" "2N75PEYXU6X556UU6VYNiHMgcCBL8XmWnc1" true

unlocked >>> add_sidechain_address account02 bitcoin "02b510a452d6e80f943e4cc85af5cad6c528bda87fc92b821dd246a1a76c175b0d" "" "0212cbe2f328ba614886ca01e1ef48688196f72fbd07b31726f160292c96da20d5" "2N75PEYXU6X556UU6VYNiHMgcCBL8XmWnc1" true 
add_sidechain_address account02 bitcoin "02b510a452d6e80f943e4cc85af5cad6c528bda87fc92b821dd246a1a76c175b0d" "" "0212cbe2f328ba614886ca01e1ef48688196f72fbd07b31726f160292c96da20d5" "2N75PEYXU6X556UU6VYNiHMgcCBL8XmWnc1" true 
{
  "ref_block_num": 116,
  "ref_block_prefix": 3650652332,
  "expiration": "2022-10-14T07:58:21",
  "operations": [[
      113,{
        "fee": {
          "amount": 0,
          "asset_id": "1.3.0"
        },
        "payer": "1.2.53",
        "sidechain_address_account": "1.2.53",
        "sidechain": "bitcoin",
        "deposit_public_key": "02b510a452d6e80f943e4cc85af5cad6c528bda87fc92b821dd246a1a76c175b0d",
        "deposit_address": "",
        "deposit_address_data": "",
        "withdraw_public_key": "0212cbe2f328ba614886ca01e1ef48688196f72fbd07b31726f160292c96da20d5",
        "withdraw_address": "2N75PEYXU6X556UU6VYNiHMgcCBL8XmWnc1"
      }
    ]
  ],
  "extensions": [],
  "signatures": [
    "207e4723ea289102d96870174fbf9d5dfcb4fc6292a2c3dbfc6551a2d33321b2a3618605ef15920756a4d81995b90e1d7ea9d2791cc77d1c531d4b2db10efb0f50"
  ]
}

Check the log


7:48 
peerplays-all-in-one_1            | 3468001ms th_a       witness.cpp:191               block_production_loo ] Generated block #115 with timestamp 2022-10-14T07:57:48 at time 2022-10-14T07:57:48
peerplays-all-in-one_1            | 3469850ms th_a       database_api.cpp:2555         get_potential_signat ] trx: {"ref_block_num":0,"ref_block_prefix":0,"expiration":"1970-01-01T00:00:00","operations":[[115,{"fee":{"amount":0,"asset_id":"1.3.0"},"payer":"1.2.53","sidechain_address_id":"1.38.1","sidechain_address_account":"1.2.53","sidechain":"bitcoin"}]],"extensions":[],"signatures":[]} 
peerplays-all-in-one_1            | 3469850ms th_a       database_api.cpp:2583         get_potential_signat ] result: ["TEST8Qynrv4EKjgSVxnnERmNKojVsyJHLWXAPiukoXTXgjrhwuyyHe"] 
peerplays-all-in-one_1            | 3469851ms th_a       database_api.cpp:2529         get_required_signatu ] trx: {"ref_block_num":0,"ref_block_prefix":0,"expiration":"1970-01-01T00:00:00","operations":[[115,{"fee":{"amount":0,"asset_id":"1.3.0"},"payer":"1.2.53","sidechain_address_id":"1.38.1","sidechain_address_account":"1.2.53","sidechain":"bitcoin"}]],"extensions":[],"signatures":[]} available_keys: ["TEST8Qynrv4EKjgSVxnnERmNKojVsyJHLWXAPiukoXTXgjrhwuyyHe"] 
peerplays-all-in-one_1            | 3469851ms th_a       database_api.cpp:2543         get_required_signatu ] result: ["TEST8Qynrv4EKjgSVxnnERmNKojVsyJHLWXAPiukoXTXgjrhwuyyHe"] 
peerplays-all-in-one_1            | 3471000ms th_a       witness.cpp:267               maybe_produce_block  ] slot: 1 scheduled_witness: 1.6.6 scheduled_time: 2022-10-14T07:57:51 now: 2022-10-14T07:57:51 
peerplays-all-in-one_1            | 3471001ms th_a       witness.cpp:191               block_production_loo ] Generated block #116 with timestamp 2022-10-14T07:57:51 at time 2022-10-14T07:57:51
peerplays-all-in-one_1            | 3472084ms th_a       database_api.cpp:2555         get_potential_signat ] trx: {"ref_block_num":0,"ref_block_prefix":0,"expiration":"1970-01-01T00:00:00","operations":[[113,{"fee":{"amount":0,"asset_id":"1.3.0"},"payer":"1.2.53","sidechain_address_account":"1.2.53","sidechain":"bitcoin","deposit_public_key":"02b510a452d6e80f943e4cc85af5cad6c528bda87fc92b821dd246a1a76c175b0d","deposit_address":"","deposit_address_data":"","withdraw_public_key":"0212cbe2f328ba614886ca01e1ef48688196f72fbd07b31726f160292c96da20d5","withdraw_address":"2N75PEYXU6X556UU6VYNiHMgcCBL8XmWnc1"}]],"extensions":[],"signatures":[]} 
peerplays-all-in-one_1            | 3472084ms th_a       database_api.cpp:2583         get_potential_signat ] result: ["TEST8Qynrv4EKjgSVxnnERmNKojVsyJHLWXAPiukoXTXgjrhwuyyHe"] 
peerplays-all-in-one_1            | 3472086ms th_a       database_api.cpp:2529         get_required_signatu ] trx: {"ref_block_num":0,"ref_block_prefix":0,"expiration":"1970-01-01T00:00:00","operations":[[113,{"fee":{"amount":0,"asset_id":"1.3.0"},"payer":"1.2.53","sidechain_address_account":"1.2.53","sidechain":"bitcoin","deposit_public_key":"02b510a452d6e80f943e4cc85af5cad6c528bda87fc92b821dd246a1a76c175b0d","deposit_address":"","deposit_address_data":"","withdraw_public_key":"0212cbe2f328ba614886ca01e1ef48688196f72fbd07b31726f160292c96da20d5","withdraw_address":"2N75PEYXU6X556UU6VYNiHMgcCBL8XmWnc1"}]],"extensions":[],"signatures":[]} available_keys: ["TEST8Qynrv4EKjgSVxnnERmNKojVsyJHLWXAPiukoXTXgjrhwuyyHe"] 
peerplays-all-in-one_1            | 3472086ms th_a       database_api.cpp:2543         get_required_signatu ] result: ["TEST8Qynrv4EKjgSVxnnERmNKojVsyJHLWXAPiukoXTXgjrhwuyyHe"]

Repeat the steps very fast, terminated and got this exception:


peerplays-all-in-one_1            | 721392ms th_a       database_api.cpp:2555         get_potential_signat ] trx: {"ref_block_num":0,"ref_block_prefix":0,"expiration":"1970-01-01T00:00:00","operations":[[115,{"fee":{"amount":0,"asset_id":"1.3.0"},"payer":"1.2.53","sidechain_address_id":"1.38.64","sidechain_address_account":"1.2.53","sidechain":"bitcoin"}]],"extensions":[],"signatures":[]} 
peerplays-all-in-one_1            | 721392ms th_a       database_api.cpp:2583         get_potential_signat ] result: ["TEST8Qynrv4EKjgSVxnnERmNKojVsyJHLWXAPiukoXTXgjrhwuyyHe"] 
peerplays-all-in-one_1            | 721393ms th_a       database_api.cpp:2529         get_required_signatu ] trx: {"ref_block_num":0,"ref_block_prefix":0,"expiration":"1970-01-01T00:00:00","operations":[[115,{"fee":{"amount":0,"asset_id":"1.3.0"},"payer":"1.2.53","sidechain_address_id":"1.38.64","sidechain_address_account":"1.2.53","sidechain":"bitcoin"}]],"extensions":[],"signatures":[]} available_keys: ["TEST8Qynrv4EKjgSVxnnERmNKojVsyJHLWXAPiukoXTXgjrhwuyyHe"] 
peerplays-all-in-one_1            | 721393ms th_a       database_api.cpp:2543         get_required_signatu ] result: ["TEST8Qynrv4EKjgSVxnnERmNKojVsyJHLWXAPiukoXTXgjrhwuyyHe"] 
peerplays-all-in-one_1            | 722402ms th_a       database_api.cpp:2555         get_potential_signat ] trx: {"ref_block_num":0,"ref_block_prefix":0,"expiration":"1970-01-01T00:00:00","operations":[[113,{"fee":{"amount":0,"asset_id":"1.3.0"},"payer":"1.2.53","sidechain_address_account":"1.2.53","sidechain":"bitcoin","deposit_public_key":"02b510a452d6e80f943e4cc85af5cad6c528bda87fc92b821dd246a1a76c175b0d","deposit_address":"","deposit_address_data":"","withdraw_public_key":"0212cbe2f328ba614886ca01e1ef48688196f72fbd07b31726f160292c96da20d5","withdraw_address":"2N75PEYXU6X556UU6VYNiHMgcCBL8XmWnc1"}]],"extensions":[],"signatures":[]} 
peerplays-all-in-one_1            | 722402ms th_a       database_api.cpp:2583         get_potential_signat ] result: ["TEST8Qynrv4EKjgSVxnnERmNKojVsyJHLWXAPiukoXTXgjrhwuyyHe"] 
peerplays-all-in-one_1            | 722403ms th_a       database_api.cpp:2529         get_required_signatu ] trx: {"ref_block_num":0,"ref_block_prefix":0,"expiration":"1970-01-01T00:00:00","operations":[[113,{"fee":{"amount":0,"asset_id":"1.3.0"},"payer":"1.2.53","sidechain_address_account":"1.2.53","sidechain":"bitcoin","deposit_public_key":"02b510a452d6e80f943e4cc85af5cad6c528bda87fc92b821dd246a1a76c175b0d","deposit_address":"","deposit_address_data":"","withdraw_public_key":"0212cbe2f328ba614886ca01e1ef48688196f72fbd07b31726f160292c96da20d5","withdraw_address":"2N75PEYXU6X556UU6VYNiHMgcCBL8XmWnc1"}]],"extensions":[],"signatures":[]} available_keys: ["TEST8Qynrv4EKjgSVxnnERmNKojVsyJHLWXAPiukoXTXgjrhwuyyHe"] 
peerplays-all-in-one_1            | 722403ms th_a       database_api.cpp:2543         get_required_signatu ] result: ["TEST8Qynrv4EKjgSVxnnERmNKojVsyJHLWXAPiukoXTXgjrhwuyyHe"] 
peerplays-all-in-one_1            | 723000ms th_a       witness.cpp:267               maybe_produce_block  ] slot: 1 scheduled_witness: 1.6.8 scheduled_time: 2022-10-14T08:12:03 now: 2022-10-14T08:12:03 
peerplays-all-in-one_1            | 723000ms th_a       undo_database.cpp:39          ~session             ] 10 assert_exception: Assert Exception
peerplays-all-in-one_1            | ok: Could not modify object, most likely an index constraint was violated
peerplays-all-in-one_1            |     {}
peerplays-all-in-one_1            |     th_a  generic_index.hpp:88 modify
peerplays-all-in-one_1            | 
peerplays-all-in-one_1            |     {}
peerplays-all-in-one_1            |     th_a  undo_database.cpp:135 undo
peerplays-all-in-one_1            | terminate called after throwing an instance of 'fc::assert_exception'

- Run QA Env - `delete_sidechain_address account02 1 true` - `add_sidechain_address account02 bitcoin "02b510a452d6e80f943e4cc85af5cad6c528bda87fc92b821dd246a1a76c175b0d" "" "0212cbe2f328ba614886ca01e1ef48688196f72fbd07b31726f160292c96da20d5" "2N75PEYXU6X556UU6VYNiHMgcCBL8XmWnc1" true` ``` unlocked >>> add_sidechain_address account02 bitcoin "02b510a452d6e80f943e4cc85af5cad6c528bda87fc92b821dd246a1a76c175b0d" "" "0212cbe2f328ba614886ca01e1ef48688196f72fbd07b31726f160292c96da20d5" "2N75PEYXU6X556UU6VYNiHMgcCBL8XmWnc1" true add_sidechain_address account02 bitcoin "02b510a452d6e80f943e4cc85af5cad6c528bda87fc92b821dd246a1a76c175b0d" "" "0212cbe2f328ba614886ca01e1ef48688196f72fbd07b31726f160292c96da20d5" "2N75PEYXU6X556UU6VYNiHMgcCBL8XmWnc1" true { "ref_block_num": 116, "ref_block_prefix": 3650652332, "expiration": "2022-10-14T07:58:21", "operations": [[ 113,{ "fee": { "amount": 0, "asset_id": "1.3.0" }, "payer": "1.2.53", "sidechain_address_account": "1.2.53", "sidechain": "bitcoin", "deposit_public_key": "02b510a452d6e80f943e4cc85af5cad6c528bda87fc92b821dd246a1a76c175b0d", "deposit_address": "", "deposit_address_data": "", "withdraw_public_key": "0212cbe2f328ba614886ca01e1ef48688196f72fbd07b31726f160292c96da20d5", "withdraw_address": "2N75PEYXU6X556UU6VYNiHMgcCBL8XmWnc1" } ] ], "extensions": [], "signatures": [ "207e4723ea289102d96870174fbf9d5dfcb4fc6292a2c3dbfc6551a2d33321b2a3618605ef15920756a4d81995b90e1d7ea9d2791cc77d1c531d4b2db10efb0f50" ] } ``` - Check the log ``` 7:48 peerplays-all-in-one_1 | 3468001ms th_a witness.cpp:191 block_production_loo ] Generated block #115 with timestamp 2022-10-14T07:57:48 at time 2022-10-14T07:57:48 peerplays-all-in-one_1 | 3469850ms th_a database_api.cpp:2555 get_potential_signat ] trx: {"ref_block_num":0,"ref_block_prefix":0,"expiration":"1970-01-01T00:00:00","operations":[[115,{"fee":{"amount":0,"asset_id":"1.3.0"},"payer":"1.2.53","sidechain_address_id":"1.38.1","sidechain_address_account":"1.2.53","sidechain":"bitcoin"}]],"extensions":[],"signatures":[]} peerplays-all-in-one_1 | 3469850ms th_a database_api.cpp:2583 get_potential_signat ] result: ["TEST8Qynrv4EKjgSVxnnERmNKojVsyJHLWXAPiukoXTXgjrhwuyyHe"] peerplays-all-in-one_1 | 3469851ms th_a database_api.cpp:2529 get_required_signatu ] trx: {"ref_block_num":0,"ref_block_prefix":0,"expiration":"1970-01-01T00:00:00","operations":[[115,{"fee":{"amount":0,"asset_id":"1.3.0"},"payer":"1.2.53","sidechain_address_id":"1.38.1","sidechain_address_account":"1.2.53","sidechain":"bitcoin"}]],"extensions":[],"signatures":[]} available_keys: ["TEST8Qynrv4EKjgSVxnnERmNKojVsyJHLWXAPiukoXTXgjrhwuyyHe"] peerplays-all-in-one_1 | 3469851ms th_a database_api.cpp:2543 get_required_signatu ] result: ["TEST8Qynrv4EKjgSVxnnERmNKojVsyJHLWXAPiukoXTXgjrhwuyyHe"] peerplays-all-in-one_1 | 3471000ms th_a witness.cpp:267 maybe_produce_block ] slot: 1 scheduled_witness: 1.6.6 scheduled_time: 2022-10-14T07:57:51 now: 2022-10-14T07:57:51 peerplays-all-in-one_1 | 3471001ms th_a witness.cpp:191 block_production_loo ] Generated block #116 with timestamp 2022-10-14T07:57:51 at time 2022-10-14T07:57:51 peerplays-all-in-one_1 | 3472084ms th_a database_api.cpp:2555 get_potential_signat ] trx: {"ref_block_num":0,"ref_block_prefix":0,"expiration":"1970-01-01T00:00:00","operations":[[113,{"fee":{"amount":0,"asset_id":"1.3.0"},"payer":"1.2.53","sidechain_address_account":"1.2.53","sidechain":"bitcoin","deposit_public_key":"02b510a452d6e80f943e4cc85af5cad6c528bda87fc92b821dd246a1a76c175b0d","deposit_address":"","deposit_address_data":"","withdraw_public_key":"0212cbe2f328ba614886ca01e1ef48688196f72fbd07b31726f160292c96da20d5","withdraw_address":"2N75PEYXU6X556UU6VYNiHMgcCBL8XmWnc1"}]],"extensions":[],"signatures":[]} peerplays-all-in-one_1 | 3472084ms th_a database_api.cpp:2583 get_potential_signat ] result: ["TEST8Qynrv4EKjgSVxnnERmNKojVsyJHLWXAPiukoXTXgjrhwuyyHe"] peerplays-all-in-one_1 | 3472086ms th_a database_api.cpp:2529 get_required_signatu ] trx: {"ref_block_num":0,"ref_block_prefix":0,"expiration":"1970-01-01T00:00:00","operations":[[113,{"fee":{"amount":0,"asset_id":"1.3.0"},"payer":"1.2.53","sidechain_address_account":"1.2.53","sidechain":"bitcoin","deposit_public_key":"02b510a452d6e80f943e4cc85af5cad6c528bda87fc92b821dd246a1a76c175b0d","deposit_address":"","deposit_address_data":"","withdraw_public_key":"0212cbe2f328ba614886ca01e1ef48688196f72fbd07b31726f160292c96da20d5","withdraw_address":"2N75PEYXU6X556UU6VYNiHMgcCBL8XmWnc1"}]],"extensions":[],"signatures":[]} available_keys: ["TEST8Qynrv4EKjgSVxnnERmNKojVsyJHLWXAPiukoXTXgjrhwuyyHe"] peerplays-all-in-one_1 | 3472086ms th_a database_api.cpp:2543 get_required_signatu ] result: ["TEST8Qynrv4EKjgSVxnnERmNKojVsyJHLWXAPiukoXTXgjrhwuyyHe"] ``` - Repeat the steps very fast, terminated and got this exception: ``` peerplays-all-in-one_1 | 721392ms th_a database_api.cpp:2555 get_potential_signat ] trx: {"ref_block_num":0,"ref_block_prefix":0,"expiration":"1970-01-01T00:00:00","operations":[[115,{"fee":{"amount":0,"asset_id":"1.3.0"},"payer":"1.2.53","sidechain_address_id":"1.38.64","sidechain_address_account":"1.2.53","sidechain":"bitcoin"}]],"extensions":[],"signatures":[]} peerplays-all-in-one_1 | 721392ms th_a database_api.cpp:2583 get_potential_signat ] result: ["TEST8Qynrv4EKjgSVxnnERmNKojVsyJHLWXAPiukoXTXgjrhwuyyHe"] peerplays-all-in-one_1 | 721393ms th_a database_api.cpp:2529 get_required_signatu ] trx: {"ref_block_num":0,"ref_block_prefix":0,"expiration":"1970-01-01T00:00:00","operations":[[115,{"fee":{"amount":0,"asset_id":"1.3.0"},"payer":"1.2.53","sidechain_address_id":"1.38.64","sidechain_address_account":"1.2.53","sidechain":"bitcoin"}]],"extensions":[],"signatures":[]} available_keys: ["TEST8Qynrv4EKjgSVxnnERmNKojVsyJHLWXAPiukoXTXgjrhwuyyHe"] peerplays-all-in-one_1 | 721393ms th_a database_api.cpp:2543 get_required_signatu ] result: ["TEST8Qynrv4EKjgSVxnnERmNKojVsyJHLWXAPiukoXTXgjrhwuyyHe"] peerplays-all-in-one_1 | 722402ms th_a database_api.cpp:2555 get_potential_signat ] trx: {"ref_block_num":0,"ref_block_prefix":0,"expiration":"1970-01-01T00:00:00","operations":[[113,{"fee":{"amount":0,"asset_id":"1.3.0"},"payer":"1.2.53","sidechain_address_account":"1.2.53","sidechain":"bitcoin","deposit_public_key":"02b510a452d6e80f943e4cc85af5cad6c528bda87fc92b821dd246a1a76c175b0d","deposit_address":"","deposit_address_data":"","withdraw_public_key":"0212cbe2f328ba614886ca01e1ef48688196f72fbd07b31726f160292c96da20d5","withdraw_address":"2N75PEYXU6X556UU6VYNiHMgcCBL8XmWnc1"}]],"extensions":[],"signatures":[]} peerplays-all-in-one_1 | 722402ms th_a database_api.cpp:2583 get_potential_signat ] result: ["TEST8Qynrv4EKjgSVxnnERmNKojVsyJHLWXAPiukoXTXgjrhwuyyHe"] peerplays-all-in-one_1 | 722403ms th_a database_api.cpp:2529 get_required_signatu ] trx: {"ref_block_num":0,"ref_block_prefix":0,"expiration":"1970-01-01T00:00:00","operations":[[113,{"fee":{"amount":0,"asset_id":"1.3.0"},"payer":"1.2.53","sidechain_address_account":"1.2.53","sidechain":"bitcoin","deposit_public_key":"02b510a452d6e80f943e4cc85af5cad6c528bda87fc92b821dd246a1a76c175b0d","deposit_address":"","deposit_address_data":"","withdraw_public_key":"0212cbe2f328ba614886ca01e1ef48688196f72fbd07b31726f160292c96da20d5","withdraw_address":"2N75PEYXU6X556UU6VYNiHMgcCBL8XmWnc1"}]],"extensions":[],"signatures":[]} available_keys: ["TEST8Qynrv4EKjgSVxnnERmNKojVsyJHLWXAPiukoXTXgjrhwuyyHe"] peerplays-all-in-one_1 | 722403ms th_a database_api.cpp:2543 get_required_signatu ] result: ["TEST8Qynrv4EKjgSVxnnERmNKojVsyJHLWXAPiukoXTXgjrhwuyyHe"] peerplays-all-in-one_1 | 723000ms th_a witness.cpp:267 maybe_produce_block ] slot: 1 scheduled_witness: 1.6.8 scheduled_time: 2022-10-14T08:12:03 now: 2022-10-14T08:12:03 peerplays-all-in-one_1 | 723000ms th_a undo_database.cpp:39 ~session ] 10 assert_exception: Assert Exception peerplays-all-in-one_1 | ok: Could not modify object, most likely an index constraint was violated peerplays-all-in-one_1 | {} peerplays-all-in-one_1 | th_a generic_index.hpp:88 modify peerplays-all-in-one_1 | peerplays-all-in-one_1 | {} peerplays-all-in-one_1 | th_a undo_database.cpp:135 undo peerplays-all-in-one_1 | terminate called after throwing an instance of 'fc::assert_exception' ```

serkixenos commented

2022-10-14 18:33:22 +00:00

(Migrated from gitlab.com)

assigned to @serkixenos

hirunda commented

2022-10-14 21:55:23 +00:00

(Migrated from gitlab.com)

@wsalloum , we are still looking into the solution (@serkixenos and I ) but I made a hot fix which should resolve the issue. Please if you are able cherry-pick MR !163 , build the witness node and give a try with manual testing as you did before. You should not encounter the issue with crashing the witness node.

I also updated unit tests with delete and create sidechain address within one block, so please run:

./chain_test --run_test="sidechain_addresses_tests/sidechain_address_delete_create_test"

./chain_test --run_test="sidechain_addresses_tests"

./chain_test

@wsalloum , we are still looking into the solution (@serkixenos and I ) but I made a hot fix which should resolve the issue. Please if you are able cherry-pick MR !163 , build the witness node and give a try with manual testing as you did before. You should not encounter the issue with crashing the witness node. I also updated unit tests with delete and create sidechain address within one block, so please run: `./chain_test --run_test="sidechain_addresses_tests/sidechain_address_delete_create_test"` `./chain_test --run_test="sidechain_addresses_tests"` `./chain_test`

hirunda commented

2022-10-14 21:56:18 +00:00

(Migrated from gitlab.com)

assigned to @wsalloum

wsalloum commented

2022-10-16 08:11:28 +00:00

(Migrated from gitlab.com)

Hi @hirunda , from the QA environment? or I need to build it from https://gitlab.com/PBSA/peerplays ?

hirunda commented

2022-10-16 22:37:54 +00:00

(Migrated from gitlab.com)

Hi @wsalloum , I don't know in which environment you reproduced the issue. You should use the same.
If you used docker, just change the docker file as following:

`RUN \

cd src && \

git clone https://gitlab.com/PBSA/peerplays.git && \

cd peerplays && \

#git checkout develop && \

**git checkout bug/462/investigate_trx_which_caused_mainnet_halt** && \

git submodule update --init --recursive && \

git branch --show-current && \

git log --oneline -n 5`

And build docker again before you start the test. Please note that docker build sometimes is cashing, and here is important branch to be checkout and peerplays to be rebuilt.

If you haven't used docker you just have to checkout branch before you build witness node:

git checkout bug/462/investigate_trx_which_caused_mainnet_halt

make -j6 witness_node

Run the cli_wallet and do the same you did to reproduce the issue.

Hi @wsalloum , I don't know in which environment you reproduced the issue. You should use the same. If you used docker, just change the docker file as following: `RUN \ cd src && \ git clone https://gitlab.com/PBSA/peerplays.git && \ cd peerplays && \ #git checkout develop && \ **git checkout bug/462/investigate_trx_which_caused_mainnet_halt** && \ git submodule update --init --recursive && \ git branch --show-current && \ git log --oneline -n 5` And build docker again before you start the test. Please note that docker build sometimes is cashing, and here is important branch to be checkout and peerplays to be rebuilt. If you haven't used docker you just have to checkout branch before you build witness node: `git checkout bug/462/investigate_trx_which_caused_mainnet_halt` `make -j6 witness_node` Run the cli_wallet and do the same you did to reproduce the issue.

wsalloum commented

2022-10-17 09:26:33 +00:00

(Migrated from gitlab.com)

I see there is no exception now

Test instructions

Update the peerplays-qa-environment/peerplays/DockerFile to

# Clone Peerplays
RUN \
    
    cd src && \
    git clone https://gitlab.com/PBSA/peerplays.git && \
    cd peerplays && \
#git checkout develop && \
    git checkout bug/462/investigate_trx_which_caused_mainnet_halt && \
    git submodule update --init --recursive && \
    git branch --show-current && \
    git log --oneline -n 5

Rebuild docker docker-compose build peerplays-base
Run QA Environment
Run the following commands (super fast)
delete_sidechain_address account02 1 true
add_sidechain_address account02 bitcoin "02b510a452d6e80f943e4cc85af5cad6c528bda87fc92b821dd246a1a76c175b0d" "" "0212cbe2f328ba614886ca01e1ef48688196f72fbd07b31726f160292c96da20d5" "2N75PEYXU6X556UU6VYNiHMgcCBL8XmWnc1" true
Repeat the previous 2 commands very fast after each other



unlocked >>> add_sidechain_address account02 bitcoin "02b510a452d6e80f943e4cc85af5cad6c528bda87fc92b821dd246a1a76c175b0d" "" "0212cbe2f328ba614886ca01e1ef48688196f72fbd07b31726f160292c96da20d5" "2N75PEYXU6X556UU6VYNiHMgcCBL8XmWnc1" true
add_sidechain_address account02 bitcoin "02b510a452d6e80f943e4cc85af5cad6c528bda87fc92b821dd246a1a76c175b0d" "" "0212cbe2f328ba614886ca01e1ef48688196f72fbd07b31726f160292c96da20d5" "2N75PEYXU6X556UU6VYNiHMgcCBL8XmWnc1" true
{
  "ref_block_num": 53,
  "ref_block_prefix": 639434586,
  "expiration": "2022-10-17T08:25:36",
  "operations": [[
      113,{
        "fee": {
          "amount": 0,
          "asset_id": "1.3.0"
        },
        "payer": "1.2.53",
        "sidechain_address_account": "1.2.53",
        "sidechain": "bitcoin",
        "deposit_public_key": "02b510a452d6e80f943e4cc85af5cad6c528bda87fc92b821dd246a1a76c175b0d",
        "deposit_address": "",
        "deposit_address_data": "",
        "withdraw_public_key": "0212cbe2f328ba614886ca01e1ef48688196f72fbd07b31726f160292c96da20d5",
        "withdraw_address": "2N75PEYXU6X556UU6VYNiHMgcCBL8XmWnc1"
      }
    ]
  ],
  "extensions": [],
  "signatures": [
    "2051d9ce427fde4eefec651acd2b599aced2c1886a764e5f8ba19535b89c5ae1821dd34a0efe69130660aef73c320d4d2ac8886a34233066ab96641bb4aaccfe35"
  ]
}

Check the log

peerplays-all-in-one_1            | 1506000ms th_a       witness.cpp:191               block_production_loo ] Generated block #53 with timestamp 2022-10-17T08:25:06 at time 2022-10-17T08:25:06
peerplays-all-in-one_1            | 1506266ms th_a       database_api.cpp:2555         get_potential_signat ] trx: {"ref_block_num":0,"ref_block_prefix":0,"expiration":"1970-01-01T00:00:00","operations":[[115,{"fee":{"amount":0,"asset_id":"1.3.0"},"payer":"1.2.53","sidechain_address_id":"1.38.66","sidechain_address_account":"1.2.53","sidechain":"bitcoin"}]],"extensions":[],"signatures":[]} 
peerplays-all-in-one_1            | 1506266ms th_a       database_api.cpp:2583         get_potential_signat ] result: ["TEST8Qynrv4EKjgSVxnnERmNKojVsyJHLWXAPiukoXTXgjrhwuyyHe"] 
peerplays-all-in-one_1            | 1506267ms th_a       database_api.cpp:2529         get_required_signatu ] trx: {"ref_block_num":0,"ref_block_prefix":0,"expiration":"1970-01-01T00:00:00","operations":[[115,{"fee":{"amount":0,"asset_id":"1.3.0"},"payer":"1.2.53","sidechain_address_id":"1.38.66","sidechain_address_account":"1.2.53","sidechain":"bitcoin"}]],"extensions":[],"signatures":[]} available_keys: ["TEST8Qynrv4EKjgSVxnnERmNKojVsyJHLWXAPiukoXTXgjrhwuyyHe"] 
peerplays-all-in-one_1            | 1506267ms th_a       database_api.cpp:2543         get_required_signatu ] result: ["TEST8Qynrv4EKjgSVxnnERmNKojVsyJHLWXAPiukoXTXgjrhwuyyHe"] 
peerplays-all-in-one_1            | 1506805ms th_a       database_api.cpp:2555         get_potential_signat ] trx: {"ref_block_num":0,"ref_block_prefix":0,"expiration":"1970-01-01T00:00:00","operations":[[113,{"fee":{"amount":0,"asset_id":"1.3.0"},"payer":"1.2.53","sidechain_address_account":"1.2.53","sidechain":"bitcoin","deposit_public_key":"02b510a452d6e80f943e4cc85af5cad6c528bda87fc92b821dd246a1a76c175b0d","deposit_address":"","deposit_address_data":"","withdraw_public_key":"0212cbe2f328ba614886ca01e1ef48688196f72fbd07b31726f160292c96da20d5","withdraw_address":"2N75PEYXU6X556UU6VYNiHMgcCBL8XmWnc1"}]],"extensions":[],"signatures":[]} 
peerplays-all-in-one_1            | 1506805ms th_a       database_api.cpp:2583         get_potential_signat ] result: ["TEST8Qynrv4EKjgSVxnnERmNKojVsyJHLWXAPiukoXTXgjrhwuyyHe"] 
peerplays-all-in-one_1            | 1506806ms th_a       database_api.cpp:2529         get_required_signatu ] trx: {"ref_block_num":0,"ref_block_prefix":0,"expiration":"1970-01-01T00:00:00","operations":[[113,{"fee":{"amount":0,"asset_id":"1.3.0"},"payer":"1.2.53","sidechain_address_account":"1.2.53","sidechain":"bitcoin","deposit_public_key":"02b510a452d6e80f943e4cc85af5cad6c528bda87fc92b821dd246a1a76c175b0d","deposit_address":"","deposit_address_data":"","withdraw_public_key":"0212cbe2f328ba614886ca01e1ef48688196f72fbd07b31726f160292c96da20d5","withdraw_address":"2N75PEYXU6X556UU6VYNiHMgcCBL8XmWnc1"}]],"extensions":[],"signatures":[]} available_keys: ["TEST8Qynrv4EKjgSVxnnERmNKojVsyJHLWXAPiukoXTXgjrhwuyyHe"] 
peerplays-all-in-one_1            | 1506806ms th_a       database_api.cpp:2543         get_required_signatu ] result: ["TEST8Qynrv4EKjgSVxnnERmNKojVsyJHLWXAPiukoXTXgjrhwuyyHe"]

I see there is no exception now Test instructions - Update the peerplays-qa-environment/peerplays/DockerFile to ``` # Clone Peerplays RUN \ cd src && \ git clone https://gitlab.com/PBSA/peerplays.git && \ cd peerplays && \ #git checkout develop && \ git checkout bug/462/investigate_trx_which_caused_mainnet_halt && \ git submodule update --init --recursive && \ git branch --show-current && \ git log --oneline -n 5 ``` - Rebuild docker `docker-compose build peerplays-base` - Run QA Environment - Run the following commands (super fast) - `delete_sidechain_address account02 1 true` - `add_sidechain_address account02 bitcoin "02b510a452d6e80f943e4cc85af5cad6c528bda87fc92b821dd246a1a76c175b0d" "" "0212cbe2f328ba614886ca01e1ef48688196f72fbd07b31726f160292c96da20d5" "2N75PEYXU6X556UU6VYNiHMgcCBL8XmWnc1" true` - Repeat the previous 2 commands very fast after each other ``` unlocked >>> add_sidechain_address account02 bitcoin "02b510a452d6e80f943e4cc85af5cad6c528bda87fc92b821dd246a1a76c175b0d" "" "0212cbe2f328ba614886ca01e1ef48688196f72fbd07b31726f160292c96da20d5" "2N75PEYXU6X556UU6VYNiHMgcCBL8XmWnc1" true add_sidechain_address account02 bitcoin "02b510a452d6e80f943e4cc85af5cad6c528bda87fc92b821dd246a1a76c175b0d" "" "0212cbe2f328ba614886ca01e1ef48688196f72fbd07b31726f160292c96da20d5" "2N75PEYXU6X556UU6VYNiHMgcCBL8XmWnc1" true { "ref_block_num": 53, "ref_block_prefix": 639434586, "expiration": "2022-10-17T08:25:36", "operations": [[ 113,{ "fee": { "amount": 0, "asset_id": "1.3.0" }, "payer": "1.2.53", "sidechain_address_account": "1.2.53", "sidechain": "bitcoin", "deposit_public_key": "02b510a452d6e80f943e4cc85af5cad6c528bda87fc92b821dd246a1a76c175b0d", "deposit_address": "", "deposit_address_data": "", "withdraw_public_key": "0212cbe2f328ba614886ca01e1ef48688196f72fbd07b31726f160292c96da20d5", "withdraw_address": "2N75PEYXU6X556UU6VYNiHMgcCBL8XmWnc1" } ] ], "extensions": [], "signatures": [ "2051d9ce427fde4eefec651acd2b599aced2c1886a764e5f8ba19535b89c5ae1821dd34a0efe69130660aef73c320d4d2ac8886a34233066ab96641bb4aaccfe35" ] } ``` - Check the log ``` peerplays-all-in-one_1 | 1506000ms th_a witness.cpp:191 block_production_loo ] Generated block #53 with timestamp 2022-10-17T08:25:06 at time 2022-10-17T08:25:06 peerplays-all-in-one_1 | 1506266ms th_a database_api.cpp:2555 get_potential_signat ] trx: {"ref_block_num":0,"ref_block_prefix":0,"expiration":"1970-01-01T00:00:00","operations":[[115,{"fee":{"amount":0,"asset_id":"1.3.0"},"payer":"1.2.53","sidechain_address_id":"1.38.66","sidechain_address_account":"1.2.53","sidechain":"bitcoin"}]],"extensions":[],"signatures":[]} peerplays-all-in-one_1 | 1506266ms th_a database_api.cpp:2583 get_potential_signat ] result: ["TEST8Qynrv4EKjgSVxnnERmNKojVsyJHLWXAPiukoXTXgjrhwuyyHe"] peerplays-all-in-one_1 | 1506267ms th_a database_api.cpp:2529 get_required_signatu ] trx: {"ref_block_num":0,"ref_block_prefix":0,"expiration":"1970-01-01T00:00:00","operations":[[115,{"fee":{"amount":0,"asset_id":"1.3.0"},"payer":"1.2.53","sidechain_address_id":"1.38.66","sidechain_address_account":"1.2.53","sidechain":"bitcoin"}]],"extensions":[],"signatures":[]} available_keys: ["TEST8Qynrv4EKjgSVxnnERmNKojVsyJHLWXAPiukoXTXgjrhwuyyHe"] peerplays-all-in-one_1 | 1506267ms th_a database_api.cpp:2543 get_required_signatu ] result: ["TEST8Qynrv4EKjgSVxnnERmNKojVsyJHLWXAPiukoXTXgjrhwuyyHe"] peerplays-all-in-one_1 | 1506805ms th_a database_api.cpp:2555 get_potential_signat ] trx: {"ref_block_num":0,"ref_block_prefix":0,"expiration":"1970-01-01T00:00:00","operations":[[113,{"fee":{"amount":0,"asset_id":"1.3.0"},"payer":"1.2.53","sidechain_address_account":"1.2.53","sidechain":"bitcoin","deposit_public_key":"02b510a452d6e80f943e4cc85af5cad6c528bda87fc92b821dd246a1a76c175b0d","deposit_address":"","deposit_address_data":"","withdraw_public_key":"0212cbe2f328ba614886ca01e1ef48688196f72fbd07b31726f160292c96da20d5","withdraw_address":"2N75PEYXU6X556UU6VYNiHMgcCBL8XmWnc1"}]],"extensions":[],"signatures":[]} peerplays-all-in-one_1 | 1506805ms th_a database_api.cpp:2583 get_potential_signat ] result: ["TEST8Qynrv4EKjgSVxnnERmNKojVsyJHLWXAPiukoXTXgjrhwuyyHe"] peerplays-all-in-one_1 | 1506806ms th_a database_api.cpp:2529 get_required_signatu ] trx: {"ref_block_num":0,"ref_block_prefix":0,"expiration":"1970-01-01T00:00:00","operations":[[113,{"fee":{"amount":0,"asset_id":"1.3.0"},"payer":"1.2.53","sidechain_address_account":"1.2.53","sidechain":"bitcoin","deposit_public_key":"02b510a452d6e80f943e4cc85af5cad6c528bda87fc92b821dd246a1a76c175b0d","deposit_address":"","deposit_address_data":"","withdraw_public_key":"0212cbe2f328ba614886ca01e1ef48688196f72fbd07b31726f160292c96da20d5","withdraw_address":"2N75PEYXU6X556UU6VYNiHMgcCBL8XmWnc1"}]],"extensions":[],"signatures":[]} available_keys: ["TEST8Qynrv4EKjgSVxnnERmNKojVsyJHLWXAPiukoXTXgjrhwuyyHe"] peerplays-all-in-one_1 | 1506806ms th_a database_api.cpp:2543 get_required_signatu ] result: ["TEST8Qynrv4EKjgSVxnnERmNKojVsyJHLWXAPiukoXTXgjrhwuyyHe"] ```

bobinson commented

2022-10-28 13:56:55 +00:00

(Migrated from gitlab.com)

mentioned in issue #468