Peerplays_Blockchain/peerplays_migrated

Fork 0

cli_wallet SON related tests randomly fail #279

New issue

Closed

opened 2022-02-15 18:37:50 +00:00 by serkixenos · 14 comments

serkixenos commented

2022-02-15 18:37:50 +00:00

(Migrated from gitlab.com)

During CI/CD execution, it may happen that some SON related tests randomly fail. eg, out of 20 executiopns, it will fail once. If restarted, CI/CD might complete successfully

    {"name":"sonaccount2"}
    th_a  wallet.cpp:1453 upgrade_account 
[1;36;49mCleanup cli_wallet::boost_fixture_test_case[0;39;49m
3241507ms th_a       database_api.cpp:405          ~database_api_impl   ] freeing database api 94431919801648
3241508ms th_a       database_api.cpp:405          ~database_api_impl   ] freeing database api 94431920445952
3241508ms th_a       database_api.cpp:405          ~database_api_impl   ] freeing database api 94431920060016
3241606ms p2p        thread.cpp:246                exec                 ] thread canceled: 9 canceled_exception: Canceled

    {}
    p2p  thread_d.hpp:477 start_next_fiber
unknown location(0): [4;31;49mfatal error: in "son_cli/select_top_fifteen_sons": unknown type[0;39;49m
/var/lib/gitlab-runner/builds/PRNhrcsP/0/PBSA/peerplays/tests/cli/son.cpp(64): [1;36;49mlast checkpoint[0;39;49m
3242135ms th_a       application.cpp:79            create_example_genes ] Allocating all stake to 5KQwrPbwdL6PhXujxW37FSSQZ1JiwsST4cqQzDeyXtP79zkvFD3
3242138ms th_a       bookie_plugin.cpp:468         plugin_initialize    ] bookie plugin: plugin_startup() begin
3242138ms th_a       db_management.cpp:294         force_slow_replays   ] enabling slow replays
3242138ms th_a       bookie_plugin.cpp:501         plugin_initialize    ] bookie plugin: plugin_startup() end
3242138ms th_a       witness.cpp:91                plugin_initialize    ] witness plugin:  plugin_initialize() begin

Eg, CI/CD fails:
https://gitlab.com/PBSA/peerplays/-/jobs/2096014739#L1156

Same code, executed again:
https://gitlab.com/PBSA/peerplays/-/jobs/2096508914

The error shows up regardless of build configurations, like build type and testnet vs mainnet

cmake -DCMAKE_BUILD_TYPE=Debug
cmake -DCMAKE_BUILD_TYPE=RelWithDebugInfo
cmake -DCMAKE_BUILD_TYPE=Release

This should mean that we are not looking into Debug vs Release build related problem.

During CI/CD execution, it may happen that some SON related tests randomly fail. eg, out of 20 executiopns, it will fail once. If restarted, CI/CD might complete successfully ``` {"name":"sonaccount2"} th_a wallet.cpp:1453 upgrade_account [1;36;49mCleanup cli_wallet::boost_fixture_test_case[0;39;49m 3241507ms th_a database_api.cpp:405 ~database_api_impl ] freeing database api 94431919801648 3241508ms th_a database_api.cpp:405 ~database_api_impl ] freeing database api 94431920445952 3241508ms th_a database_api.cpp:405 ~database_api_impl ] freeing database api 94431920060016 3241606ms p2p thread.cpp:246 exec ] thread canceled: 9 canceled_exception: Canceled {} p2p thread_d.hpp:477 start_next_fiber unknown location(0): [4;31;49mfatal error: in "son_cli/select_top_fifteen_sons": unknown type[0;39;49m /var/lib/gitlab-runner/builds/PRNhrcsP/0/PBSA/peerplays/tests/cli/son.cpp(64): [1;36;49mlast checkpoint[0;39;49m 3242135ms th_a application.cpp:79 create_example_genes ] Allocating all stake to 5KQwrPbwdL6PhXujxW37FSSQZ1JiwsST4cqQzDeyXtP79zkvFD3 3242138ms th_a bookie_plugin.cpp:468 plugin_initialize ] bookie plugin: plugin_startup() begin 3242138ms th_a db_management.cpp:294 force_slow_replays ] enabling slow replays 3242138ms th_a bookie_plugin.cpp:501 plugin_initialize ] bookie plugin: plugin_startup() end 3242138ms th_a witness.cpp:91 plugin_initialize ] witness plugin: plugin_initialize() begin ``` Eg, CI/CD fails: https://gitlab.com/PBSA/peerplays/-/jobs/2096014739#L1156 Same code, executed again: https://gitlab.com/PBSA/peerplays/-/jobs/2096508914 The error shows up regardless of build configurations, like build type and testnet vs mainnet ``` cmake -DCMAKE_BUILD_TYPE=Debug cmake -DCMAKE_BUILD_TYPE=RelWithDebugInfo cmake -DCMAKE_BUILD_TYPE=Release ``` This should mean that we are not looking into Debug vs Release build related problem.

serkixenos commented

2022-02-23 00:42:57 +00:00

(Migrated from gitlab.com)

mentioned in issue #229

serkixenos commented

2022-02-28 23:10:38 +00:00

(Migrated from gitlab.com)

mentioned in merge request !60

serkixenos commented

2022-03-01 04:42:32 +00:00

(Migrated from gitlab.com)

This error error: in "son_cli/select_top_fifteen_sons": unknown type usually means that we are working with invalid object id, eg that we expected that some object is created, but it is not.

This error ```error: in "son_cli/select_top_fifteen_sons": unknown type``` usually means that we are working with invalid object id, eg that we expected that some object is created, but it is not.

vampik commented

2022-03-01 20:43:30 +00:00

(Migrated from gitlab.com)

assigned to @vampik

vampik commented

2022-03-04 17:18:19 +00:00

(Migrated from gitlab.com)

mentioned in commit e28853ebd4

mentioned in commit e28853ebd4aa4094d22347f57617c44fc8e28f90

vampik commented

2022-03-04 21:46:32 +00:00

(Migrated from gitlab.com)

mentioned in merge request !77

vampik commented

2022-03-04 21:47:06 +00:00

(Migrated from gitlab.com)

Add additional generate_block, so we don't have randomly test fail

vampik commented

2022-03-04 21:49:53 +00:00

(Migrated from gitlab.com)

The core problems is:

In function sign_transaction

      set<public_key_type> pks = _remote_db->get_potential_signatures(tx);
      flat_set<public_key_type> owned_keys;
      owned_keys.reserve(pks.size());
      std::copy_if(pks.begin(), pks.end(), std::inserter(owned_keys, owned_keys.end()),
                   [this](const public_key_type &pk) { return _keys.find(pk) != _keys.end(); });
      tx.clear_signatures();
      set<public_key_type> approving_key_set = _remote_db->get_required_signatures(tx, owned_keys);

The pks key for signature can't be found in _keys map. So we execute get_required_signatures with empty owned_keys

The core problems is: In function `sign_transaction` ``` set<public_key_type> pks = _remote_db->get_potential_signatures(tx); flat_set<public_key_type> owned_keys; owned_keys.reserve(pks.size()); std::copy_if(pks.begin(), pks.end(), std::inserter(owned_keys, owned_keys.end()), [this](const public_key_type &pk) { return _keys.find(pk) != _keys.end(); }); tx.clear_signatures(); set<public_key_type> approving_key_set = _remote_db->get_required_signatures(tx, owned_keys); ``` The `pks` key for signature can't be found in `_keys` map. So we execute `get_required_signatures` with empty `owned_keys`

vampik commented

2022-03-04 21:50:18 +00:00

(Migrated from gitlab.com)

added 10h of time spent

serkixenos commented

2022-03-04 22:02:21 +00:00

(Migrated from gitlab.com)

assigned to @prandnum and unassigned @vampik

vampik commented

2022-03-04 22:02:42 +00:00

(Migrated from gitlab.com)

mentioned in commit ae5237a781