automation: Report SON down #252

New issue

Closed

opened 2022-01-17 11:12:56 +00:00 by pavel.baykov · 7 comments

pavel.baykov commented

2022-01-17 11:12:56 +00:00

(Migrated from gitlab.com)

Consider an active SON configured on some node say node x.
Verify the current SON status by using below command

get_son <<son_account>>
From the result of previous step, fetch statistics object id from statistics property.

Get SON statistics using below command:
get_object <<statistics_object_id>>
STOP the node running SON using below command
sudo systemctl stop witness.service
Wait for next 2 heartbeats to happen
(currently 1 heartbeat = 3 mins) and then check the status of SON
get_son <<son_account>>
Verify fee for son_report_down_operation is deducted from son-account and son_report_down proposal fee is deducted from the SON which is reporting above son is down using below command.

get_account_history son-account 10 10

Consider an active SON configured on some node say node x. Verify the current SON status by using below command get_son <<son_account>> From the result of previous step, fetch statistics object id from statistics property. Get SON statistics using below command: get_object <<statistics_object_id>> STOP the node running SON using below command sudo systemctl stop witness.service Wait for next 2 heartbeats to happen (currently 1 heartbeat = 3 mins) and then check the status of SON get_son <<son_account>> Verify fee for son_report_down_operation is deducted from son-account and son_report_down proposal fee is deducted from the SON which is reporting above son is down using below command. get_account_history son-account 10 10

pavel.baykov commented

2022-01-17 11:12:56 +00:00

(Migrated from gitlab.com)

assigned to @pavel.baykov

pavel.baykov commented

2022-01-21 09:23:32 +00:00

(Migrated from gitlab.com)

to test:
pytest -s -v test.py::test_Report_SON_Down

to test: pytest -s -v test.py::test_Report_SON_Down

pavel.baykov commented

2022-01-21 09:32:19 +00:00

(Migrated from gitlab.com)

#TODO parse account_history

serkixenos commented

2022-01-31 03:22:29 +00:00

(Migrated from gitlab.com)

Test fails

test.py::test_Report_SON_Down
Wait son sends next heartbeat...
Wait son sends next heartbeat...
Wait son sends next heartbeat...
Wait son sends next heartbeat...
Wait son sends next heartbeat...
Wait son sends next heartbeat...
Wait son sends next heartbeat...
Wait son sends next heartbeat...
Wait son sends next heartbeat...
Wait son sends next heartbeat...
Wait son sends next heartbeat...
Wait son sends next heartbeat...
Wait son sends next heartbeat...
Wait son sends next heartbeat...
Wait son sends next heartbeat...
Wait son sends next heartbeat...
Wait son sends next heartbeat...
Wait son sends next heartbeat...
Wait son sends next heartbeat...
Wait son sends next heartbeat...
Wait son sends next heartbeat...
Wait son sends next heartbeat...
Wait son sends next heartbeat...
Stopping docker: abd011ec1b09
son_down_time = 360
FAILED

Test fails ``` test.py::test_Report_SON_Down Wait son sends next heartbeat... Wait son sends next heartbeat... Wait son sends next heartbeat... Wait son sends next heartbeat... Wait son sends next heartbeat... Wait son sends next heartbeat... Wait son sends next heartbeat... Wait son sends next heartbeat... Wait son sends next heartbeat... Wait son sends next heartbeat... Wait son sends next heartbeat... Wait son sends next heartbeat... Wait son sends next heartbeat... Wait son sends next heartbeat... Wait son sends next heartbeat... Wait son sends next heartbeat... Wait son sends next heartbeat... Wait son sends next heartbeat... Wait son sends next heartbeat... Wait son sends next heartbeat... Wait son sends next heartbeat... Wait son sends next heartbeat... Wait son sends next heartbeat... Stopping docker: abd011ec1b09 son_down_time = 360 FAILED ```

pavel.baykov commented

2022-02-01 13:28:16 +00:00

(Migrated from gitlab.com)

i executed this test +- 10 times, and sometimes i have passing
i think test itself is good , but such tests have issues with Docker to accounts mapping or there is a problem in the code itself

i executed this test +- 10 times, and sometimes i have passing i think test itself is good , but such tests have issues with Docker to accounts mapping or there is a problem in the code itself

serkixenos commented

2022-03-11 00:28:56 +00:00

(Migrated from gitlab.com)

@pavel.baykov

pytest -s -v --order-dependencies --rundockers=yes --initdockers=yes --shutdowndockers=no ./end_to_end_tests/test_Report_SON_Down.py

Result:

================================================================================================ FAILURES =================================================================================================
__________________________________________________________________________________________ test_Report_SON_Down ___________________________________________________________________________________________

setup_test_env = None, skipall = 'no'

    def test_Report_SON_Down(setup_test_env, skipall):
        if "yes" == skipall:
            pytest.skip("Skip all tests")
    
        SON_DOWN_ACCOUNT = 'sonaccount05'
        son_id = endtoendtests.son_owner_account_name_to_son_id[SON_DOWN_ACCOUNT]
        print(son_id)
    
        account = SON_DOWN_ACCOUNT
        docker_name = endtoendtests.son_id_to_docker_name[son_id]
        print(docker_name)
    
        #1. Consider an active SON configured on some node say node x
        #2. Verify the current SON status
        assert 'active' == get_son_status(endtoendtests.cli_wallet, account)
    
        #3. From the result of previous step, fetch statistics object id from statistics property.
        #   Verify last_active_timestamp value is within last 3 minutes (1 heartbeat) from current time
    
        statistics = get_son_statistics(endtoendtests.cli_wallet, account)
        timestamp = get_object_timestamp(endtoendtests.cli_wallet, statistics)
        timeout = 0
        heartbeat_timeout = get_son_heartbeat_frequency(endtoendtests.cli_wallet)
        #Make sure timestamp changed during heartbeat_timeout interval
        while timestamp == get_object_timestamp(endtoendtests.cli_wallet, statistics):
            print('Wait son sends next heartbeat...')
            assert timeout <= heartbeat_timeout
    
            time.sleep(5)
            timeout += 5
    
        #4. STOP the node running SON
        #Stop the node running SON
        docker_id = suspend_docker(docker_name)
    
        #5. Wait for next 2 heartbeats to happen status property should contain the value - "in_maintenance"
        time.sleep(4 * heartbeat_timeout)
    
        #Check the status of SON
>       assert 'in_maintenance' == get_son_status(endtoendtests.cli_wallet, account)
E       AssertionError: assert 'in_maintenance' == 'active'
E         - active
E         + in_maintenance

end_to_end_tests/test_Report_SON_Down.py:49: AssertionError
========================================================================================= short test summary info =========================================================================================
FAILED end_to_end_tests/test_Report_SON_Down.py::test_Report_SON_Down - AssertionError: assert 'in_maintenance' == 'active'
===================================================================================== 1 failed in 1825.98s (0:30:25) ======================================================================================

@pavel.baykov ``` pytest -s -v --order-dependencies --rundockers=yes --initdockers=yes --shutdowndockers=no ./end_to_end_tests/test_Report_SON_Down.py ``` Result: ``` ================================================================================================ FAILURES ================================================================================================= __________________________________________________________________________________________ test_Report_SON_Down ___________________________________________________________________________________________ setup_test_env = None, skipall = 'no' def test_Report_SON_Down(setup_test_env, skipall): if "yes" == skipall: pytest.skip("Skip all tests") SON_DOWN_ACCOUNT = 'sonaccount05' son_id = endtoendtests.son_owner_account_name_to_son_id[SON_DOWN_ACCOUNT] print(son_id) account = SON_DOWN_ACCOUNT docker_name = endtoendtests.son_id_to_docker_name[son_id] print(docker_name) #1. Consider an active SON configured on some node say node x #2. Verify the current SON status assert 'active' == get_son_status(endtoendtests.cli_wallet, account) #3. From the result of previous step, fetch statistics object id from statistics property. # Verify last_active_timestamp value is within last 3 minutes (1 heartbeat) from current time statistics = get_son_statistics(endtoendtests.cli_wallet, account) timestamp = get_object_timestamp(endtoendtests.cli_wallet, statistics) timeout = 0 heartbeat_timeout = get_son_heartbeat_frequency(endtoendtests.cli_wallet) #Make sure timestamp changed during heartbeat_timeout interval while timestamp == get_object_timestamp(endtoendtests.cli_wallet, statistics): print('Wait son sends next heartbeat...') assert timeout <= heartbeat_timeout time.sleep(5) timeout += 5 #4. STOP the node running SON #Stop the node running SON docker_id = suspend_docker(docker_name) #5. Wait for next 2 heartbeats to happen status property should contain the value - "in_maintenance" time.sleep(4 * heartbeat_timeout) #Check the status of SON > assert 'in_maintenance' == get_son_status(endtoendtests.cli_wallet, account) E AssertionError: assert 'in_maintenance' == 'active' E - active E + in_maintenance end_to_end_tests/test_Report_SON_Down.py:49: AssertionError ========================================================================================= short test summary info ========================================================================================= FAILED end_to_end_tests/test_Report_SON_Down.py::test_Report_SON_Down - AssertionError: assert 'in_maintenance' == 'active' ===================================================================================== 1 failed in 1825.98s (0:30:25) ====================================================================================== ```

pavel.baykov commented

2022-03-11 18:12:07 +00:00

(Migrated from gitlab.com)

it’s not passing all the time,
i think it’s not a test problem, it’s a problem with the c++,
there is no issues with the mapping

it’s not passing all the time, i think it’s not a test problem, it’s a problem with the c++, there is no issues with the mapping

serkixenos (Migrated from gitlab.com) closed this issue

2022-03-21 15:22:48 +00:00