Testnet chain halts

rilesdun commented

2021-09-02 18:51:49 +00:00

(Migrated from gitlab.com)

Yesterday we experienced issues on the Irona testnet, in which decided to restart the chain. After this I was still experencing issues so went ahead to make the same tests on fred.

When checking the status of witness.service I was given the following output

Still I decided to log into the cli_wallet and make my test, but the cli_wallet is unable to run as you can see in the following output:

Resetting fred would likely solve this issue for sometime, but we must determine what the actual cause of this halt is.

Yesterday we experienced issues on the Irona testnet, in which decided to restart the chain. After this I was still experencing issues so went ahead to make the same tests on fred. When checking the status of `witness.service` I was given the following output ![image](/uploads/4374be681f7a33354feb690a633c31d7/image.png) Still I decided to log into the cli_wallet and make my test, but the cli_wallet is unable to run as you can see in the following output:![image](/uploads/7fec77fd33c8c68b45e149d24585c48b/image.png) Resetting fred would likely solve this issue for sometime, but we must determine what the actual cause of this halt is.

hbelakon commented

2021-09-02 18:57:20 +00:00

(Migrated from gitlab.com)

assigned to @sierra19XX

hbelakon commented

2021-09-02 18:59:09 +00:00

(Migrated from gitlab.com)

assigned to @serkixenos

hbelakon commented

2021-09-02 18:59:34 +00:00

(Migrated from gitlab.com)

@serkixenos can you please investigate this issue?

serkixenos commented

2021-09-08 00:53:45 +00:00

(Migrated from gitlab.com)

assigned to @rilesdun and unassigned @sierra19XX and @serkixenos

serkixenos commented

2021-09-08 00:54:43 +00:00

(Migrated from gitlab.com)

As far as I know, both Irona and Fred runs on Proxmox servers. Due to known issues with this environment, we would like to see problem replicated in another, more stable environment, like AWS, or physical servers. We are unable to replicate this problem on docker based test environments (we have 2 of them at the moment).

rilesdun commented

2021-09-08 11:28:57 +00:00

(Migrated from gitlab.com)

@serkixenos both Irona and Fred have always been running on AWS, but they are not running on with Docker. I'm thinking that more testing should be done without using Docker as this is how the environment that crashed was running

serkixenos commented

2021-09-08 15:06:15 +00:00

(Migrated from gitlab.com)

OK. Since the error is coming from elasti csearch plugin, can you let me know what elastic search operations are you using, and share versions and config files of elastic search environment (for both elastic search and kibana).

bobinson commented

2021-09-09 11:51:49 +00:00

(Migrated from gitlab.com)

The configuration file (without the keys)
logs in text format
Elastic Search details

This will help @serkixenos or other developers.

1. The configuration file (without the keys) 2. logs in text format 3. Elastic Search details This will help @serkixenos or other developers.

serkixenos commented

2021-09-09 12:53:49 +00:00

(Migrated from gitlab.com)

According to last info from daily meeting, elastic search features are not used in Irona and Fred.

So, there is a chance that witnesses are misconfigured, and elastic search plugin is enabled, but there is no infrastructure behind it to support it.

@rilesdun Please verify plugins value from witness config file, and if it contains elasticsearch and es_objects, remove them, and restart the nodes.

Plugins enabled by default should be:

witness account_history market_history accounts_list affiliate_stats bookie

If running SONs, plugins enabled by default should be:

witness account_history market_history accounts_list affiliate_stats bookie peerplays_sidechain

According to last info from daily meeting, elastic search features are not used in Irona and Fred. So, there is a chance that witnesses are misconfigured, and elastic search plugin is enabled, but there is no infrastructure behind it to support it. @rilesdun Please verify plugins value from witness config file, and if it contains elasticsearch and es_objects, remove them, and restart the nodes. Plugins enabled by default should be: ``` witness account_history market_history accounts_list affiliate_stats bookie ``` If running SONs, plugins enabled by default should be: ``` witness account_history market_history accounts_list affiliate_stats bookie peerplays_sidechain ```

rilesdun commented

2021-09-09 13:47:36 +00:00

(Migrated from gitlab.com)

Good day @serkixenos, it seems configuration is correct with SONs active. You can see the following witness plugins here:

Good day @serkixenos, it seems configuration is correct with SONs active. You can see the following witness plugins here: ![image](/uploads/150b4d6e141f0822ec857fe760a11598/image.png)

serkixenos commented

2021-09-09 14:35:28 +00:00

(Migrated from gitlab.com)

Check the command line used for starting the node, because es_object plugin is active. It can be activated from config file, or with command line params.

rilesdun commented

2021-09-10 13:47:44 +00:00

(Migrated from gitlab.com)

@serkixenos Apologies for the misinformation in the above screenshot; as that is the current config for Irona. Fred actually only has the following plugins enabled as you predicted; you can see Freds config in the screenshot below.

Please confirm that the next following steps should be restarting fred with an updated config containing the following plugins: witness account_history market_history accounts_list affiliate_stats bookie

@serkixenos Apologies for the misinformation in the above screenshot; as that is the current config for Irona. Fred actually only has the following plugins enabled as you predicted; you can see Freds config in the screenshot below. ![image](/uploads/c3dd297d3f1e03b094770c822eb1acf9/image.png) Please confirm that the next following steps should be restarting fred with an updated config containing the following plugins: `witness account_history market_history accounts_list affiliate_stats bookie`

rilesdun commented

2021-09-10 16:26:55 +00:00

(Migrated from gitlab.com)

I have restarted Fred with the following parameters: witness account_history market_history accounts_list affiliate_stats bookie

Now I'll monitor this testnet carefully to ensure our problem is solved. For now, I will close this issue and reopen if we experience similar testnet halting in the future.

I have restarted Fred with the following parameters: `witness account_history market_history accounts_list affiliate_stats bookie` Now I'll monitor this testnet carefully to ensure our problem is solved. For now, I will close this issue and reopen if we experience similar testnet halting in the future.

rilesdun (Migrated from gitlab.com) closed this issue

2021-09-10 16:26:55 +00:00

yviter commented

2021-10-18 11:45:11 +00:00

(Migrated from gitlab.com)

mentioned in commit 39263ba774

mentioned in commit 39263ba7745b074d389ba83191f7c56dd56bcbac

serkixenos commented

2021-11-22 19:38:12 +00:00

(Migrated from gitlab.com)

mentioned in commit 890079c33f

mentioned in commit 890079c33f605f6408d3c8cdaf433bfff56cb005