Verify Elastic Search plugin on 1.5.22-alpha

bobinson commented

2023-04-05 14:00:49 +00:00

(Migrated from gitlab.com)

Available documentation:

https://peerplays.atlassian.net/wiki/spaces/PROJECTS/pages/1159299078/Quick+setup+of+Elasticsearch+on+AWS+and+connecting+a+witness+node+to+it

https://peerplays.atlassian.net/wiki/spaces/PIX/pages/364609555/ElasticSearch+Plugin+for+Peerplays

https://peerplays.atlassian.net/wiki/spaces/PROJECTS/pages/1367113733/Quick+setup+of+Elasticsearch+plugin+on+local+computer

cc @tai.sama : We may need this plugin for the explorer's REST API.

Available documentation: https://peerplays.atlassian.net/wiki/spaces/PROJECTS/pages/1159299078/Quick+setup+of+Elasticsearch+on+AWS+and+connecting+a+witness+node+to+it https://peerplays.atlassian.net/wiki/spaces/PIX/pages/364609555/ElasticSearch+Plugin+for+Peerplays https://peerplays.atlassian.net/wiki/spaces/PROJECTS/pages/1367113733/Quick+setup+of+Elasticsearch+plugin+on+local+computer cc @tai.sama : We may need this plugin for the explorer's REST API.

bobinson commented

2023-04-05 14:00:49 +00:00

(Migrated from gitlab.com)

assigned to @christophersanborn

christophersanborn commented

2023-04-12 16:00:12 +00:00

(Migrated from gitlab.com)

changed due date to April 19, 2023

tai.sama commented

2023-04-17 12:36:19 +00:00

(Migrated from gitlab.com)

@christophersanborn can i have an update for this

christophersanborn commented

2023-04-19 13:05:44 +00:00

(Migrated from gitlab.com)

changed due date to April 21, 2023

christophersanborn commented

2023-04-19 13:12:45 +00:00

(Migrated from gitlab.com)

Updating due date to Friday.

Test strategy:
- Sync a 1.5.22-alpha node to mainnet,
- Enable and configure ES plugin, replay to generate ES DB.
- Inspect and confirm general consistency between ES db and chain data.
Progress thus far:
- A 1.5.22-alpha node with post-dated HF dates is now synced to mainnet.
  - Difficulties encountered:
    - Could not sync pre-built (gitlab artifact) 1.5.22-alpha node to mainnet because HF date is now in the past. Mainnet blocks beyond this date did not follow new consensus rules. Necessitated compiling a version with adjusted HF dates. Necessitated setting up a suitable build environment.

@tai.sama @bobinson

Updating due date to Friday. - Test strategy: - Sync a 1.5.22-alpha node to mainnet, - Enable and configure ES plugin, replay to generate ES DB. - Inspect and confirm general consistency between ES db and chain data. - Progress thus far: - A 1.5.22-alpha node with post-dated HF dates is now synced to mainnet. - Difficulties encountered: - Could not sync pre-built (gitlab artifact) 1.5.22-alpha node to mainnet because HF date is now in the past. Mainnet blocks beyond this date did not follow new consensus rules. Necessitated compiling a version with adjusted HF dates. Necessitated setting up a suitable build environment. @tai.sama @bobinson

christophersanborn commented

2023-04-19 16:08:42 +00:00

(Migrated from gitlab.com)

Experiment 01 — Report:

Test setup:

A 1.5.22-alpha node with modified (future-dated) HF dates synced with mainnet, running inside an Ubuntu 20.04 docker container.
An ElasticSearch 6.4.3 node running in a separate docker container on the same machine, mapping port 9200.
- Note: Latest version of ElasticSearch (8.7.0) rapidly consumed >50% of system RAM even before loading any data, so I selected 6.4.3 instead, as it is the closest version available on DockerHub to the last known working ES version for the ES plugin on BitShares (v. 6.2.0)
Peerplays node configured with ONLY the elasticsearch plugin loaded.

Test outcome: ❌ Failure

The peerplays node errors out with error message: ES database is not up in url http://<redacted>:9200

root@a6333dae7181:/srv/Node/peerplays-main# ./bin/witness_node --replay-blockchain
2728818ms th_a       bookie_plugin.cpp:412         plugin_initialize    ] bookie plugin: plugin_startup() begin
2728818ms th_a       db_management.cpp:295         force_slow_replays   ] enabling slow replays
2728818ms th_a       bookie_plugin.cpp:435         plugin_initialize    ] bookie plugin: plugin_startup() end
2728818ms th_a       elasticsearch_plugin.cpp:698  plugin_initialize    ] elasticsearch ACCOUNT HISTORY: plugin_initialize() begin
2728818ms th_a       main.cpp:190                  main                 ] Exiting with error:
0 exception: unspecified
ES database is not up in url http://<redacted>:9200
    {"url":"http://<redacted>:9200"}
    th_a  elasticsearch_plugin.cpp:723 plugin_initialize
root@a6333dae7181:/srv/Node/peerplays-main#

This is despite the ES node in fact being up and running and reachable from the container:

root@a6333dae7181:/srv/Node/peerplays-main# curl -X GET 'http://<redacted>:9200/peerplays-*/data/_count?pretty=true' -H 'Content-Type: application/json' -d '{"query" : {"bool" : { "must" : [{"match_all": {}}] }}}'
{
  "count" : 0,
  "_shards" : {
    "total" : 0,
    "successful" : 0,
    "skipped" : 0,
    "failed" : 0
  }
}
root@a6333dae7181:/srv/Node/peerplays-main#

Investigation:

Further investigation (e.g. reviewing the plugin code that opens the ES connection) is underway.

Observations:

Replicating the experiment with 1.5.19 does NOT change the outcome. Likely the problem has long existed and was not introduced in 1.5.22-alpha or any of the recent alpha releases.
After inspecting code, noticed: Plugin code constructs the health check URL as: curl_request.url = es.elasticsearch_url + "_nodes". This fails to insert a / between server and file endpoint. Thus, ensuring value of elasticsearch-node-url in config.ini has a trailing slash should fix the error.
With the trailing slash added to the url, replay continues until the fist 10000 lines are queued for submission to the ES node. However, at this point 1.5.22-alpha exits with error: Error sending 10000 lines of bulk data to Elastic Search, the first lines are: ... Failed to push new block: 3100000 plugin_exception: plugin exception, Error populating ES database, we are going to keep trying. ... Exiting with error: 3100000 plugin_exception: plugin exception (Full log attached) During this time, the output of elasticsearch is unremarkable. (No error messages.)
Replicating the experiment with 1.5.19 again does NOT change the outcome. Likely, whatever the problem with pushing data to the ES node is, it had already existed and was not introduced in 1.5.22-alpha or any of the recent alpha releases.

Summary:

Node replays until is has queued the first data bundle ("10000 lines") to push to the elasticsearch node. But... it fails when it actually attempts to push the data. As of yet, the reason for this failure is unclear.

Note on documentation:

Note that the documentation links in the issue description are not working for me. (I get 403 - Forbidden) As an alternative, I have been following BitShares docs from these sources:

@bobinson @tai.sama

## Experiment 01 — Report: ### Test setup: - A 1.5.22-alpha node with modified (future-dated) HF dates synced with mainnet, running inside an Ubuntu 20.04 docker container. - An ElasticSearch 6.4.3 node running in a separate docker container on the same machine, mapping port 9200. - Note: Latest version of ElasticSearch (8.7.0) rapidly consumed >50% of system RAM even before loading any data, so I selected 6.4.3 instead, as it is the closest version available on DockerHub to the last known working ES version for the ES plugin on BitShares (v. 6.2.0) - Peerplays node configured with ONLY the elasticsearch plugin loaded. ### Test outcome: :x: Failure The peerplays node errors out with error message: `ES database is not up in url http://<redacted>:9200` ``` root@a6333dae7181:/srv/Node/peerplays-main# ./bin/witness_node --replay-blockchain 2728818ms th_a bookie_plugin.cpp:412 plugin_initialize ] bookie plugin: plugin_startup() begin 2728818ms th_a db_management.cpp:295 force_slow_replays ] enabling slow replays 2728818ms th_a bookie_plugin.cpp:435 plugin_initialize ] bookie plugin: plugin_startup() end 2728818ms th_a elasticsearch_plugin.cpp:698 plugin_initialize ] elasticsearch ACCOUNT HISTORY: plugin_initialize() begin 2728818ms th_a main.cpp:190 main ] Exiting with error: 0 exception: unspecified ES database is not up in url http://<redacted>:9200 {"url":"http://<redacted>:9200"} th_a elasticsearch_plugin.cpp:723 plugin_initialize root@a6333dae7181:/srv/Node/peerplays-main# ``` This is despite the ES node in fact being up and running and reachable from the container: ``` root@a6333dae7181:/srv/Node/peerplays-main# curl -X GET 'http://<redacted>:9200/peerplays-*/data/_count?pretty=true' -H 'Content-Type: application/json' -d '{"query" : {"bool" : { "must" : [{"match_all": {}}] }}}' { "count" : 0, "_shards" : { "total" : 0, "successful" : 0, "skipped" : 0, "failed" : 0 } } root@a6333dae7181:/srv/Node/peerplays-main# ``` ### Investigation: Further investigation (e.g. reviewing the plugin code that opens the ES connection) is underway. #### Observations: - Replicating the experiment with 1.5.19 does NOT change the outcome. Likely the problem has long existed and was not introduced in 1.5.22-alpha or any of the recent alpha releases. - After inspecting code, noticed: Plugin code constructs the health check URL as: `curl_request.url = es.elasticsearch_url + "_nodes"`. This fails to insert a `/` between server and file endpoint. Thus, ensuring value of `elasticsearch-node-url` in config.ini has a trailing slash should fix the error. - With the trailing slash added to the url, replay continues until the fist 10000 lines are queued for submission to the ES node. However, at this point 1.5.22-alpha exits with error: `Error sending 10000 lines of bulk data to Elastic Search, the first lines are:` ... ` Failed to push new block: 3100000 plugin_exception: plugin exception, Error populating ES database, we are going to keep trying.` ... `Exiting with error: 3100000 plugin_exception: plugin exception` (Full log attached) During this time, the output of elasticsearch is unremarkable. (No error messages.) - Replicating the experiment with 1.5.19 again does NOT change the outcome. Likely, whatever the problem with pushing data to the ES node is, it had already existed and was not introduced in 1.5.22-alpha or any of the recent alpha releases. ### Summary: Node replays until is has queued the first data bundle ("10000 lines") to push to the elasticsearch node. But... it fails when it actually attempts to push the data. As of yet, the reason for this failure is unclear. ### Note on documentation: Note that the documentation links in the issue description are not working for me. (I get `403 - Forbidden`) As an alternative, I have been following BitShares docs from these sources: - https://dev.bitshares.works/en/latest/supports_dev/elastic_search_plugin.html - https://github.com/bitshares/bitshares-core/wiki/ElasticSearch-Plugin @bobinson @tai.sama

bobinson commented

2023-04-20 08:07:36 +00:00

(Migrated from gitlab.com)

@christophersanborn Please login to confluence to access the documentation

...

Sent from Outlook for iOShttps://aka.ms/o0ukef

From: gitlab@mg.gitlab.com gitlab@mg.gitlab.com on behalf of Bobinson K B (@bobinson) gitlab@mg.gitlab.com
Sent: Thursday, April 20, 2023 6:20:15 AM
To: bobinson@gmail.com bobinson@gmail.com
Subject: Re: peerplays | Verify Elastic Search plugin on 1.5.22-alpha (#528)

Bobinson K Bhttps://gitlab.com/bobinson's issue #528 https://gitlab.com/PBSA/peerplays/-/issues/528 is due soon.

Assignee: Christopher Sanborn

This issue is due on: Apr 21, 2023

Available documentation:

https://peerplays.atlassian.net/wiki/spaces/PROJECTS/pages/1159299078/Quick+setup+of+Elasticsearch+on+AWS+and+connecting+a+witness+node+to+it

https://peerplays.atlassian.net/wiki/spaces/PIX/pages/364609555/ElasticSearch+Plugin+for+Peerplays

https://peerplays.atlassian.net/wiki/spaces/PROJECTS/pages/1367113733/Quick+setup+of+Elasticsearch+plugin+on+local+computer

cc @tai.samahttps://gitlab.com/tai.sama : We may need this plugin for the explorer's REST API.

—
Reply to this email directly or view it on GitLabhttps://gitlab.com/PBSA/peerplays/-/issues/528.
You're receiving this email because of your account on gitlab.comhttps://gitlab.com. Unsubscribehttps://gitlab.com/-/sent_notifications/REDACTED/unsubscribe from this thread · Manage all notificationshttps://gitlab.com/-/profile/notifications · Helphttps://gitlab.com/help

@christophersanborn Please login to confluence to access the documentation <details><summary>...</summary> Sent from Outlook for iOS<https://aka.ms/o0ukef> ________________________________ From: gitlab@mg.gitlab.com <gitlab@mg.gitlab.com> on behalf of Bobinson K B (@bobinson) <gitlab@mg.gitlab.com> Sent: Thursday, April 20, 2023 6:20:15 AM To: bobinson@gmail.com <bobinson@gmail.com> Subject: Re: peerplays | Verify Elastic Search plugin on 1.5.22-alpha (#528) Bobinson K B<https://gitlab.com/bobinson>'s issue #528<https://gitlab.com/PBSA/peerplays/-/issues/528> is due soon. Assignee: Christopher Sanborn This issue is due on: Apr 21, 2023 Available documentation: https://peerplays.atlassian.net/wiki/spaces/PROJECTS/pages/1159299078/Quick+setup+of+Elasticsearch+on+AWS+and+connecting+a+witness+node+to+it https://peerplays.atlassian.net/wiki/spaces/PIX/pages/364609555/ElasticSearch+Plugin+for+Peerplays https://peerplays.atlassian.net/wiki/spaces/PROJECTS/pages/1367113733/Quick+setup+of+Elasticsearch+plugin+on+local+computer cc @tai.sama<https://gitlab.com/tai.sama> : We may need this plugin for the explorer's REST API. — Reply to this email directly or view it on GitLab<https://gitlab.com/PBSA/peerplays/-/issues/528>. You're receiving this email because of your account on gitlab.com<https://gitlab.com>. Unsubscribe<https://gitlab.com/-/sent_notifications/REDACTED/unsubscribe> from this thread · Manage all notifications<https://gitlab.com/-/profile/notifications> · Help<https://gitlab.com/help> </details>

tai.sama commented

2023-04-20 13:13:12 +00:00

(Migrated from gitlab.com)

after getting the elastic search verified, we need to get the chainworker running to have the APIs for total suppy / max supply / circulating supply

christophersanborn commented

2023-04-26 17:14:26 +00:00

(Migrated from gitlab.com)

OK, so we don't have (that I know about) a "simple rudimentary" block explorer running (yet) to serve the info needed for CMC submission. We could stand up at least the API for one fairly easily with a lightweight API server package like Falcon API. Slapping a simple front-end GUI on top of that wouldn't be too hard either.

IMHO, a solution based on ElasticSearch is too heavy. Very basic chain exploration can be achieved without it. There are also repos in https://gitlab.com/PBSA/peerplays-1.0/dapps/block-explorer, but these are all based on populating SQL databases and imho are also "heavy".

The basic info requested can be retrieved from any core node with API port open as follows. Note, it's not in the requested format, but it is the correct information.

Max Supply:

Request: "Max Supply = maximum # of coins that will ever exist in the lifetime of the cryptocurrency."
How to retrieve: Get object 1.3.0, which is the asset object of the PPY token. With curl:

curl https://api.artcasa.gallery/ws/ppy -d '{"method": "call", "params": [0, "get_objects", [["1.3.0"]]], "jsonrpc": "2.0", "id": 2}'

The result will be the whole asset object. To get specifically max supply, extract property max_supply from 'optionsfromresult[0]. With jq`, this would look like:

curl https://api.artcasa.gallery/ws/ppy -d '{"method": "call", "params": [0, "get_objects", [["1.3.0"]]], "jsonrpc": "2.0", "id": 2}' | jq .result[0].options.max_supply

Note: The result is in satoshi units. To represent in "human" units, you need to also retrieve the asset precision and divide by 10**prec
Note 2: On Graphene chains the issuer of an asset can modify the max_supply parameter, so it is unclear whether this parameter really agrees with "maximum # of coins that will ever exist". The issuer of the PPY token is committee-account.

Circulating Supply:

Request: "Circulating Supply = # of coins that are circulating in the market and in the general public's hands."
How to retrieve: This is tracked in the current_supply property of the asset statistics object. (for PPY, this is object 2.3.0.) Retrieve with curl and jq as follows:

curl https://api.artcasa.gallery/ws/ppy -d '{"method": "call", "params": [0, "get_objects", [["2.3.0"]]], "jsonrpc": "2.0", "id": 2}' | jq .result[0].current_supply

Note: The result is in satoshi units. To represent in "human" units, you need to also retrieve the asset precision and divide by 10**prec

Total Supply:

Request: "Total Supply = # of coins in existence right now (minus any coins that have been verifiably burned."
Arguments can be made for this quantity being equivalent to either the Max Supply or the Circulating Supply. But whatever view you take,... it's one of these.

Rich List:

This can be retrieved via the RPC get_asset_holders. However, by default most nodes will not have this RPC call enabled. The node must be configured to offer the "asset API". If thus configured, the rich list can be retrieved as follows:

curl https://api.artcasa.gallery/ws/ppy -d '{"method": "call", "params": ["asset", "get_asset_holders", ["PPY","0","20"]], "jsonrpc": "2.0", "id": 2}' | jq .result[]

Note: The amount results are in satoshi units. To represent in "human" units, you need to also retrieve the asset precision and divide by 10**prec

@tai.sama
@bobinson

OK, so we don't have (that I know about) a "simple rudimentary" block explorer running (yet) to serve the info needed for CMC submission. We could stand up at least the API for one fairly easily with a lightweight API server package like [Falcon API](https://pypi.org/project/falcon/). Slapping a simple front-end GUI on top of that wouldn't be too hard either. IMHO, a solution based on ElasticSearch is too heavy. Very basic chain exploration can be achieved without it. There are also repos in https://gitlab.com/PBSA/peerplays-1.0/dapps/block-explorer, but these are all based on populating SQL databases and imho are also "heavy". The basic info requested can be retrieved from any core node with API port open as follows. Note, it's not in the requested format, but it is the correct information. ### Max Supply: * Request: _"Max Supply = maximum # of coins that will ever exist in the lifetime of the cryptocurrency."_ * How to retrieve: Get object 1.3.0, which is the asset object of the PPY token. With curl: ``` curl https://api.artcasa.gallery/ws/ppy -d '{"method": "call", "params": [0, "get_objects", [["1.3.0"]]], "jsonrpc": "2.0", "id": 2}' ``` The result will be the whole asset object. To get specifically max supply, extract property `max_supply` from 'options` from `result[0]`. With `jq`, this would look like: ``` curl https://api.artcasa.gallery/ws/ppy -d '{"method": "call", "params": [0, "get_objects", [["1.3.0"]]], "jsonrpc": "2.0", "id": 2}' | jq .result[0].options.max_supply ``` * Note: The result is in satoshi units. To represent in "human" units, you need to also retrieve the asset precision and divide by 10**prec * Note 2: On Graphene chains the issuer of an asset can modify the max_supply parameter, so it is unclear whether this parameter really agrees with "maximum # of coins that will _**ever**_ exist". The issuer of the PPY token is `committee-account`. ### Circulating Supply: * Request: _"Circulating Supply = # of coins that are circulating in the market and in the general public's hands."_ * How to retrieve: This is tracked in the `current_supply` property of the asset statistics object. (for PPY, this is object 2.3.0.) Retrieve with curl and jq as follows: ``` curl https://api.artcasa.gallery/ws/ppy -d '{"method": "call", "params": [0, "get_objects", [["2.3.0"]]], "jsonrpc": "2.0", "id": 2}' | jq .result[0].current_supply ``` * Note: The result is in satoshi units. To represent in "human" units, you need to also retrieve the asset precision and divide by 10**prec ### Total Supply: * Request: _"Total Supply = # of coins in existence right now (minus any coins that have been verifiably burned."_ * Arguments can be made for this quantity being equivalent to either the Max Supply or the Circulating Supply. But whatever view you take,... it's one of these. ### Rich List: * This can be retrieved via the RPC `get_asset_holders`. However, by default most nodes will not have this RPC call enabled. The node must be configured to offer the "asset API". If thus configured, the rich list can be retrieved as follows: ``` curl https://api.artcasa.gallery/ws/ppy -d '{"method": "call", "params": ["asset", "get_asset_holders", ["PPY","0","20"]], "jsonrpc": "2.0", "id": 2}' | jq .result[] ``` * Note: The amount results are in satoshi units. To represent in "human" units, you need to also retrieve the asset precision and divide by 10**prec @tai.sama @bobinson

tai.sama commented

2023-05-03 13:06:45 +00:00

(Migrated from gitlab.com)

1.) A subdomain or endpoint for a minimally-sufficient explorer.

https://explorer.peerplays.com/

2.) Something hosted at that endpoint.

=> part of NEX, so no need to be hosted separately

3.) Functional explorer API

peerplays-explorer-api DOES WORK! But returns errors for requests is cannot satisfy (Any queries which rely on elastic search will fail. BUT — the ones CMC needs do not need elastic search.)
EXAMPLE: http://96.46.48.205:5100/apidocs/ works. But note, this is connected to TESTNET.
The explorer is a dated python2.7 code that is a little finicky to get running, and may not be stable, but… does work.
Links to the API endpoints would return either plaintext or JSON, but would arguably satisfy CMC’s requirements.
Some cleanup of the peerplays-explorer-api code might be desired to disable or hide the non-functional request types. But CMC may never check these so it might not matter.

4.) At least one main-net API node needs to have “Asset API” enabled.

This is a very simple config file addition. This API is needed for the explorer to get the “rich list.” We’ve been discussing this in the test-net group.

1.) A subdomain or endpoint for a minimally-sufficient explorer. https://explorer.peerplays.com/ 2.) Something hosted at that endpoint. => part of NEX, so no need to be hosted separately 3.) Functional explorer API peerplays-explorer-api DOES WORK! But returns errors for requests is cannot satisfy (Any queries which rely on elastic search will fail. BUT — the ones CMC needs do not need elastic search.) EXAMPLE: http://96.46.48.205:5100/apidocs/ works. But note, this is connected to TESTNET. The explorer is a dated python2.7 code that is a little finicky to get running, and may not be stable, but… does work. Links to the API endpoints would return either plaintext or JSON, but would arguably satisfy CMC’s requirements. Some cleanup of the peerplays-explorer-api code might be desired to disable or hide the non-functional request types. But CMC may never check these so it might not matter. 4.) At least one main-net API node needs to have “Asset API” enabled. This is a very simple config file addition. This API is needed for the explorer to get the “rich list.” We’ve been discussing this in the test-net group.

tai.sama commented

2023-05-03 13:13:19 +00:00

(Migrated from gitlab.com)

changed due date to April 24, 2023

tai.sama commented

2023-05-03 13:13:25 +00:00

(Migrated from gitlab.com)

changed due date to May 08, 2023

christophersanborn commented

2023-05-10 10:08:07 +00:00

(Migrated from gitlab.com)

Sample "explorer" providing JUST basics like (max, total, circulating) supply, and rich list, hosted here: http://96.46.48.205:5200/apidocs

The above will need to be hosted somewhere with a DNS entry.

Open question: In our ecosystem, is "total supply" equivalent to "max supply"? or "circulating supply"? I have mapped it to "max supply".

@tai.sama @bobinson

Sample "explorer" providing JUST basics like (max, total, circulating) supply, and rich list, hosted here: http://96.46.48.205:5200/apidocs The above will need to be hosted somewhere with a DNS entry. Open question: In our ecosystem, is "total supply" equivalent to "max supply"? or "circulating supply"? I have mapped it to "max supply". @tai.sama @bobinson

tai.sama (Migrated from gitlab.com) closed this issue

2023-06-14 13:11:23 +00:00

bobinson commented

2023-06-14 13:12:31 +00:00

(Migrated from gitlab.com)

The functionality is implemented by marketcap API

Verify Elastic Search plugin on 1.5.22-alpha #528

Experiment 01 — Report:

Test setup:

Test outcome: ❌ Failure

Investigation:

Observations:

Summary:

Note on documentation:

Max Supply:

Circulating Supply:

Total Supply:

Rich List: