peerplays logs are pushed using witness id as id due to which documents are getting overwritten #203

New issue

Closed

opened 2021-10-26 19:07:57 +00:00 by prandnum · 10 comments

prandnum commented

2021-10-26 19:07:57 +00:00

(Migrated from gitlab.com)

peerplays logs are pushed using witness id as id due to which documents are getting overwritten. view the screenshot below:

We need to remove witnessid as id and use incrementing counter so that the document is not overwritten.

document examples

{
  "_index": "ppobjects-witness",
  "_type": "data",
  "_id": "1.6.4",
  "_version": 101,
  "_score": null,
  "_source": {
    "id": "1.6.4",
    "witness_account": "1.2.10",
    "last_aslot": 3270,
    "signing_key": "TEST6MRyAjQq8ud7hVNYcfnVPJqcVpscN5So8BhtHuGYqET5GDW5CV",
    "next_secret_hash": "b13da4d33fc9d12a89cbf673b07cbdcd91acb2d5",
    "previous_secret": "aab4e7a4a1212f69e8737e8759a2fbda89f3cbf1",
    "vote_id": "1:3",
    "total_votes": 0,
    "url": "",
    "total_missed": 0,
    "last_confirmed_block_num": 1081,
    "object_id": "1.6.4",
    "block_time": "2021-10-26T18:44:54",
    "block_number": 1081
  },
  "fields": {
    "block_time": [
      "2021-10-26T18:44:54.000Z"
    ]
  },
  "sort": [
    1635273894000
  ]
}


{
  "_index": "ppobjects-witness",
  "_type": "data",
  "_id": "1.6.11",
  "_version": 100,
  "_score": null,
  "_source": {
    "id": "1.6.11",
    "witness_account": "1.2.17",
    "last_aslot": 3266,
    "signing_key": "TEST6MRyAjQq8ud7hVNYcfnVPJqcVpscN5So8BhtHuGYqET5GDW5CV",
    "next_secret_hash": "a93fbd81c9d22b8fb63de1436d0692bd8689752d",
    "previous_secret": "f6c9e6340fda4c9ac71c40babe0fb70e2841cbe2",
    "vote_id": "1:10",
    "total_votes": 0,
    "url": "",
    "total_missed": 0,
    "last_confirmed_block_num": 1077,
    "object_id": "1.6.11",
    "block_time": "2021-10-26T18:44:42",
    "block_number": 1077
  },
  "fields": {
    "block_time": [
      "2021-10-26T18:44:42.000Z"
    ]
  },
  "sort": [
    1635273882000
  ]
}

peerplays logs are pushed using witness id as id due to which documents are getting overwritten. view the screenshot below: ![elk](/uploads/cae3bacf6945aa979f9e2e426fafef35/elk.jpg) We need to remove witnessid as id and use incrementing counter so that the document is not overwritten. document examples ``` { "_index": "ppobjects-witness", "_type": "data", "_id": "1.6.4", "_version": 101, "_score": null, "_source": { "id": "1.6.4", "witness_account": "1.2.10", "last_aslot": 3270, "signing_key": "TEST6MRyAjQq8ud7hVNYcfnVPJqcVpscN5So8BhtHuGYqET5GDW5CV", "next_secret_hash": "b13da4d33fc9d12a89cbf673b07cbdcd91acb2d5", "previous_secret": "aab4e7a4a1212f69e8737e8759a2fbda89f3cbf1", "vote_id": "1:3", "total_votes": 0, "url": "", "total_missed": 0, "last_confirmed_block_num": 1081, "object_id": "1.6.4", "block_time": "2021-10-26T18:44:54", "block_number": 1081 }, "fields": { "block_time": [ "2021-10-26T18:44:54.000Z" ] }, "sort": [ 1635273894000 ] } { "_index": "ppobjects-witness", "_type": "data", "_id": "1.6.11", "_version": 100, "_score": null, "_source": { "id": "1.6.11", "witness_account": "1.2.17", "last_aslot": 3266, "signing_key": "TEST6MRyAjQq8ud7hVNYcfnVPJqcVpscN5So8BhtHuGYqET5GDW5CV", "next_secret_hash": "a93fbd81c9d22b8fb63de1436d0692bd8689752d", "previous_secret": "f6c9e6340fda4c9ac71c40babe0fb70e2841cbe2", "vote_id": "1:10", "total_votes": 0, "url": "", "total_missed": 0, "last_confirmed_block_num": 1077, "object_id": "1.6.11", "block_time": "2021-10-26T18:44:42", "block_number": 1077 }, "fields": { "block_time": [ "2021-10-26T18:44:42.000Z" ] }, "sort": [ 1635273882000 ] } ```

prandnum commented

2021-10-26 19:07:57 +00:00

(Migrated from gitlab.com)

assigned to @serkixenos

prandnum commented

2021-10-26 19:08:22 +00:00

(Migrated from gitlab.com)

@bobinson @serkixenos

CC: @hbelakon

@bobinson @serkixenos CC: @hbelakon

bobinson commented

2021-10-26 20:22:53 +00:00

(Migrated from gitlab.com)

@prandnum - Is it possible to configure ElasticSearch to use different field rather than witness id as id ?

I assume essentially we are looking for a way to keep near identical logs from different environments.

@prandnum - Is it possible to configure ElasticSearch to use different field rather than witness id as id ? I assume essentially we are looking for a way to keep near identical logs from different environments.

serkixenos commented

2021-10-26 22:18:20 +00:00

(Migrated from gitlab.com)

unassigned @serkixenos

serkixenos commented

2021-10-26 22:53:51 +00:00

(Migrated from gitlab.com)

@prandnum This is a devops task, not development. When we use elastic search plugin, we are sending objects to ES, not logs. You need to talk to whoever configured redirecting logs to elastic search to fix this.

serkixenos commented

2021-10-28 19:05:48 +00:00

(Migrated from gitlab.com)

assigned to @sivayavvari

serkixenos commented

2021-10-28 19:05:56 +00:00

(Migrated from gitlab.com)

assigned to @rilesdun

rilesdun commented

2022-07-15 18:28:36 +00:00

(Migrated from gitlab.com)

mentioned in issue PBSA/tools-libs/infrastructure/devops-misc-issues#15