Rally benchmark aws.ec2 logs #8416

aspacca · 2023-11-07T08:38:15Z

Enhancement

Proposed commit message

Add artifacts for elastic-package rally benchmark

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
~~- [ ] I have added an entry to my package's changelog.yml file.~~
~~- [ ] I have verified that Kibana version constraints are current according to guidelines.~~

Author's Checklist

[ ]

How to test this PR locally

Checkout branch from elastic/elastic-package#1522, build elastic-package and execute from aws package root (remember to bring up the elastic-package stack before):
./elastic-package benchmark rally --benchmark ec2logs-benchmark -v

Related issues

Screenshots

--- Benchmark results for package: aws - START ---
╭──────────────────────────────────────────────────────────────────────────────────────────────────╮
│ info                                                                                             │
├────────────────────────┬─────────────────────────────────────────────────────────────────────────┤
│ benchmark              │                                                      ec2_logs-benchmark │
│ description            │                            Benchmark 20000 aws.ec2_logs events ingested │
│ run ID                 │                                    6488a6a7-8b97-4e14-8331-dee3876ae886 │
│ package                │                                                                     aws │
│ start ts (s)           │                                                              1699346225 │
│ end ts (s)             │                                                              1699346258 │
│ duration               │                                                                     33s │
│ generated corpora file │ /Users/andreaspacca/.elastic-package/tmp/rally_corpus/corpus-2835545848 │
╰────────────────────────┴─────────────────────────────────────────────────────────────────────────╯
╭────────────────────────────────────────────────────────────────────────╮
│ parameters                                                             │
├─────────────────────────────────┬──────────────────────────────────────┤
│ package version                 │                                2.8.5 │
│ data_stream.name                │                             ec2_logs │
│ corpora.generator.total_events  │                                20000 │
│ corpora.generator.template.path │ ./ec2_logs-benchmark/template.ndjson │
│ corpora.generator.template.raw  │                                      │
│ corpora.generator.template.type │                               gotext │
│ corpora.generator.config.path   │      ./ec2_logs-benchmark/config.yml │
│ corpora.generator.config.raw    │                                map[] │
│ corpora.generator.fields.path   │      ./ec2_logs-benchmark/fields.yml │
│ corpora.generator.fields.raw    │                                map[] │
╰─────────────────────────────────┴──────────────────────────────────────╯
╭───────────────────────╮
│ cluster info          │
├───────┬───────────────┤
│ name  │ elasticsearch │
│ nodes │             1 │
╰───────┴───────────────╯
╭───────────────────────────────────────────────────╮
│ data stream stats                                 │
├────────────────────────────┬──────────────────────┤
│ data stream                │ logs-aws.ec2_logs-ep │
│ approx total docs ingested │                20000 │
│ backing indices            │                    2 │
│ store size bytes           │              7511247 │
│ maximum ts (ms)            │        1699378626497 │
╰────────────────────────────┴──────────────────────╯
╭───────────────────────────────────────╮
│ disk usage for index .ds-logs-aws.ec2 │
│ _logs-ep-2023.11.07-000002 (for all f │
│ ields)                                │
├──────────────────────────────┬────────┤
│ total                        │ 7.1 MB │
│ inverted_index.total         │ 2.7 MB │
│ inverted_index.stored_fields │ 2.4 MB │
│ inverted_index.doc_values    │ 2.0 MB │
│ inverted_index.points        │  86 kB │
│ inverted_index.norms         │    0 B │
│ inverted_index.term_vectors  │    0 B │
│ inverted_index.knn_vectors   │    0 B │
╰──────────────────────────────┴────────╯
╭────────────────────────────────────╮
│ disk usage for index .ds-logs-aws. │
│ ec2_logs-ep-2023.11.07-000001 (for │
│ all fields)                        │
├──────────────────────────────┬─────┤
│ total                        │ 0 B │
│ inverted_index.total         │ 0 B │
│ inverted_index.stored_fields │ 0 B │
│ inverted_index.doc_values    │ 0 B │
│ inverted_index.points        │ 0 B │
│ inverted_index.norms         │ 0 B │
│ inverted_index.term_vectors  │ 0 B │
│ inverted_index.knn_vectors   │ 0 B │
╰──────────────────────────────┴─────╯
╭──────────────────────────────────────────────────────────────────────────────╮
│ pipeline logs-aws.ec2_logs-2.8.5 stats in node 7AYCd2EXQaCSOf-0fKxFBg        │
├─────────────────────────────────────┬────────────────────────────────────────┤
│ Totals                              │ Count: 20000 | Failed: 0 | Time: 874ms │
│ set ()                              │  Count: 20000 | Failed: 0 | Time: 24ms │
│ rename ()                           │  Count: 20000 | Failed: 0 | Time: 36ms │
│ remove ()                           │   Count: 20000 | Failed: 0 | Time: 6ms │
│ grok ()                             │ Count: 20000 | Failed: 0 | Time: 338ms │
│ date ()                             │ Count: 20000 | Failed: 0 | Time: 312ms │
│ remove ()                           │   Count: 20000 | Failed: 0 | Time: 8ms │
│ remove ()                           │        Count: 0 | Failed: 0 | Time: 0s │
│ pipeline (logs-aws.ec2_logs@custom) │   Count: 20000 | Failed: 0 | Time: 5ms │
╰─────────────────────────────────────┴────────────────────────────────────────╯
╭─────────────────────────────────────────────────────────────────────────────────────────────╮
│ rally stats                                                                                 │
├────────────────────────────────────────────────────────────────┬────────────────────────────┤
│ Cumulative indexing time of primary shards                     │     0.5697333333333333 min │
│ Min cumulative indexing time across primary shards             │                      0 min │
│ Median cumulative indexing time across primary shards          │   0.008183333333333332 min │
│ Max cumulative indexing time across primary shards             │                  0.095 min │
│ Cumulative indexing throttle time of primary shards            │                      0 min │
│ Min cumulative indexing throttle time across primary shards    │                      0 min │
│ Median cumulative indexing throttle time across primary shards │                    0.0 min │
│ Max cumulative indexing throttle time across primary shards    │                      0 min │
│ Cumulative merge time of primary shards                        │                0.15155 min │
│ Cumulative merge count of primary shards                       │                        142 │
│ Min cumulative merge time across primary shards                │                      0 min │
│ Median cumulative merge time across primary shards             │                0.00025 min │
│ Max cumulative merge time across primary shards                │   0.020783333333333334 min │
│ Cumulative merge throttle time of primary shards               │                      0 min │
│ Min cumulative merge throttle time across primary shards       │                      0 min │
│ Median cumulative merge throttle time across primary shards    │                    0.0 min │
│ Max cumulative merge throttle time across primary shards       │                      0 min │
│ Cumulative refresh time of primary shards                      │                 0.0821 min │
│ Cumulative refresh count of primary shards                     │                       4664 │
│ Min cumulative refresh time across primary shards              │                      0 min │
│ Median cumulative refresh time across primary shards           │  0.0016833333333333333 min │
│ Max cumulative refresh time across primary shards              │   0.006866666666666666 min │
│ Cumulative flush time of primary shards                        │     2.8998166666666667 min │
│ Cumulative flush count of primary shards                       │                       4398 │
│ Min cumulative flush time across primary shards                │ 3.3333333333333335e-05 min │
│ Median cumulative flush time across primary shards             │   0.006491666666666667 min │
│ Max cumulative flush time across primary shards                │    0.22751666666666667 min │
│ Total Young Gen GC time                                        │                    0.075 s │
│ Total Young Gen GC count                                       │                          2 │
│ Total Old Gen GC time                                          │                        0 s │
│ Total Old Gen GC count                                         │                          0 │
│ Store size                                                     │     0.37140679359436035 GB │
│ Translog size                                                  │  0.00015551503747701645 GB │
│ Heap used for segments                                         │                       0 MB │
│ Heap used for doc values                                       │                       0 MB │
│ Heap used for terms                                            │                       0 MB │
│ Heap used for norms                                            │                       0 MB │
│ Heap used for points                                           │                       0 MB │
│ Heap used for stored fields                                    │                       0 MB │
│ Segment count                                                  │                        701 │
│ Total Ingest Pipeline count                                    │                      20018 │
│ Total Ingest Pipeline time                                     │                    2.084 s │
│ Total Ingest Pipeline failed                                   │                          0 │
│ Min Throughput                                                 │            28833.54 docs/s │
│ Mean Throughput                                                │            28833.54 docs/s │
│ Median Throughput                                              │            28833.54 docs/s │
│ Max Throughput                                                 │            28833.54 docs/s │
│ 50th percentile latency                                        │       599.2343544999983 ms │
│ 100th percentile latency                                       │        605.730125000001 ms │
│ 50th percentile service time                                   │       599.2343544999983 ms │
│ 100th percentile service time                                  │        605.730125000001 ms │
│ error rate                                                     │                     0.00 % │
╰────────────────────────────────────────────────────────────────┴────────────────────────────╯

--- Benchmark results for package: aws - END   ---
Done

elasticmachine · 2023-11-07T08:49:26Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-11-24T03:18:07.500+0000
Duration: 78 min 25 sec

Test stats 🧪

Test	Results
Failed	0
Passed	223
Skipped	3
Total	226

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

aspacca · 2023-11-08T00:42:26Z

/test

elasticmachine · 2023-11-08T02:54:39Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`17/17`)	💚
Files	94.444% (`17/18`)	👍
Classes	94.444% (`17/18`)	👍
Methods	89.701% (`270/301`)	👍
Lines	86.083% (`7571/8795`)	👍
Conditionals	100.0% (`0/0`)	💚

tommyers-elastic · 2023-11-10T19:18:34Z

packages/aws/_dev/benchmark/rally/ec2logs-benchmark/config.yml

+    value: "da6cb4c8-c84c-4c5f-97c7-f8586a098af4"
+  - name: cloud.region
+    enum: ["us-east-1", "us-east-2", "us-west-1", "us-west-2", "ap-south-1", "ap-northeast-3", "ap-northeast-2", "ap-southeast-1", "ap-southeast-2", "ap-northeast-1", "ca-central-1", "eu-central-1", "eu-west-1", "eu-west-2", "eu-west-3", "eu-north-1", "sa-east-1", "af-south-1", "ap-east-1", "ap-south-2", "ap-southeast-3", "eu-south-2", "eu-central-2", "me-south-1", "me-central-1"]
+    cardinality: 100


same question here as in the other PR - how can a field have cardinality of 100, when it has fewer possible values?

wanted to follow up on this cardinality value - where does 100 come from?

I just want to have 100 ec2 instances (aws.cloudwatch.log_stream/host.name/aws.ec2.ip_address): since cloud.region is an enum with length 25, 100 itself is the LCM

so if you didn't specify this - you would end up with 25 unique instances? by specifying this, for each region, you generate 4 unique values of the log_stream, host.name and ip_address, resulting in 100 unique instances?

aspacca · 2023-11-16T00:33:24Z

@tommyers-elastic all good here? :)

aspacca · 2023-11-22T23:05:21Z

@elastic/ecosystem I'd need your CR

Andrea Spacca added 5 commits November 7, 2023 12:15

add aws.ec2_logs rally benchmark

cf36d26

fix typo in template

64e8a87

fix data_stream name

246d4a6

fix data_stream.namespace

ed0e8f7

fix event.dataset

dc7f053

aspacca self-assigned this Nov 7, 2023

aspacca requested a review from a team as a code owner November 7, 2023 08:38

aspacca mentioned this pull request Nov 7, 2023

Add schema-b artificats currently on corpus generator's repo to artifacts in elastic-package for rally benchmark command elastic/elastic-package#1476

Closed

andrewkroh added the Integration:aws AWS label Nov 7, 2023

bump github.com/elastic/package-spec/v3@v3.0.1

eae6f14

aspacca requested a review from a team as a code owner November 8, 2023 01:01

Merge branch 'main' into rally_benchmark_aws.ec2_logs

2d146b4

rename ec2_logs to ec2logs

75a66c0

tommyers-elastic reviewed Nov 10, 2023

View reviewed changes

Merge branch 'main' into rally_benchmark_aws.ec2_logs

dc88e64

jsoriano requested review from a team and removed request for a team November 23, 2023 13:09

fix period

5b237bb

tommyers-elastic approved these changes Dec 4, 2023

View reviewed changes

aspacca merged commit d0de339 into elastic:main Dec 4, 2023

ali786XI mentioned this pull request Feb 15, 2024

[Meta] Add more benchmark schema templates to integrations #8636

Closed

29 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Rally benchmark aws.ec2 logs #8416

Rally benchmark aws.ec2 logs #8416

Uh oh!

aspacca commented Nov 7, 2023 •

edited

Loading

Uh oh!

elasticmachine commented Nov 7, 2023 •

edited

Loading

Build stats

Test stats 🧪

Uh oh!

aspacca commented Nov 8, 2023

Uh oh!

elasticmachine commented Nov 8, 2023 •

edited

Loading

Uh oh!

tommyers-elastic Nov 10, 2023

Uh oh!

tommyers-elastic Nov 23, 2023

Uh oh!

aspacca Nov 23, 2023

Uh oh!

tommyers-elastic Dec 4, 2023

Uh oh!

aspacca commented Nov 16, 2023

Uh oh!

aspacca commented Nov 22, 2023

Uh oh!

Uh oh!

Rally benchmark aws.ec2 logs #8416

Rally benchmark aws.ec2 logs #8416

Uh oh!

Conversation

aspacca commented Nov 7, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

elasticmachine commented Nov 7, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

Uh oh!

aspacca commented Nov 8, 2023

Uh oh!

elasticmachine commented Nov 8, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🌐 Coverage report

Uh oh!

tommyers-elastic Nov 10, 2023

Choose a reason for hiding this comment

Uh oh!

tommyers-elastic Nov 23, 2023

Choose a reason for hiding this comment

Uh oh!

aspacca Nov 23, 2023

Choose a reason for hiding this comment

Uh oh!

tommyers-elastic Dec 4, 2023

Choose a reason for hiding this comment

Uh oh!

aspacca commented Nov 16, 2023

Uh oh!

aspacca commented Nov 22, 2023

Uh oh!

Uh oh!

aspacca commented Nov 7, 2023 •

edited

Loading

elasticmachine commented Nov 7, 2023 •

edited

Loading

elasticmachine commented Nov 8, 2023 •

edited

Loading