[flaky test] GCP test package keep failing with could not find hits in metrics-gcp.compute-ep data stream

[kaiyan-sheng]

After increasing the timeout in #1366, gcp test package is still failing in #1379.

This can be reproduced with the gcp test package in this repository:

cd test/packages/parallel/gcp
go run github.com/elastic/elastic-package stack up -v -d
eval $(go run github.com/elastic/elastic-package stack shellinit)
go run github.com/elastic/elastic-package test -v

[kaiyan-sheng]

This test is skipped for now: #1382

[tdancheva]

Testing the input/metrics with compute, we are not getting the expected 1 document/instance per 60s (the default collection period), and thereby the current timeout of 40min doesn't cut it. Investigating further as to why.

[gpop63]

Just tested this locally and it seems to work.

2023/09/21 01:31:35 DEBUG found 0 hits in metrics-gcp.compute-ep data stream: index_not_found_exception: no such index [metrics-gcp.compute-ep] Status=404
2023/09/21 01:31:36 DEBUG found 0 hits in metrics-gcp.compute-ep data stream: index_not_found_exception: no such index [metrics-gcp.compute-ep] Status=404
2023/09/21 01:31:37 DEBUG found 0 hits in metrics-gcp.compute-ep data stream: index_not_found_exception: no such index [metrics-gcp.compute-ep] Status=404
2023/09/21 01:31:38 DEBUG found 0 hits in metrics-gcp.compute-ep data stream: index_not_found_exception: no such index [metrics-gcp.compute-ep] Status=404
2023/09/21 01:31:39 DEBUG found 0 hits in metrics-gcp.compute-ep data stream
2023/09/21 01:31:40 DEBUG found 32 hits in metrics-gcp.compute-ep data stream
...
--- Test results for package: gcp - START ---
╭─────────┬─────────────┬───────────┬───────────┬────────┬─────────────────╮
│ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME │ RESULT │    TIME ELAPSED │
├─────────┼─────────────┼───────────┼───────────┼────────┼─────────────────┤
│ gcp     │ compute     │ system    │ default   │ PASS   │ 1m55.750454897s │
╰─────────┴─────────────┴───────────┴───────────┴────────┴─────────────────╯
--- Test results for package: gcp - END   ---
Done

[tdancheva]

Test enabled and working.