Skip to content

capture extra vsphere metrics (cpu %, disk average, disk rate, disk n… #44205

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 28 commits into from
May 26, 2025

Conversation

stefans-elastic
Copy link
Contributor

@stefans-elastic stefans-elastic commented May 5, 2025

…umber, memory usage %)

Proposed commit message

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

Author's Checklist

  • [ ]

How to test this PR locally

Related issues

Use cases

Screenshots

Using vcsim simulation of vSphere API not all metrics were available to check (it appears that vcsim doesn't generate disk usage metrics).
So only 2 of the new metrics were generated by vsim (cpu.percent and memory.percent):
Screenshot 2025-05-19 at 17 31 55

However when I tried commenting out the check if a metric is in list of "available" metrics in the code (line 70 in metricbeat/module/vsphere/client/performance.go) it showed more metrics:
Screenshot 2025-05-19 at 17 41 21

So only disk.numberWrite and disk.numberRead I wasn't able to test at all.

Logs

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label May 5, 2025
Copy link
Contributor

mergify bot commented May 5, 2025

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @stefans-elastic? 🙏.
For such, you'll need to label your PR with:

  • The upcoming major version of the Elastic Stack
  • The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit
  • backport-active-all is the label that automatically backports to all active branches.
  • backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
  • backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

@stefans-elastic stefans-elastic added the Team:Obs-InfraObs Label for the Observability Infrastructure Monitoring team label May 5, 2025
@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label May 5, 2025
Copy link
Contributor

mergify bot commented May 17, 2025

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b vsphere-metrics upstream/vsphere-metrics
git merge upstream/main
git push upstream vsphere-metrics

@stefans-elastic stefans-elastic marked this pull request as ready for review May 21, 2025 07:56
@stefans-elastic stefans-elastic requested a review from a team as a code owner May 21, 2025 07:56
@ishleenk17
Copy link
Member

@stefans-elastic
Since, we have moved the perf manager code to a common place (good step).
I hope we have tested the metrics of host and datastore that were using the previous implementation.

@stefans-elastic
Copy link
Contributor Author

@stefans-elastic Since, we have moved the perf manager code to a common place (good step). I hope we have tested the metrics of host and datastore that were using the previous implementation.

I've just tried it again and host and datastore streams appear to be working fine. Here are screenshots with some of the properties:
Screenshot 2025-05-22 at 15 20 09
Screenshot 2025-05-22 at 15 22 57

Copy link
Member

@ishleenk17 ishleenk17 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for addressing all the comments.
GTG!

@stefans-elastic stefans-elastic merged commit 14ac057 into elastic:main May 26, 2025
46 checks passed
@stefans-elastic stefans-elastic added backport-8.18 Automated backport to the 8.18 branch backport-9.0 Automated backport to the 9.0 branch backport-8.19 Automated backport to the 8.19 branch labels May 26, 2025
mergify bot pushed a commit that referenced this pull request May 26, 2025
#44205)

* capture extra vsphere metrics (cpu %, disk average, disk rate, disk number, memory usage %)

* add missing license headers

* mage fmt

* convert percentage values from fixed point integers

* add CHANGELOG.next.asciidoc entry

* refactor performance data fetching for datastore

* document new fields

* remove unused function

* remove debug log

* add unit tests

* mage fmt

* fixed precision issue for percentage performance manager values

* correct percentage property names

* converts kilobytes per sec to bytes per sec

* update fields.go

* fix linter

* fix type assertion pabic

* fix unit test failure

* address PR comments

* changed percentage fields to scaled_float type

* make field names same between fields.go and fields.yml

---------

Co-authored-by: Ishleen Kaur <102962586+ishleenk17@users.noreply.github.com>
(cherry picked from commit 14ac057)
mergify bot pushed a commit that referenced this pull request May 26, 2025
#44205)

* capture extra vsphere metrics (cpu %, disk average, disk rate, disk number, memory usage %)

* add missing license headers

* mage fmt

* convert percentage values from fixed point integers

* add CHANGELOG.next.asciidoc entry

* refactor performance data fetching for datastore

* document new fields

* remove unused function

* remove debug log

* add unit tests

* mage fmt

* fixed precision issue for percentage performance manager values

* correct percentage property names

* converts kilobytes per sec to bytes per sec

* update fields.go

* fix linter

* fix type assertion pabic

* fix unit test failure

* address PR comments

* changed percentage fields to scaled_float type

* make field names same between fields.go and fields.yml

---------

Co-authored-by: Ishleen Kaur <102962586+ishleenk17@users.noreply.github.com>
(cherry picked from commit 14ac057)

# Conflicts:
#	auditbeat/module/file_integrity/mock_performance.go
#	auditbeat/module/file_integrity/performance.go
#	auditbeat/module/file_integrity/performance_test.go
mergify bot pushed a commit that referenced this pull request May 26, 2025
#44205)

* capture extra vsphere metrics (cpu %, disk average, disk rate, disk number, memory usage %)

* add missing license headers

* mage fmt

* convert percentage values from fixed point integers

* add CHANGELOG.next.asciidoc entry

* refactor performance data fetching for datastore

* document new fields

* remove unused function

* remove debug log

* add unit tests

* mage fmt

* fixed precision issue for percentage performance manager values

* correct percentage property names

* converts kilobytes per sec to bytes per sec

* update fields.go

* fix linter

* fix type assertion pabic

* fix unit test failure

* address PR comments

* changed percentage fields to scaled_float type

* make field names same between fields.go and fields.yml

---------

Co-authored-by: Ishleen Kaur <102962586+ishleenk17@users.noreply.github.com>
(cherry picked from commit 14ac057)
stefans-elastic added a commit that referenced this pull request May 28, 2025
…erage, disk rate, disk n… (#44494)

* capture extra vsphere metrics (cpu %, disk average, disk rate, disk n… (#44205)

* capture extra vsphere metrics (cpu %, disk average, disk rate, disk number, memory usage %)

* add missing license headers

* mage fmt

* convert percentage values from fixed point integers

* add CHANGELOG.next.asciidoc entry

* refactor performance data fetching for datastore

* document new fields

* remove unused function

* remove debug log

* add unit tests

* mage fmt

* fixed precision issue for percentage performance manager values

* correct percentage property names

* converts kilobytes per sec to bytes per sec

* update fields.go

* fix linter

* fix type assertion pabic

* fix unit test failure

* address PR comments

* changed percentage fields to scaled_float type

* make field names same between fields.go and fields.yml

---------

Co-authored-by: Ishleen Kaur <102962586+ishleenk17@users.noreply.github.com>
(cherry picked from commit 14ac057)

* clean up changelog

* fix logger in tests

* mage update

---------

Co-authored-by: stefans-elastic <stefan.stas@elastic.co>
Co-authored-by: Ishleen Kaur <102962586+ishleenk17@users.noreply.github.com>
stefans-elastic added a commit that referenced this pull request May 28, 2025
…erage, disk rate, disk n… (#44495)

* capture extra vsphere metrics (cpu %, disk average, disk rate, disk n… (#44205)

* capture extra vsphere metrics (cpu %, disk average, disk rate, disk number, memory usage %)

* add missing license headers

* mage fmt

* convert percentage values from fixed point integers

* add CHANGELOG.next.asciidoc entry

* refactor performance data fetching for datastore

* document new fields

* remove unused function

* remove debug log

* add unit tests

* mage fmt

* fixed precision issue for percentage performance manager values

* correct percentage property names

* converts kilobytes per sec to bytes per sec

* update fields.go

* fix linter

* fix type assertion pabic

* fix unit test failure

* address PR comments

* changed percentage fields to scaled_float type

* make field names same between fields.go and fields.yml

---------

Co-authored-by: Ishleen Kaur <102962586+ishleenk17@users.noreply.github.com>
(cherry picked from commit 14ac057)

# Conflicts:
#	auditbeat/module/file_integrity/mock_performance.go
#	auditbeat/module/file_integrity/performance.go
#	auditbeat/module/file_integrity/performance_test.go

* resolve merge

* clean up changelog

* fix missing imports

* update docs/fields.asciidoc

* fix linter

---------

Co-authored-by: stefans-elastic <stefan.stas@elastic.co>
stefans-elastic added a commit that referenced this pull request May 28, 2025
…rage, disk rate, disk n… (#44496)

* capture extra vsphere metrics (cpu %, disk average, disk rate, disk n… (#44205)

* capture extra vsphere metrics (cpu %, disk average, disk rate, disk number, memory usage %)

* add missing license headers

* mage fmt

* convert percentage values from fixed point integers

* add CHANGELOG.next.asciidoc entry

* refactor performance data fetching for datastore

* document new fields

* remove unused function

* remove debug log

* add unit tests

* mage fmt

* fixed precision issue for percentage performance manager values

* correct percentage property names

* converts kilobytes per sec to bytes per sec

* update fields.go

* fix linter

* fix type assertion pabic

* fix unit test failure

* address PR comments

* changed percentage fields to scaled_float type

* make field names same between fields.go and fields.yml

---------

Co-authored-by: Ishleen Kaur <102962586+ishleenk17@users.noreply.github.com>
(cherry picked from commit 14ac057)

* clean up changelog

* go mod tidy

* update NOTICE.txt

---------

Co-authored-by: stefans-elastic <stefan.stas@elastic.co>
Co-authored-by: Ishleen Kaur <102962586+ishleenk17@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-8.18 Automated backport to the 8.18 branch backport-8.19 Automated backport to the 8.19 branch backport-9.0 Automated backport to the 9.0 branch enhancement Team:Obs-InfraObs Label for the Observability Infrastructure Monitoring team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Beats. vSphere metrics to be added
2 participants