[Observability] Add observability rules priviliges feature !!

[shahzad31]

Summary

Closes https://github.com/elastic/observability-dev/issues/4683 !!

Enabling this feature will grant user access to all observability rules and alerts.

Observability is added as a consumer and an option for role visibility. Any user with read permissions to the new rules privilege will be able to see rules created for any observability feature, including logs and metrics/infrastructure.

Users who are given read/write access to the new rules feature, but no access to other observability features, may have issues creating rules as they may not be able to view data.

This PR DOES NOT change any ui permissions settings for existing plugins with respect to alerting. Users who only have the "Observability Rules and Alerts" privilege will not be able to manage alerts from the respective plugin header menus if the existing capabilities check for write permissions for that plugin.

Testing

Existing users should be able to access alerts and rules as is.

Plugin consumed rules:

1. Create a new role with "all" permissions for the new "Observability Alerts and Rules" privilege and any necessary "read" permissions for the plugin that produces alerts. Assign to a user.
2. User should be able to create all rules produced by the plugin and view alerts.

Generic rules:

1. Create a new role with "all" permissions for the new "Observability Alerts and Rules" privilege and any index privilege required to create a custom threshold rule. For extensive testing, choose "None" for all other observability privileges.
2. The user should be able to access the observability plugin from the home page and side navigation, as well as alerts. All observability alerts and rules should be visible.
3. Create a custom threshold role (or similar rule not tied to a plugin). If no other plugin permissions were figured, the consumer selection should not be visible and default to observability.
4. The default consumer should still be observability even if the user has metrics and logs permissions. The user should be allowed to change the consumer.

[elasticmachine]

Pinging @elastic/obs-ux-management-team (Team:obs-ux-management)

[baileycash-elastic]

lgtm!

[baileycash-elastic]

This PR more appropriately lays the groundwork for the universal ability to manage alerts and rules in an app/plugin without having to create a subfeature in each one. The only gap I can identify after working on this issue is the ability to manage alert default settings in synthetics.

[cnasikas]

@shahzad31 Could you please update the description with all assumptions? What is the expected behavior of the new feature privilege? What is the expected behavior of the rest of the o11y feature privileges regarding rules and alerts? How does the PR avoid breaking changes? How will this new feature privilege work on serverless? It will help a lot with the review.

[azasypkin]

Can we also turn this one into a Draft until CI is green? 🙏 Just to help manage the review queue more effectively.

[baileycash-elastic]

/ci

[baileycash-elastic]

/ci

[baileycash-elastic]

/ci

[baileycash-elastic]

/ci

[elasticmachine]

Pinging @elastic/obs-ux-infra_services-team (Team:obs-ux-infra_services)

[shahzad31]

You will need to give this feature permission to read all saved objects types being read in rules, examples being synthetics monitors, slo saved objects, infra sources configuration and potentially apm settings

[baileycash-elastic]

Do these work?

[jasonrhodes]

You will need to give this feature permission to read all saved objects types being read in rules, examples being synthetics monitors, slo saved objects, infra sources configuration and potentially apm settings

I wasn't anticipating this, but I see why you mention it. I was picturing this was only to grant the rule permissions, and any additional permissions for the apps would need to be provided by way of the app permissions. But yeah, all these rules will be weirdly broken if we don't provide this access. But then do the APM rules use the APM Kibana REST APIs for their queries? If so, we need to provide that read access in this rule, too? Does the Burn Rate rule use SLO APIs?

[kibanamachine]

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#8907

[❌] x-pack/test_serverless/functional/test_suites/observability/config.ts: 0/1 tests passed.

see run history

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: ca4f8b0

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
observability	633	635	+2

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
alerting	91.6KB	91.6KB	-2.0B
apm	2.6MB	2.6MB	+128.0B
datasetQuality	418.9KB	419.0KB	+125.0B
discover	1.1MB	1.1MB	+128.0B
embeddableAlertsTable	940.9KB	940.9KB	-16.0B
infra	1.0MB	1.0MB	+143.0B
ml	5.3MB	5.3MB	+143.0B
monitoring	630.9KB	631.1KB	+125.0B
observability	1.3MB	1.3MB	+168.0B
securitySolution	9.8MB	9.8MB	-16.0B
slo	979.6KB	979.7KB	+113.0B
synthetics	1.0MB	1.0MB	+128.0B
transform	622.4KB	622.5KB	+125.0B
triggersActionsUi	1.5MB	1.5MB	+143.0B
uptime	491.5KB	491.6KB	+125.0B
total			+1.5KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
cases	135.2KB	135.2KB	-18.0B
infra	50.1KB	50.1KB	-15.0B
ml	85.8KB	85.8KB	-4.0B
observability	155.5KB	155.6KB	+83.0B
observabilityShared	67.1KB	67.2KB	+112.0B
triggersActionsUi	105.6KB	105.5KB	-15.0B
total			+143.0B

Unknown metric groups

API count

id	before	after	diff
observability	641	643	+2

History

• 💛 Build #324926 was flaky 9f0d7c1
• 💔 Build #324897 failed b83175f
• 💔 Build #324830 failed 8fb4d52
• 💔 Build #324502 failed d69d85b
• 💔 Build #324421 failed 33b9254
• 💔 Build #324137 failed 35e43fa