[streams] Allow Beats run by Elastic Agent to route data to the "logs" index

[cmacknz]

• Relates [streams] Confirm setting the target index to "logs" routes data to streams #45505

Beats run by agent rigidly follow the datastream naming scheme of type-dataset-namespace. To allow routing data to the streams experience, users need to be able to set the index name to logs which does not follow this convention.

beats/x-pack/libbeat/management/generate.go

Lines 214 to 221 in 87f0e9c

	// injectIndexStream is an emulation of the InjectIndexProcessor AST code
	// this adds the `index` field, based on the data_stream info we get from the config
	func injectIndexStream(defaultDataStreamType string, expected *proto.UnitExpectedConfig, streamExpected *proto.Stream, stream map[string]interface{}) map[string]interface{} {
	streamType, dataset, namespace := metadataFromDatastreamValues(defaultDataStreamType, expected, streamExpected)
	index := fmt.Sprintf("%s-%s-%s", streamType, dataset, namespace)
	stream["index"] = index
	return stream
	}

The code block above shows that an input level index field is always generated and will default to logs-default-generic when the datastream fields are missing. This will prevent input level index settings from working, though depending on precedence setting the index via a processor or in the output may still work.

Confirm that the 3 ways for setting the index parameter for standalone Beats described in #45505 all work consistently, and make any code change necessary to have them work.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)