Temporarily import hard-coded list of common fields

[jsoriano]

This issue is a temporary workaround for elastic/package-spec#441 and #1017, discard it if the other issues are prioritized first.

For packages importing fields from ECS, import a hard-coded list of fields that are commonly used and can be a source of conflicts if they don't have any mapping.

Update: instead of adding specific fields, include dynamic mappings as described in #1018 (comment). We have to try how to add the dynamic mappings:

• If we add these dynamic mappings using fields.yml, this would only be supported by versions of kibana implementing [Fleet] Support dynamic_template mappings from object field kibana#137772 (>= 8.4.0).
• If we add the dynamic mappings in the manifest, we will cover more versions of the stack, but we have to be careful of merging correctly with other existing configurations, and it may be difficult to check for documented fields in tests.
  • Add support for elasticsearch.index_template key in packages and a new flag to opt in Add elasticsearch mappings definitions into input packages and a key to opt-in to import common definitions package-spec#455
  • Import templates in elastic-package Add common dynamic mappings and properties from ECS automatically #1073
  • Add imported templates as definitions to be used in tests Add ECS mappings during test runner executions #1090

Fields imported this way should be imported in a new well-known fields file that can be discarded later by Fleet if elastic/kibana#144048 is implemented. For example they could be included in a file called fields/imported_elastic_package.yml.

Fixes elastic/integrations#4236.

[P1llus]

Don't think I forgot anything here, but feel free to double check @andrewkroh @jsoriano .
We also have a few processors adding non-ecs fields, a bit unsure what we want to do with those, but added them to the list.

Agent:

- external: ecs
  name: agent.name
- external: ecs
  name: agent.type
- external: ecs
  name: agent.id
- external: ecs
  name: agent.ephemeral_id
- external: ecs
  name: agent.version

Host:

- external: ecs
  name: host.architecture
- external: ecs
  name: host.domain
- external: ecs
  name: host.hostname
- external: ecs
  name: host.id
- external: ecs
  name: host.ip
- external: ecs
  name: host.mac
- external: ecs
  name: host.name
- external: ecs
  name: host.os.family
- external: ecs
  name: host.os.kernel
- external: ecs
  name: host.os.name
- external: ecs
  name: host.os.platform
- external: ecs
  name: host.os.version
- external: ecs
  name: host.os.build
- external: ecs
  name: host.os.codename
- external: ecs
  name: host.os.type
- external: ecs
  name: host.text
- external: ecs
  name: host.type
- external: ecs
  name: host.containerized

Geo:

- external: ecs
  name: geo.continent_name
- external: ecs
  name: geo.country_iso_code
- external: ecs
  name: geo.region_name
- external: ecs
  name: geo.region_iso_code
- external: ecs
  name: geo.city_name
- external: ecs
  name: geo.name
- external: ecs
  name: geo.location

Cloud:

- external: ecs
  name: cloud.account.id
- external: ecs
  name: cloud.availability_zone
- external: ecs
  name: cloud.instance.id
- external: ecs
  name: cloud.instance.name
- external: ecs
  name: cloud.machine.type
- external: ecs
  name: cloud.provider
- external: ecs
  name: cloud.region
- external: ecs
  name: cloud.project.id
- external: ecs
  name: cloud.image.id

Docker (NON ECS FIELDS):

    - name: docker
      default_field: true
      type: group
      fields:
        - name: container.id
          type: alias
          path: container.id
          migration: true

        - name: container.image
          type: alias
          path: container.image.name
          migration: true

        - name: container.name
          type: alias
          path: container.name
          migration: true

Kubernetes (NON ECS FIELDS):

    - name: kubernetes
      default_field: true
      type: group
      fields:
        - name: pod.name
          type: keyword
          description: >
            Kubernetes pod name

        - name: pod.uid
          type: keyword
          description: >
            Kubernetes Pod UID

        - name: pod.ip
          type: ip
          description: >
            Kubernetes Pod IP

        - name: namespace
          type: keyword
          description: >
            Kubernetes namespace

        - name: node.name
          type: keyword
          description: >
            Kubernetes node name

        - name: node.hostname
          type: keyword
          description: >
            Kubernetes hostname as reported by the node’s kernel

        - name: labels.*
          type: object
          object_type: keyword
          object_type_mapping_type: "*"
          description: >
            Kubernetes labels map

        - name: annotations.*
          type: object
          object_type: keyword
          object_type_mapping_type: "*"
          description: >
            Kubernetes annotations map

        - name: selectors.*
          type: object
          object_type: keyword
          object_type_mapping_type: "*"
          description: >
            Kubernetes selectors map

        - name: replicaset.name
          type: keyword
          description: >
            Kubernetes replicaset name

        - name: deployment.name
          type: keyword
          description: >
            Kubernetes deployment name

        - name: statefulset.name
          type: keyword
          description: >
            Kubernetes statefulset name

        - name: container.name
          type: keyword
          description: >
            Kubernetes container name (different than the name from the runtime)

Process:

    - name: process
      default_field: true
      type: group
      fields:
        - name: exe
          type: alias
          path: process.executable
          migration: true
        - name: owner
          type: group
          description: Process owner information.
          fields:
            - name: id
              type: keyword
              ignore_above: 1024
              description: Unique identifier of the user.
            - name: name
              type: keyword
              ignore_above: 1024
              multi_fields:
              - name: text
                type: text
                norms: false
              description: Short name or login of the user.
              example: albert

[andrewkroh]

What produces those "Geo" fields? ^geo.* are not valid ECS fields as per https://www.elastic.co/guide/en/ecs/current/ecs-geo.html#_field_reuse_10.

In general I think it would be good to call out specifically what produces each set of fields. For example, say they are produced by add_kubernetes_metadata and link to its documentation.

IIRC there are also some elastic_agent.* fields that are injected into each input's config by the Agent.

[jsoriano]

@P1llus thanks for collecting the list of fields, I think this will be useful for elastic/package-spec#441.

For the matter of this workaround I was thinking on a reduced list of fields, whose lack of mappings is currently causing known issues. I wouldn't like to hard-code many fields here. If we cannot benefit of a small list of hard-coded fields, I would wait to have elastic/package-spec#441 and #1017.

Would you have a list of fields currently causing issues?

[P1llus]

@jsoriano the issue is mostly regarding any ECS field that processors add, because it will conflict with fields installed by other integrations.
The best possible workaround right now would simply be to update the custom input packages field definition with these ECS fields, that way we wont have to wait for the other more permanent fixes.

WDYT @jsoriano @andrewkroh ?

[jsoriano]

the issue is mostly regarding any ECS field that processors add, because it will conflict with fields installed by other integrations.

Yes, but I wouldn't like to hard-code it in packages or in elastic-package, this is information that is not actually related to integrations. As mentioned, I would accept to hard-code a reduced set of fields if this is a low-hanging fruit to solve actual fixes that we have now.

The best possible workaround right now would simply be to update the custom input packages field definition with these ECS fields, that way we wont have to wait for the other more permanent fixes.

Why input packages and not all packages? Documents of any input or integration can be enriched with processors.

Hard-coding many fields has some problems:

• We have to decide what to do with all the duplicated fields that will appear now (Mario already faced this in Import hard-coded list of common fields #1025).
• When we have a more maintainable long-term solution, we will have to revert all these hard-codings and it may be difficult to track all the implications if many fields were added.

[jsoriano]

We have been discussing about this, and we are going to include a set of hard-coded mappings that will include all ECS definitions. Longer term these mappings will be included by Fleet or Elasticsearch (see elastic/elasticsearch#85692), but for packages supporting current and old versions of the stack, we will continue using the method implemented here.

We have to create a POC implementation in elastic-package, and decide if we need a change in the package-spec for this. We want this to be opt-in per package, and these dynamic mappings would be included in build time, so we probably need to add a setting under _dev/build.

Dynamic template to include can be found in https://gist.github.com/P1llus/e0de7b3a7824a41a29660e253c6cce6b

Thanks @P1llus!

[mrodm]

@P1llus we've been checking this issue in depth and it looks like that for input packages is not going to work. This issue is pending for input packages elastic/kibana#145529

Would that be a problem ? Is it ok to just make it working for integration packages ?

[P1llus]

@P1llus we've been checking this issue in depth and it looks like that for input packages is not going to work. This issue is pending for input packages elastic/kibana#145529

Would that be a problem ? Is it ok to just make it working for integration packages ?

@mrodm are you talking about dynamic templates or? The input packages we already have seems to not have any problems installing things like index mappings etc?