[connector/elasticapm] Configure overflow limits

[lahsivjar]

Overflow limits should be configured to create an upper bound for the aggregation resource usage.

[isaacaflores2]

@lahsivjar I might need to more context for this task. My understanding so far is we need to expose configs so the connector can propagate them to the lsmintervalprocessor. The limit configs I see are listed here. Am I on the right track?

[isaacaflores2]

I reviewed the code a bit more (assuming I am on the right track). Do we need to provide the ability to configure each limit for resource, scope, metric, and datapoint? So an example full config for the connector would look like:

elasticapm:
  aggregation:
    directory: /path/to/aggregation/state
    metadata_keys: [a, B, c]
    intervals: [1s, 1m]
    limits:
      resource:
        max_cardinality: 1
        overflow:
          attributes:
            - key: "k1"
              value: "v1"
            - key: "k2"
              value: "v2"
      scope:
        max_cardinality: 1
        overflow:
          attributes:
            - key: "k1"
              value: "v1"
            - key: "k2"
              value: "v2"
      metric:
        max_cardinality: 1
        overflow:
          attributes:
            - key: "k1"
              value: "v1"
            - key: "k2"
              value: "v2"
      datapoint:
        max_cardinality: 1
        overflow:
          attributes:
            - key: "k1"
              value: "v1"
            - key: "k2"
              value: "v2"

[lahsivjar]

@isaacaflores2 You are on the right track. For the values, let’s take the limits from the apm-aggregation setup available for MIS.

[isaacaflores2]

@lahsivjar @vigneshshanmugam Once these changes are merged, should we define the limits in the motel-aggregation-collector?

Since the aggregation collector is multi-tenant I guess these limits would apply to multiple projects 🤔 So maybe it would be worth testing with the current aggregation collector resources configuration to determine a new set of overflow limits?

[lahsivjar]

Sorry, your PR is in my review list but I am not able to get to it. Will try to prioritise it this week.

Once these changes are merged, should we define the limits in the motel-aggregation-collector?

Yes we should.

Since the aggregation collector is multi-tenant I guess these limits would apply to multiple projects

Yes

So maybe it would be worth testing with the current aggregation collector resources configuration to determine a new set of overflow limits?

There is also a product consideration here, we don't want to go too low in the limits. I think for now we should keep the APM limits and revisit this as we get more datapoints. CC: @axw @simitt @vigneshshanmugam (if you have anything to add here).

[simitt]

Sorry, missed this ping.

I think for now we should keep the APM limits and revisit this as we get more datapoints.

Do you mean the MIS limits? (I believe in APM we increase them with increasing deployment size).