Filebeat loss of logs

[sirisha-29]

Please post all questions and issues on https://discuss.elastic.co/c/beats
before opening a Github Issue. Your questions will reach a wider audience there,
and if we confirm that there is a bug, then you can open a new issue.

For security vulnerabilities please only send reports to security@elastic.co.
See https://www.elastic.co/community/security for more information.

Please include configurations and logs if available.

For confirmed bugs, please report:

• Version: 8.17.8
• Operating System:
• Discuss Forum URL:
• Steps to Reproduce: Send 1500log/sec for 1hours and we observe that not all logs are harvested by filebeat. We expected log count to be 5400000 and the log reached configured output is 4852102. The configmap used is attached below,

Filebeat configmap:

`filebeat.yml:

filebeat.autodiscover:
providers:

• type: kubernetes
  namespace: "test"
  hints.enabled: false
  add_resource_metadata:
  deployment: false
  cronjob: false
  namespace:
  enabled: false
  node:
  enabled: false
  templates:
  • config:
    • type: container
      paths:
      • /var/log/pods/${data.kubernetes.container.id}/*-json.log
      • /var/log/pods/${data.kubernetes.namespace}${data.kubernetes.pod.name}${data.kubernetes.pod.uid}/${data.kubernetes.container.name}/*.log
        appenders:
  • type: config
    config:
    fields:
    logplane: "app-logs"
    fields_under_root: true
    close_timeout: "5m"
    processors:
    • decode_json_fields:
      fields: ["message"]
      target: "json"
      max_depth: 1
      output.logstash:
      hosts: "log-transformer:5044"
      ssl.certificate_authorities: "${TRUSTED_INTERNAL_ROOT_CA_PATH}/ca.crt"
      ssl.certificate: "${LT_CLIENT_CERT_PATH}/${CERT}"
      ssl.key: "${LT_CLIENT_CERT_PATH}/${KEY}"
      ssl.verification_mode: "full"
      ssl.renegotiation: "freely"
      ssl.supported_protocols: ["TLSv1.2", "TLSv1.3"]
      ssl.cipher_suites: []
      bulk_max_size: 2048
      worker: 1
      pipelining: 0
      ttl: 30
      queue.mem:
      flush.timeout: 1s
      filebeat.registry.flush: 5s
      logging.level: "info"
      logging.metrics.enabled: false
      http.enabled: true
      http.host: localhost
      http.port: 5066
      `

Filebeat stats:

curl http://localhost:5066/stats {"beat":{"cpu":{"system":{"ticks":42270,"time":{"ms":42270}},"total":{"ticks":746410,"time":{"ms":746410},"value":746410},"user":{"ticks":704140,"time":{"ms":704140}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":15},"info":{"ephemeral_id":"b0f5cf81-fe2c-4cd0-870e-7d21c6b60f28","name":"filebeat","uptime":{"ms":9052890},"version":"8.17.8"},"memstats":{"gc_next":13486418,"memory_alloc":8765360,"memory_sys":96723224,"memory_total":73510964736,"rss":104333312},"runtime":{"goroutines":105}},"filebeat":{"events":{"active":0,"added":4852102,"done":4852102},"harvester":{"closed":374,"open_files":2,"running":2,"skipped":0,"started":376},"input":{"log":{"files":{"renamed":0,"truncated":0}}}},"libbeat":{"config":{"module":{"running":13,"starts":19,"stops":6},"reloads":0,"scans":0},"output":{"batches":{"split":0},"events":{"acked":4851034,"active":0,"batches":3200,"dead_letter":0,"dropped":0,"duplicates":0,"failed":0,"toomany":0,"total":4851034},"read":{"bytes":562092,"errors":0},"type":"logstash","write":{"bytes":558237554,"errors":0,"latency":{"histogram":{"count":3200,"max":7804,"mean":1305.2724609375,"median":1231,"min":2,"p75":1476.5,"p95":1941,"p99":2433.75,"p999":7724.150000000072,"stddev":479.74560212390173}}}},"pipeline":{"clients":13,"events":{"active":0,"dropped":0,"failed":0,"filtered":1068,"published":4851034,"retry":210,"total":4852102},"queue":{"acked":4851034,"added":{"bytes":0,"events":4851034},"consumed":{"bytes":0,"events":4851034},"filled":{"bytes":0,"events":0,"pct":0},"max_bytes":0,"max_events":3200,"removed":{"bytes":0,"events":4851034}}}},"registrar":{"states":{"cleanup":306,"current":13,"update":4852102},"writes":{"fail":0,"success":847,"total":847}},"system":{"cpu":{"cores":8},"load":{"1":4.2,"15":4.27,"5":5.1,"norm":{"1":0.525,"15":0.5337,"5":0.6375}}}}

We are stepping up from 8.16.3 to 8.17.8, and in the 8.17.8 we are facing loss of logs at throughput of 1500log/sec which we didn't see in 8.16.3 and the above configmap is been used. Kindly look into this and give the way forward as soon as possible.

[botelastic]

This issue doesn't have a Team:<team> label.