Persisting Data in Jaeger with Badger Storage

[devbaran]

Issue: Persisting Data in Jaeger with Badger Storage

Environment

Using Docker Compose to run Jaeger with Badger storage:

services:
  conversation-backend-jaeger-dev:
    container_name: c_conversation_jaeger_dev
    image: jaegertracing/jaeger:latest
    restart: always
    ports:
      - "16686:16686"  # Jaeger UI
      - "4317:4317"    # OTLP gRPC receiver
      - "4318:4318"    # OTLP HTTP receiver
    
    environment:
      - SPAN_STORAGE_TYPE=badger       # Set storage type to Badger
      - BADGER_DIRECTORY=/badger      # Badger path inside container
      - BADGER_EPHEMERAL=false        # Ensure data persistence

    volumes:
      - conversation_backend_jaeger_dev-volume:/badger

volumes:
  conversation_backend_jaeger_dev-volume:
    name: conversation_backend_jaeger_dev-volume
    driver: local

---

Expected Behavior

The Badger storage should persist tracing data across container restarts.

---

Actual Behavior

When running with the configuration above, logs show:

2025/08/27 09:47:24 application version: git-commit=57f65148781af756f323ec27d8d080b20ff0bde1, git-version=v2.9.0, build-date=2025-08-06T18:10:13Z
2025/08/27 09:47:24 No '--config' flags detected, using default All-in-One configuration with memory storage.
2025/08/27 09:47:24 To customize All-in-One behavior, pass a proper configuration.

This indicates that the storage is not using Badger as expected and defaults to in-memory storage.

---

Attempted Fix 1: Using Command Flags

Updated Docker Compose to use command flags:

services:
  conversation-backend-jaeger-dev:
    container_name: c_conversation_jaeger_dev
    image: jaegertracing/jaeger:latest
    restart: always
    ports:
      - "16686:16686"
      - "4317:4317"
      - "4318:4318"

    command: [
      "--storage.type=badger",
      "--storage.badger.ephemeral=false",
      "--storage.badger.directory-key=/badger/key",
      "--storage.badger.directory-value=/badger/data"
    ]
    
    volumes:
      - conversation_backend_jaeger_dev-volume:/badger

Result:

c_conversation_jaeger_dev exited with code 1
c_conversation_jaeger_dev  | Error: unknown flag: --storage.type

The jaegertracing/jaeger:latest image does not recognize --storage.type flags, causing the container to crash.

---

Attempted Fix 2: Using All-in-One Image

Tried switching to all-in-one image:

services:
  conversation-backend-jaeger-dev:
    container_name: c_conversation_jaeger_dev
    image: jaegertracing/all-in-one:latest
    restart: always
    ports:
      - "16686:16686"
      - "4317:4317"
      - "4318:4318"
    environment:
      - SPAN_STORAGE_TYPE=badger
      - BADGER_DIRECTORY=/badger
      - BADGER_EPHEMERAL=false
    volumes:
      - conversation_backend_jaeger_dev-volume:/badger

Error:

c_conversation_jaeger_dev  | {"level":"fatal","ts":1756288149.459645,"caller":"all-in-one/main.go:107","msg":"Failed to init storage factory","error":"Error Creating Dir: \"/go/bin/data/keys\" err: mkdir /go/bin/data: permission denied", ...}

This indicates a permission issue when trying to create directories inside the container.

---

Problem Summary

1. The jaegertracing/jaeger image does not support the --storage.type flag and crashes when used.
2. The all-in-one image with environment variables results in permission denied when Badger tries to create directories.
3. As a result, persistent storage with Badger is not working.

---

Question

How can I properly configure Jaeger with Badger storage to persist tracing data across container restarts without encountering permission issues or unsupported flags?

[dosubot[bot]]

Hi @devbaran! I'm Dosu and I’m helping the jaeger team.

To persist Jaeger data with Badger storage in Docker Compose, use the jaegertracing/all-in-one image and configure storage via environment variables, not CLI flags—CLI flags are no longer supported in Jaeger 2.x+ and will cause errors. Your environment variables should include SPAN_STORAGE_TYPE=badger, BADGER_DIRECTORY=/badger, and BADGER_EPHEMERAL=false. Mount a Docker volume to /badger to persist data.

The "permission denied" error happens because the container user cannot write to the mounted directory. To fix this, ensure the host directory or Docker volume you mount to /badger is owned by the same UID/GID as the container user, or is world-writable. For example, you can run sudo chown -R 1000:1000 /path/to/your/host/volume (replace 1000 with the container's user ID if different) or use chmod 777 for testing (not recommended for production) before starting the container. This is a common Docker operational issue, not a Jaeger config bug source.

If you want to use a config file instead of environment variables (required for some advanced options in Jaeger 2.x+), mount your config file into the container and set the --config-file flag in your Docker Compose command. The config file should specify Badger storage with the correct directories and ephemeral: false.

Example Docker Compose snippet for persistent Badger storage:

services:
  jaeger:
    image: jaegertracing/all-in-one:latest
    environment:
      - SPAN_STORAGE_TYPE=badger
      - BADGER_DIRECTORY=/badger
      - BADGER_EPHEMERAL=false
    volumes:
      - jaeger-badger-data:/badger
    ports:
      - "16686:16686"
      - "4317:4317"
      - "4318:4318"

volumes:
  jaeger-badger-data:
    driver: local

Before starting, ensure the volume directory is writable by the container user to avoid permission errors. This setup will persist tracing data across restarts as long as the volume is not deleted.

If you need to fine-tune Badger (e.g., set TTL), use a config file and mount it, then reference it with command: ["--config-file=/etc/jaeger/config.yaml"] and mount your config to /etc/jaeger/config.yaml source.

If you still see permission errors, double-check the volume's permissions and ownership on the host.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}    

[devbaran]

@yurishkuro Without env vars or CLI flags, Jaeger prints:
No '--config' flags detected, using default All-in-One configuration with memory storage.

This suggests that some configuration (via CLI flags or env vars) is required in order to use storage backends.
Here is a Docker Compose example without any CLI flags or env vars:

services:
  conversation-backend-jaeger-dev:
    container_name: c_conversation_jaeger_dev
    image: jaegertracing/jaeger:latest
    restart: always
    ports:
      - "16686:16686"  # Jaeger UI
      - "4317:4317"    # OTLP gRPC receiver
      - "4318:4318"    # OTLP HTTP receiver
  
    volumes:
      - conversation_backend_jaeger_dev-volume:/badger

volumes:
  conversation_backend_jaeger_dev-volume:
    name: conversation_backend_jaeger_dev-volume
    driver: local

Is my understanding correct that Jaeger v2 requires explicit configuration through CLI flags or env vars to enable Badger storage?

[yurishkuro]

You pass a config file via --config cli option

[devbaran]

Thanks for the suggestion, but we already tried passing a config file via the --config CLI option as shown in Attempted Fix 1. Have you had a chance to try it yourself?

[yurishkuro]

Attempted Fix 1 does not use --config file.yaml, it tries to pass non-existing CLI flags that were only valid for Jaeger-v1

jaeger --config config.yaml

[devbaran]

Okay, I created a config YAML file like this, but I’m still getting an exception:

I added and edited this config based on the documentation link you provided, but I’m wondering if there’s an error somewhere.

Here’s my config:

service:
  extensions: [jaeger_storage, jaeger_query, healthcheckv2]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [jaeger_storage_exporter]

extensions:
  healthcheckv2:
    use_v2: true
    http:

  jaeger_query:
    storage:
      traces: main_store
    ui:
      config_file: ./cmd/jaeger/config-ui.json

  jaeger_storage:
    backends:
      main_store:
        badger:
          directories:
            keys: "/tmp/jaeger/"
            values: "/tmp/jaeger/"
          ephemeral: false
          ttl:
            spans: 720h # 2 days

receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  batch:

exporters:
  jaeger_storage_exporter:
    trace_storage: main_store