Documentation Grafana Mimir Configure Tracing

Configure Grafana Mimir tracing

Distributed tracing is a valuable tool for troubleshooting the behavior of Grafana Mimir in production.

Grafana Mimir supports distributed tracing using OpenTelemetry and provides backward compatibility with Jaeger configuration.

Mimir automatically detects which tracing configuration method to use based on the environment variables you set:

• OpenTelemetry format: Uses standard OTel environment variables for modern tracing backends.
• Jaeger format: Uses Jaeger-specific environment variables for backward compatibility.

Dependencies

Set up a tracing backend to collect and store traces from Grafana Mimir:

• For OpenTelemetry: Use any OTLP-compatible tracing backend such as Jaeger, Tempo, or cloud-based solutions.
• For Jaeger: Set up a Jaeger deployment including either the Jaeger all-in-one binary or a distributed system of agents, collectors, and queriers.

If you run Grafana Mimir on Kubernetes, refer to Jaeger Kubernetes.

Configuration

Grafana Mimir automatically detects the tracing configuration method based on the environment variables you set. You can use either OpenTelemetry standard environment variables or Jaeger-specific environment variables.

OpenTelemetry Configuration (Recommended)

To configure Grafana Mimir with OpenTelemetry format tracing, set any of the following environment variables.

OpenTelemetry configuration follows the standard documentation from OpenTelemetry SDK Configuration.

• OTEL_EXPORTER_OTLP_ENDPOINT: The OTLP endpoint URL. For example, http://otlp-endpoint:4318.
• OTEL_EXPORTER_OTLP_TRACES_ENDPOINT: The OTLP traces-specific endpoint URL. This value overrides the previous variable. For example, http://tempo:4318/v1/traces.
• OTEL_TRACES_EXPORTER: The traces exporter to use. Default: otlp.
• OTEL_TRACES_SAMPLER: The sampling strategy to use.

Traces exporter configuration

The OTEL_TRACES_EXPORTER environment variable specifies which trace exporter to use:

• otlp (default): OpenTelemetry Protocol exporter
• none: No trace exporter

Sampling configuration

Configure sampling behavior using the OTEL_TRACES_SAMPLER environment variable:

• always_on: Sample all traces
• always_off: Sample no traces
• traceidratio: Sample traces based on trace ID ratio
• parentbased_always_on (default): Sample based on parent span decision. Always sample root spans.
• parentbased_always_off: Sample based on parent span decision. Never sample root spans.
• parentbased_traceidratio: Sample based on parent span decision, use ratio for root spans
• jaeger_remote: Use Jaeger remote sampling
• parentbased_jaeger_remote: Use parent-based Jaeger remote sampling

For jaeger_remote and parentbased_jaeger_remote samplers, you must also set OTEL_TRACES_SAMPLER_ARG with the configuration in the format:

Copy

endpoint=http://localhost:14250,pollingIntervalMs=5000,initialSamplingRate=0.25

Propagation configuration

Configure trace context propagation using OTEL_PROPAGATORS:

• tracecontext: W3C Trace Context (default)
• baggage: W3C Baggage (default)
• jaeger: Jaeger propagation (default)
• none: No propagation

Default propagators include tracecontext, baggage, and jaeger.

Example OpenTelemetry configuration

bash Copy

# Basic OTLP configuration
export OTEL_EXPORTER_OTLP_ENDPOINT="http://jaeger:4318"
export OTEL_TRACES_SAMPLER="always_on"

# Advanced configuration with Jaeger remote sampling
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="http://jaeger:4318/v1/traces"
export OTEL_TRACES_SAMPLER="jaeger_remote"
export OTEL_TRACES_SAMPLER_ARG="endpoint=http://jaeger:14250,pollingIntervalMs=5000,initialSamplingRate=0.25"
export OTEL_PROPAGATORS="tracecontext,baggage,jaeger"

Jaeger configuration (deprecated)

To configure Grafana Mimir with Jaeger format tracing, set any of the following environment variables:

• JAEGER_AGENT_HOST: Jaeger agent hostname
• JAEGER_ENDPOINT: Jaeger collector endpoint URL
• JAEGER_SAMPLER_MANAGER_HOST_PORT: Jaeger sampler manager endpoint

Required environment variables

You must specify at least one of the following environment variables to enable Jaeger configuration:

• JAEGER_AGENT_HOST
• JAEGER_ENDPOINT
• JAEGER_SAMPLER_MANAGER_HOST_PORT

Sampling configuration

Configure sampling using Jaeger-specific environment variables:

• JAEGER_SAMPLER_TYPE: Sampling strategy (const, probabilistic, remote)
• JAEGER_SAMPLER_PARAM: Sampling parameter value
• JAEGER_SAMPLER_MANAGER_HOST_PORT: Remote sampling endpoint
• JAEGER_SAMPLING_ENDPOINT: Alternative remote sampling endpoint

Additional Jaeger environment variables

• JAEGER_AGENT_PORT: Jaeger agent port (default: 6831)
• JAEGER_TAGS: Additional tags to add to spans (format: key1=value1,key2=value2)

Example Jaeger configuration

bash Copy

# Basic agent configuration
export JAEGER_AGENT_HOST="jaeger-agent"
export JAEGER_SAMPLER_TYPE="const"
export JAEGER_SAMPLER_PARAM="1"

# Collector endpoint configuration
export JAEGER_ENDPOINT="http://jaeger-collector:14268/api/traces"
export JAEGER_SAMPLER_TYPE="probabilistic"
export JAEGER_SAMPLER_PARAM="0.1"

# Remote sampling configuration
export JAEGER_AGENT_HOST="jaeger-agent"
export JAEGER_SAMPLER_MANAGER_HOST_PORT="jaeger-agent:5778"

For a complete list of Jaeger environment variables, refer to the Jaeger Client Go documentation.

Enabling sampling

Enable sampling in the appropriate components:

• The ingester and ruler self-initiate traces. You should explicitly enable sampling for these components.
• You can enable sampling for the distributor and query-frontend in Grafana Mimir or in an upstream service, like a proxy or gateway running in front of Grafana Mimir