Distributed Tracing #

In traditional monolithic architecture, tracking request latency bottlenecks is relatively easy because the entire code execution runs inside one memory process on a single host machine. We just attach a code profiler or read chronological logs to find which function is running slowly. However, in microservices architectures running on Kubernetes, one user request can cross a dozen different services, moving from the API Gateway to gRPC authentication, triggering asynchronous processing in message brokers (Kafka/RabbitMQ), and finally touching various distributed databases. Without a Distributed Tracing system, diagnosing why a request runs slowly becomes nearly impossible. We need a unified tracking mechanism mapping the request’s entire journey end-to-end across container network process boundaries.

Basic Concepts: Trace and Span Anatomy #

Distributed tracing works by inserting unique identifiers into every request flow. There are two fundamental terms we must understand:

  • Trace: The end-to-end representation of one complete request’s journey from start to finish. One trace is identified by one unique Trace ID shared across all services.
  • Span: The smallest work unit representing one specific step or operation inside a trace (e.g. a SQL query, an HTTP client call, or token validation). Every span has a name, start/end timestamps, attributes (metadata), and a unique Span ID.

Sequence Diagram and Context Propagation #

To unify spans from various application containers into one complete trace timeline, we must forward trace metadata across network boundaries. This metadata forwarding process is called Context Propagation.

sequenceDiagram
    autonumber
    actor User as Client (Web Browser)
    participant Gateway as API Gateway (Ingress)
    participant Auth as Auth-Service (gRPC)
    participant Order as Order-Service (HTTP)
    participant DB as Postgres Database
    
    User->>Gateway: HTTP GET /api/orders (No Trace ID)
    Note over Gateway: "Initialize Trace:<br/>Trace ID: 4bf92f3577b34da6<br/>Parent Span ID: 00f067aa0ba902b7"
    
    Gateway->>Auth: gRPC validateToken() + W3C Metadata
    Note over Auth: "Extract gRPC Context<br/>Child Span 1.1"
    Auth-->>Gateway: HTTP 200 (Token Valid)
    
    Gateway->>Order: HTTP GET /order/details + traceparent Header
    Note over Order: "Extract HTTP Header<br/>Child Span 1.2"
    
    Order->>DB: SQL SELECT * FROM orders
    Note over DB: "Database Span (Child Span 1.2.1)"
    DB-->>Order: Return the Query Data
    
    Order-->>Gateway: Return the Order Detail Payload
    Gateway-->>User: HTTP 200 (Success)

The most commonly used contextual propagation standard today is W3C Trace Context. This standard defines the traceparent HTTP header with the following format:

W3C traceparent Header Format:
00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
│  │                                └─ Span ID (8-byte hex)
│  └─ Trace ID (16-byte hex)
└─ Specification version (00)

Application Instrumentation Using OpenTelemetry #

OpenTelemetry (OTel) is an open standard under the Cloud Native Computing Foundation (CNCF) providing uniform SDKs and APIs for collecting telemetry data (traces, metrics, logs). With OTel, we instrument the code once, and can send that data to any visualization backend (like Jaeger, Grafana Tempo, or Datadog) without modifying application code.

Here’s a comparison between asynchronous handling that breaks the trace context and safe context propagation implementation in Go:

// ANTI-PATTERN: Running an async goroutine without forwarding the context.
// This breaks the Trace ID, so the goroutine's internal operations aren't recorded in the parent trace.
func HandleOrder(w http.ResponseWriter, r *http.Request) {
    ctx := r.Context()
    go func() {
        // DON'T: The goroutine loses the parent context
        db.Query("SELECT * FROM inventory") // Exec query runs without a Trace ID
    }()
}

// ==============================================================================
// CORRECT: Forwarding the Context to Async Goroutines to Keep the Trace Flow.
func HandleOrder(w http.ResponseWriter, r *http.Request) {
    ctx := r.Context()
    
    // Pull the current active span from the request context
    parentSpan := trace.SpanFromContext(ctx)
    
    // Safely copy the trace context to the async goroutine
    go func(asyncCtx context.Context) {
        // Create a new span inside the async goroutine with the correct parent span
        tracer := otel.Tracer("order-service")
        childCtx, span := tracer.Start(asyncCtx, "async_inventory_check")
        defer span.End()
        
        // The database operation is now bound to the same Trace ID
        db.QueryContext(childCtx, "SELECT * FROM inventory")
    }(trace.ContextWithSpan(context.Background(), parentSpan))
}

Python Instrumentation Code Example (FastAPI & OTel SDK) #

Here’s a complete example of initializing the OTel Tracer Provider and automatically exporting spans via the OTLP/gRPC protocol in Python:

import time
from fastapi import FastAPI
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor

# 1. Initialize the Tracing SDK Provider
provider = TracerProvider()

# 2. Send trace data to the local OTel Collector via gRPC port 4317
otlp_exporter = OTLPSpanExporter(endpoint="http://otel-collector.monitoring.svc:4317", insecure=True)

# 3. Use BatchSpanProcessor for performance efficiency (don't send spans one by one synchronously)
provider.add_span_processor(BatchSpanProcessor(otlp_exporter))
trace.set_tracer_provider(provider)

# 4. Initialize the Tracer for manual instrumentation
tracer = trace.get_tracer("order-service")

app = FastAPI()

@app.get("/process-order/{order_id}")
def process_order(order_id: str):
    # Create a manual span for critical business logic
    with tracer.start_as_current_span("calculate_payment") as span:
        span.set_attribute("order.id", order_id)
        time.sleep(0.5) # Simulate delay
        span.set_attribute("payment.status", "approved")
    return {"status": "success"}

# 5. Install FastAPI auto-instrumentation
FastAPIInstrumentor.instrument_app(app)

OpenTelemetry Collector Architecture and Deployment #

The OpenTelemetry Collector is an intermediary component (proxy agent) tasked with receiving, processing, filtering, and exporting telemetry data. We must not send trace data directly from applications to storage backends (like Jaeger/Tempo) because:

  • Applications get burdened with log sending queue and retry logic.
  • It’s hard to do centralized filtering or metadata enrichment.

In Kubernetes, the OTel Collector is deployed as a DaemonSet on every worker node (for low-latency local processing) or as a centralized Deployment (for mass aggregation).

OTel Collector Deployment Manifest #

apiVersion: apps/v1
kind: Deployment
metadata:
  name: otel-collector
  namespace: monitoring
spec:
  replicas: 2
  selector:
    matchLabels:
      app: otel-collector
  template:
    metadata:
      labels:
        app: otel-collector
    spec:
      containers:
      - name: otel-collector
        image: otel/opentelemetry-collector-contrib:0.95.0
        ports:
        - containerPort: 4317 # OTLP/gRPC receiver port
        - containerPort: 4318 # OTLP/HTTP receiver port
        volumeMounts:
        - name: config-volume
          mountPath: /etc/otelcol
        resources:
          limits:
            cpu: "500m"
            memory: "1Gi"
          requests:
            cpu: "200m"
            memory: "512Mi"
      volumes:
      - name: config-volume
        configMap:
          name: otel-collector-config

OTel Collector Pipeline Configuration (ConfigMap) #

The OTel Collector configuration uses a pipeline structure defined in three main parts: receivers (receivers), processors (processors/filters), and exporters (senders).

apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector-config
  namespace: monitoring
data:
  otel-collector-config.yaml: |
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318

    processors:
      # 1. Limit the collector memory so it doesn't trigger OOMKilled in Kubernetes
      memory_limiter:
        check_interval: 1s
        limit_percentage: 80
        spike_limit_percentage: 20

      # 2. Combine spans into batches for network efficiency
      batch:
        send_batch_size: 1024
        timeout: 5s

      # 3. Insert cluster metadata into every span
      resourcedetection:
        detectors: [env, system]

    exporters:
      # Send traces to the Grafana Tempo backend
      otlp:
        endpoint: "tempo-distributor.monitoring.svc.cluster.local:4317"
        tls:
          insecure: true

    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [memory_limiter, batch, resourcedetection]
          exporters: [otlp]

Sampling Strategies #

In high-traffic systems (e.g. processing 20,000 requests per second), recording 100% of trace data burdens the cluster’s internal network bandwidth and requires very expensive storage capacity. Therefore, we must apply Sampling techniques.

1. Head-Based Sampling #

The decision to record or ignore a trace is taken at the start of the request cycle (usually at the API Gateway or client SDK).

  • Advantages: Very CPU and bandwidth efficient because unselected traces are immediately ignored upstream.
  • Disadvantages: We can randomly lose valuable data. If an error occurs mid-request, that request might not be recorded because it didn’t pass sampling at the start.
# Head-Based Sampler Configuration in the SDK: Save only 10% of requests randomly
from opentelemetry.sdk.trace.sampling import ParentBased, TraceIdRatioBased
sampler = ParentBased(root=TraceIdRatioBased(0.10))

2. Tail-Based Sampling #

The decision is taken after the request is fully processed. The OTel Collector first gathers all spans of one trace in a memory buffer, analyzes the results, then makes the decision.

  • Advantages: Very precise. We can create policies like: “Always save 100% of traces producing 5xx error statuses or taking > 2 seconds of latency, and only save 1% of successful requests.”
  • Disadvantages: Requires fairly large RAM memory allocation on the OTel Collector because it must hold temporary trace data before deciding.
# Add the tail_sampling configuration to the processors in the OTel Collector
processors:
  tail_sampling:
    decision_wait: 10s # Wait 10 seconds for all spans to collect
    num_traces: 10000
    expected_new_traces_per_sec: 2000
    policies:
      # Rule A: Always save if the span status is ERROR
      - name: error-policy
        type: status_code
        status_code: {status_codes: [ERROR]}
      # Rule B: Always save if the request duration > 1.5 seconds
      - name: latency-policy
        type: latency
        latency: {threshold_ms: 1500}
      # Rule C: Save 5% of the remaining normal requests randomly
      - name: probabilistic-policy
        type: probabilistic
        probabilistic: {sampling_percentage: 5.0}

Tracing Integration with Message Brokers (Kafka/RabbitMQ) #

Another common mistake is the Trace ID breaking when requests enter message broker queue systems (message queues). Distributed tracing across message brokers requires injecting context into the message header metadata.

Here’s an example of injecting trace context into RabbitMQ (AMQP) messages using Go:

// Sending RabbitMQ messages with Trace Context
func PublishOrderEvent(ctx context.Context, ch *amqp.Channel, event OrderEvent) {
    // Create a special span for the event publishing activity
    tracer := otel.Tracer("order-service")
    ctx, span := tracer.Start(ctx, "publish_order_event")
    defer span.End()

    // The RabbitMQ header container map
    headers := amqp.Table{}

    // Inject the traceparent context from context.Context into the headers map
    otel.GetTextMapPropagator().Inject(ctx, propagation.MapCarrier(headers))

    body, _ := json.Marshal(event)

    // Send the message with headers enriched with the Trace ID
    ch.Publish(
        "",
        "order-queue",
        false,
        false,
        amqp.Publishing{
            Headers:     headers, // Must include the propagation header
            ContentType: "application/json",
            Body:        body,
        },
    )
}

On the consumer side, the receiving application must read those headers and extract them using otel.GetTextMapPropagator().Extract(ctx, propagation.MapCarrier(headers)) before starting new processing so the Trace ID stays linearly connected.

Distributed Tracing Practice Anti-Patterns #

Avoid the following bad habits when designing cluster tracing:

1. Manual Span Pollution #

Creating manual spans for every small internal function (like string formatting functions or mathematical multiplications). This floods application memory, produces very dense unreadable trace graphs in Jaeger, and needlessly burdens CPU. Limit manual spans to only network operations, database queries, or critical business logic flows.

2. Letting Silent Exceptions Go Unmarked #

Catching errors in business code using try-catch blocks but not registering that error status into the active span. As a result, the trace visualization on dashboards still looks green (success), making it hard for triage teams to detect code failure locations.

# ANTI-PATTERN: Catching errors without marking the span status.
with tracer.start_as_current_span("process_payment") as span:
    try:
        charge_credit_card()
    except Exception as e:
        # DON'T: Tracing will consider this transaction successful (green)
        handle_error_gracefully(e)

# ==============================================================================
# CORRECT: Register the exception and set the span status to ERROR.
with tracer.start_as_current_span("process_payment") as span:
    try:
        charge_credit_card()
    except Exception as e:
        span.set_status(trace.StatusCode.ERROR, str(e))
        span.record_exception(e) # Record the error stacktrace in the span metadata
        raise e

Distributed Tracing Implementation Checklist #

Make sure all the following check items are fulfilled before launching the distributed tracing system to production:

CODE INSTRUMENTATION & PROPAGATION:
  □ Applications are instrumented using the OpenTelemetry API & SDK standard.
  □ Autoinstrumentation is applied for HTTP frameworks (FastAPI, Gin, Express) and DB drivers (SQLAlchemy, Gorm).
  □ All goroutines/async threads safely forward the parent context (no lost context).
  □ The W3C 'traceparent' header is inserted on every outgoing HTTP client call.
  □ Context credentials are inserted into message broker record headers (Kafka/RabbitMQ).
  □ All error handling blocks register exceptions and mark the span status as ERROR.

OTEL COLLECTOR & BACKEND CONFIGURATION:
  □ The OTel Collector is deployed using the proper memory_limiter resource limits.
  □ The OTel Collector uses a Batch Processor before sending spans to the database.
  □ A tail-based sampling strategy is applied to save 100% of errors and 5% of normal requests.
  □ Logs-to-traces and traces-to-metrics jump exploration is configured at the Grafana dashboard level.

Summary #

  • OpenTelemetry Is the Industry Standard — OTel separates instrumentation code from backend vendors; we can migrate visualization backends without touching application code.
  • Context Propagation Connects Spans — Make sure the traceparent header is sent across HTTP, gRPC, and message queue network boundaries to keep the Trace ID connected.
  • Limit Manual Span Creation — Focus span tracking on network I/O operations, database queries, and message queues; manual span pollution makes visualizations unreadable.
  • Use Tail-Based Sampling in the Collector — Save only important data (errors and slow latency) in full to save storage capacity without losing failure visibility.
  • Register Exceptions to Spans — Always call the record_exception function and set the span status to ERROR when catching code failures so trace visualizations turn red on dashboards.
  • Use the OTel Collector as a Buffer — Deploy the OTel Collector inside the cluster to do batching and data cleansing before traces are stored in permanent databases.

← Previous: Alerting   Next: Grafana Dashboard →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact