Distributed tracing in Node.js with traceId propagation across gRPC metadata

You've opened Jaeger only to find traces broken apart at service boundaries — a situation you've probably encountered at least once. HTTP services stay connected thanks to the traceparent header, but the moment a call crosses a gRPC boundary, orphan spans multiply. I was confused when first instrumenting gRPC services, wondering "isn't W3C TraceContext HTTP-only?" — but the short answer is that the same standard works for gRPC too, just through a channel called gRPC Metadata instead of HTTP headers.

This post first identifies three root causes of context loss, then walks through real code covering auto-instrumentation via @opentelemetry/instrumentation-grpc, custom interceptors, Baggage, and the OpenTelemetry Collector pipeline. OpenTelemetry is a CNCF Graduated project, and as of 2026 the trace signal of the Node.js SDK has long been Stable — so if the criteria below fit your team size or service count, it's worth seriously considering adoption.

• Two or more gRPC-connected services in a mixed-language environment
• Already injecting traceIds into logs for correlation, but lacking span-tree visualization
• Currently tracing p99 latency root causes in production without code instrumentation

Why Traces Break at gRPC Boundaries

traceparent Flows Through Metadata

gRPC is an HTTP/2-based protocol. Context propagation uses the same traceparent / tracestate key-value pairs, but carries them in metadata encoded as HPACK headers. The channel differs, but the W3C TraceContext specification itself applies identically.

OpenTelemetry writes the current context into metadata with propagation.inject(), restores it on the receiving side with propagation.extract(), and wraps the handler with context.with() to form a parent-child span relationship. Any gap in this process breaks the trace.

Three Root Causes of Loss

First, missing instrumentation. In a multi-language environment, if even one Python service skips OpenTelemetry initialization, the chain breaks at that point. A service with no instrumentation in the middle won't propagate traceparent to the next service, even if it receives one.

Second, no interceptor. In situations where the auto-instrumentation library cannot intercept — such as a custom gRPC client — if there is no interceptor injecting traceparent into the metadata, the request goes out without context.

Third, unhandled async boundaries. The Node.js SDK uses AsyncLocalStorageContextManager backed by AsyncLocalStorage to automatically handle async boundaries like Promise and setTimeout (the @opentelemetry/context-async-hooks package implements this). However, EventEmitter is slightly different. Listener functions run in the context of when the event is emitted, not when the listener was registered, so creating a span inside a listener can attach it to an unexpected parent span or find no context at all. When needed, it's safer to explicitly bind the context when registering the listener with context.bind(context.active(), listener).

worker_threads is another special case. Due to process-level isolation, AsyncLocalStorage cannot cross worker thread boundaries, so you must include the serialized context directly in the message payload.

From SDK Initialization to the Collector Pipeline

1. Attaching gRPC Instrumentation to the Node.js SDK

The first thing to verify is initialization order. The SDK file must be imported before all other imports for auto-instrumentation to work correctly.

// tracing.ts
import { NodeSDK } from '@opentelemetry/sdk-node';
import { GrpcInstrumentation } from '@opentelemetry/instrumentation-grpc';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-grpc';
import { Resource } from '@opentelemetry/resources';
 
const sdk = new NodeSDK({
  resource: new Resource({
    'service.name': 'order-service',
  }),
  traceExporter: new OTLPTraceExporter({
    url: 'http://localhost:4317',
  }),
  instrumentations: [
    new GrpcInstrumentation(),
  ],
});
 
sdk.start();
 
process.on('SIGTERM', () => {
  sdk.shutdown().finally(() => process.exit(0));
});

// index.ts — import tracing.ts first
import './tracing';
import express from 'express';
// ...

The url option of @opentelemetry/exporter-trace-otlp-grpc supports the scheme-less form localhost:4317 as well as http://localhost:4317 / https://. The same applies when specified via the environment variable (OTEL_EXPORTER_OTLP_ENDPOINT). See the official exporters guide for more options.

Once GrpcInstrumentation is registered, interceptors are automatically attached to both grpc-js-based clients and servers. On client calls, traceparent is injected into the metadata; on server reception, it is extracted and a parent-child span relationship is formed. In most cases, this is all you need.

2. When Auto-Instrumentation Doesn't Work — Custom Interceptors

If you have a legacy gRPC client or a situation where the instrumentation library cannot intercept, you need to write the interceptor yourself.

// otel-client-interceptor.ts
import { context, propagation } from '@opentelemetry/api';
import * as grpc from '@grpc/grpc-js';
 
export function otelClientInterceptor(
  options: grpc.CallOptions,
  nextCall: (options: grpc.CallOptions) => grpc.InterceptingCall,
): grpc.InterceptingCall {
  return new grpc.InterceptingCall(nextCall(options), {
    start(
      metadata: grpc.Metadata,
      listener: grpc.InterceptingListener,
      next: (metadata: grpc.Metadata, listener: grpc.InterceptingListener) => void,
    ) {
      propagation.inject(context.active(), metadata, {
        set(carrier: grpc.Metadata, key: string, value: string) {
          carrier.set(key, value);
        },
      });
      next(metadata, listener);
    },
  });
}
 
const client = new OrderServiceClient(
  'order-service:50051',
  grpc.credentials.createInsecure(),
  { interceptors: [otelClientInterceptor] },
);

On the server side, restore the context from the metadata when the handler is entered, then wrap the subsequent logic with context.with().

// order-service-handler.ts
import {
  context,
  propagation,
  ROOT_CONTEXT,
  SpanStatusCode,
  trace,
} from '@opentelemetry/api';
import * as grpc from '@grpc/grpc-js';
 
const metadataGetter = {
  get(carrier: grpc.Metadata, key: string) {
    const values = carrier.get(key);
    return values.length > 0 ? String(values[0]) : undefined;
  },
  keys(carrier: grpc.Metadata) {
    return Object.keys(carrier.getMap());
  },
};
 
async function handleCreateOrder(
  call: grpc.ServerUnaryCall<CreateOrderRequest, CreateOrderResponse>,
  callback: grpc.sendUnaryData<CreateOrderResponse>,
) {
  const parentCtx = propagation.extract(ROOT_CONTEXT, call.metadata, metadataGetter);
 
  await context.with(parentCtx, async () => {
    const tracer = trace.getTracer('order-service');
    const span = tracer.startSpan('handle-create-order');
    try {
      const response = await processOrder(call.request);
      span.end();
      callback(null, response);
    } catch (err) {
      span.recordException(err as Error);
      span.setStatus({ code: SpanStatusCode.ERROR });
      span.end();
      callback(err as grpc.ServiceError);
    }
  });
}

3. Propagating Business Context with Baggage

While traceparent and tracestate are channels for connecting traces, Baggage is a separate channel for flowing business identifiers like userId or tenantId throughout the entire call chain. Putting them directly in gRPC metadata makes them visible only to the immediate receiver, but Baggage is automatically propagated to all downstream services.

import { context, propagation } from '@opentelemetry/api';
 
async function handleHttpRequest(req: Request) {
  const baggage = propagation.createBaggage({
    'user.id': { value: req.user.id },
    'tenant.id': { value: req.tenant.id },
  });
 
  const ctxWithBaggage = propagation.setBaggage(context.active(), baggage);
 
  await context.with(ctxWithBaggage, async () => {
    await orderServiceClient.createOrder(request);
  });
}
 
function getCurrentTenantId(): string | undefined {
  const baggage = propagation.getBaggage(context.active());
  return baggage?.getEntry('tenant.id')?.value;
}

Two things to keep in mind. First, since Baggage propagates to all downstream services, including sensitive information (tokens, passwords) can lead to unintended exposure. It is recommended to limit it to business identifiers only. Second, the W3C Baggage specification imposes header size limits (8192 bytes total, 64 entries) so you cannot carry arbitrarily large payloads. Exceeding these limits can cause headers to be truncated by proxies or gateways, so it's safer to carry only summarized IDs and fetch detailed information separately.

4. Configuring the Collector Pipeline

Outside of application code, the Collector's core role is handling batching, sampling, and metadata enrichment. In Kubernetes environments, a two-tier architecture combining a sidecar (agent) Collector and a central Gateway Collector is widely used.

The sidecar Collector automatically attaches K8s Pod attributes and forwards them to the Gateway.

# collector-sidecar.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
 
processors:
  batch:
    timeout: 5s
    send_batch_size: 1024
  k8sattributes:
    extract:
      metadata:
        - k8s.pod.name
        - k8s.namespace.name
        - k8s.node.name
 
exporters:
  otlp:
    endpoint: gateway-collector:4317
    tls:
      insecure: true
 
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch, k8sattributes]
      exporters: [otlp]

There is one essential prerequisite here. The k8sattributes processor queries the K8s API to retrieve Pod metadata, so the Collector service account requires the following ClusterRole.

# collector-rbac.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: otel-collector-k8sattributes
rules:
  - apiGroups: [""]
    resources: [pods, namespaces]
    verbs: [get, watch, list]
  - apiGroups: [apps]
    resources: [replicasets]
    verbs: [get, watch, list]
  - apiGroups: [extensions]
    resources: [replicasets]
    verbs: [get, watch, list]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: otel-collector-k8sattributes
subjects:
  - kind: ServiceAccount
    name: otel-collector
    namespace: observability
roleRef:
  kind: ClusterRole
  name: otel-collector-k8sattributes
  apiGroup: rbac.authorization.k8s.io

Without this RBAC, the Collector starts but Pod attributes will not be attached, or API access failure logs will accumulate continuously.

Add tail sampling to the Gateway Collector. Head-based sampling makes decisions early in the request, which can miss errors that occur later, so a mixed strategy — capturing all error traces while rate-sampling normal traces — is commonly used (see Uptrace's sampling guide).

# gateway-collector.yaml (key section)
processors:
  batch:
    timeout: 5s
    send_batch_size: 1024
  k8sattributes:
    extract:
      metadata:
        - k8s.pod.name
        - k8s.namespace.name
        - k8s.node.name
  tail_sampling:
    decision_wait: 30s
    num_traces: 50000
    policies:
      - name: errors-policy
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: slow-traces-policy
        type: latency
        latency:
          threshold_ms: 500
      - name: normal-traces-policy
        type: probabilistic
        probabilistic:
          sampling_percentage: 10
 
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [k8sattributes, tail_sampling, batch]
      exporters: [otlp]

With this in place, even in an environment mixing Node.js, Python, and Java, the entire chain is connected under a single traceId. Here is the full flow with both sidecar and gateway as a sequence diagram.

Tradeoffs and Common Pitfalls

The gRPC streaming pitfall. @opentelemetry/instrumentation-grpc creates spans not only for unary RPCs but also for server, client, and bidirectional streaming. However, context is not automatically carried through each callback while data flows over an open stream. You need to capture the context at the moment the stream is opened into a local variable and wrap data callbacks with context.with().

OTLP gRPC load balancing caution. Because gRPC maintains long-lived connections, a standard L4 load balancer will not distribute traffic evenly across Gateway Collector replicas. You can solve this by using a gRPC-aware (L7) load balancer or by placing an agent Collector layer in between.

Memory pressure from tail sampling. The Collector keeps spans in memory for the duration of decision_wait while waiting for a complete trace. If num_traces is not tuned to match the expected number of concurrent traces, OOM can occur under high traffic.

What This Architecture Does Not Cover

The configuration above is essentially the skeleton needed to avoid losing traceIds at gRPC boundaries. However, the following situations require separate design decisions.

• mTLS without a service mesh. When doing mTLS at the application level without Istio or Linkerd, the X-Forwarded-* hints that a proxy would normally attach are absent, which can make service topology detection inaccurate. Specifying the service.name resource attribute clearly for each service becomes especially important.
• Non-OTLP backends. When sending to backends that only support their own protocol — like Datadog Agent or New Relic — you need to route through a Collector exporter module for conversion, and in that process some attributes may be lost or their names remapped. Checking the per-backend attribute mapping documentation in advance can save a lot of headaches.
• Baggage size limits. As mentioned above, the header size cap means large contexts cannot be carried. Data like a full user profile is safer stored in a separate store keyed by traceId and fetched on demand.
• Traces crossing async queues. When crossing a message broker like Kafka, RabbitMQ, or SQS, the protocol itself does not automatically forward W3C TraceContext. You need to separately introduce an instrumentation library (e.g., @opentelemetry/instrumentation-kafkajs) that manually carries traceparent in message headers or attributes.

The overall approach has already been covered above — start with auto-instrumentation and, if any of these four items apply to your situation, work out the scenarios before rolling out to production. In a Kubernetes environment, it's also worth evaluating the OpenTelemetry Operator's Pod auto-instrumentation injection feature — it activates gRPC interceptors and context propagation through Deployment annotations alone, making it a useful standardized instrumentation deployment method for organizations with many services.

References

• OpenTelemetry Official Docs - Baggage
• OpenTelemetry Official Docs - Collector Architecture
• OpenTelemetry Official Docs - JS Exporters
• W3C Baggage Specification
• W3C Trace Context Specification
• How to Propagate OpenTelemetry Trace Context Through gRPC Metadata - OneUptime
• How to Implement OpenTelemetry Tracing for gRPC Services - OneUptime
• How to Add Distributed Tracing to gRPC with OpenTelemetry - OneUptime
• OTel Trace Context Propagation for gRPC Streams - Tracetest Blog
• OpenTelemetry Context Propagation: W3C TraceContext & Troubleshooting Guide - Uptrace
• How to Propagate Trace Context Across Async Boundaries - OneUptime
• How to Implement Distributed Tracing in Node.js Microservices - OneUptime
• OpenTelemetry Collector: Beginner's Guide to Telemetry Pipelines - Dash0
• OTLP gRPC Exporter: A Practical Guide - Dash0
• OpenTelemetry Context Propagation Explained - Better Stack
• OpenTelemetry Sampling: head-based and tail-based - Uptrace
• OpenTelemetry Collector in Kubernetes - groundcover
• @opentelemetry/instrumentation-grpc - npm
• k8sattributes processor - OpenTelemetry Collector Contrib
• How to Trace gRPC Calls Across Kubernetes Services with OpenTelemetry - OneUptime
• Essential OpenTelemetry Best Practices - Better Stack