Consolidating node signals and propagating backpressure through an OTel Collector DaemonSet

When operating a Kubernetes cluster, you'll inevitably reach a point where your observability data collection architecture becomes fragmented. Prometheus scrapes node metrics, Fluentd collects logs separately, and traces are handled by yet another dedicated agent. I've inherited and operated this kind of setup on my team — tuning each agent's configuration separately, managing RBAC independently for each, and figuring out which agent dropped data when the backend went down for a moment was a job in itself.

Deploying the OpenTelemetry Collector as a DaemonSet lets a single process per node handle all three signals. Host metrics, kubelet stats, and container logs are collected locally without a network hop. While service.pipelines defines separate pipelines for traces, metrics, and logs, the same processor instances are shared within the same Collector process. This allows backpressure control and metadata tagging to be applied consistently in one place.

This post covers how to set up a unified pipeline, how the three backpressure components (memory_limiter, sending_queue, retry_on_failure) actually behave, and the pitfalls you'll commonly step on in production. It also covers what changed after the v1.49.0/v0.143.0 release in January 2026.

Why DaemonSet, and Why Bundle Everything into One Collector

Node-Local Access Is the Key

Host metrics (/proc, /sys) and container logs (/var/log/pods) must be read directly from the node they reside on. Running a Collector as a Deployment requires complex workarounds to access these paths, or you end up needing a separate agent on each node anyway — a contradiction.

A DaemonSet automatically deploys a Collector Pod when a node is added. This means your observability infrastructure scales out automatically.

Keep Per-Signal Pipelines, but Share Processors

Strictly speaking, the OTel Collector is designed to separate pipelines by signal type — you cannot combine traces, metrics, and logs into a single pipeline definition. Instead, referencing the same component ID (name) across three pipelines causes them to share the same internal instance. For example, if the k8sattributes processor is referenced by traces, metrics, and logs pipelines, the same instance attaches identical k8s metadata to all three signals. Conversely, using different IDs like k8sattributes/traces and k8sattributes/metrics treats them as separate instances. Even if the configuration is identical, different names mean no sharing — this is an important distinction.

The same principle applies to memory_limiter. Referencing the same ID means a single instance, and that processor evaluates threshold breaches based on the entire process's memory usage. There is no independent memory budget per pipeline; when the single process reaches a danger level, reception is refused at the front of all pipelines. The reason you must still place it first in each pipeline's processors array is not that the instances are independent — it's that you need to block data before downstream processors expand or replicate it and consume more memory.

The diagram above shows logical sharing relationships, not physical structure. In practice, service.pipelines defines separate traces/metrics/logs pipelines, each referencing processor instances by the same name.

The 2-Tier Architecture Is the Current Recommended Pattern

In large clusters, you don't handle all processing with just a DaemonSet. As of 2026, the DaemonSet (lightweight collection) → Deployment Gateway (aggregation, transformation, routing) two-tier structure is the pattern recommended by the official blueprint.

Keep the DaemonSet as lightweight as possible (collection + basic tagging + batching), and delegate heavy transformation, filtering, and tail-sampling logic to the Gateway.

Practical Configuration: Unified Pipeline in Code

Full Collector Configuration

Below is a complete configuration for handling node-level metrics, logs, and traces.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
  hostmetrics:
    collection_interval: 30s
    scrapers:
      cpu: {}
      memory: {}
      disk: {}
      filesystem: {}
      network: {}
  kubeletstats:
    collection_interval: 30s
    auth_type: serviceAccount
    endpoint: ${env:K8S_NODE_IP}:10250
    # For development environments where the kubelet certificate is not in a separate trust chain
    # In production, disable insecure_skip_verify and mount the CA bundle instead
    insecure_skip_verify: true
  filelog:
    include:
      - /var/log/pods/*/*/*.log
    include_file_path: true
    operators:
      - type: container
        id: container-parser
 
processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 20
  k8sattributes:
    auth_type: serviceAccount
    passthrough: false
    extract:
      metadata:
        - k8s.pod.name
        - k8s.namespace.name
        - k8s.node.name
  batch:
    send_batch_size: 1000
    timeout: 10s
 
exporters:
  otlp:
    endpoint: otel-gateway:4317
    sending_queue:
      enabled: true
      num_consumers: 10
      queue_size: 5000
      storage: file_storage
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 300s
 
extensions:
  file_storage:
    directory: /var/lib/otelcol/queue
 
service:
  extensions: [file_storage]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, k8sattributes, batch]
      exporters: [otlp]
    metrics:
      receivers: [otlp, hostmetrics, kubeletstats]
      processors: [memory_limiter, k8sattributes, batch]
      exporters: [otlp]
    logs:
      receivers: [otlp, filelog]
      processors: [memory_limiter, k8sattributes, batch]
      exporters: [otlp]

Note that the order in which processors are defined in the processors block does not determine execution order. The actual execution order is determined by the order of the processors array under each pipeline. If memory_limiter is not first in the array, the downstream processors will have already expanded the data before the limit kicks in, weakening the OOM Kill protection.

Key Parts of the DaemonSet Manifest

env:
  - name: K8S_NODE_IP
    valueFrom:
      fieldRef:
        fieldPath: status.hostIP
  - name: GOMEMLIMIT
    value: "400MiB"   # approximately 80% of the container memory limit
 
resources:
  limits:
    memory: 500Mi
    cpu: 500m
  requests:
    memory: 200Mi
    cpu: 100m
 
volumeMounts:
  - name: varlogpods
    mountPath: /var/log/pods
    readOnly: true
  - name: otelcol-queue
    mountPath: /var/lib/otelcol/queue
 
volumes:
  - name: varlogpods
    hostPath:
      path: /var/log/pods
  - name: otelcol-queue
    hostPath:
      path: /var/lib/otelcol/queue

The reason for setting GOMEMLIMIT to approximately 80% of the container memory limit is different from the common misconception. Since Go 1.19, the runtime can detect cgroup memory limits on its own, and since recent OTel Collector releases are based on Go 1.21+, this is not an issue. The real reason GOMEMLIMIT is needed is to separately control the hard limit (OOM Kill) and the soft limit (GC trigger). By keeping the container limit as-is and setting the soft limit lower, GC begins aggressively reclaiming memory before the hard limit is reached, giving memory_limiter room to engage. The triple defense of memory_limiter (block reception) + GOMEMLIMIT (early GC trigger) + container limit (final safety net) is the standardized combination as of 2026.

Why the Persistent Queue Matters

The file_storage extension acts as the backend for sending_queue. An in-memory queue loses its data when the Collector restarts, but a disk-based WAL (Write-Ahead Log) queue recovers unsent data after a restart. When planning for prolonged backend outages, you need to size both the per-node queue capacity (queue_size: 5000) and the available disk space together.

RBAC: Which Component Requires What Permissions

kubeletstats and k8sattributes target different APIs. Lumping everything into one ClusterRole makes it difficult to identify which component is lacking permissions during troubleshooting. It's better to understand them by purpose.

• When the kubeletstats receiver communicates directly with kubelet (port 10250): nodes/stats permission is required. The configuration in this post, using endpoint: ${env:K8S_NODE_IP}:10250 to connect directly, falls into this category.
• When using kubeletstats via the API server proxy path (/api/v1/nodes/{node}/proxy/stats/summary): nodes/proxy permission is additionally required.
• k8sattributes processor: get/list/watch on pods and namespaces is required to look up Pod and namespace metadata.

Here is an example for using all three components together.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: otel-collector
rules:
  # kubeletstats: direct kubelet communication mode
  - apiGroups: [""]
    resources: ["nodes/stats"]
    verbs: ["get"]
  # Add only when using kubeletstats via API server proxy
  # - apiGroups: [""]
  #   resources: ["nodes/proxy"]
  #   verbs: ["get"]
  # k8sattributes: metadata lookup
  - apiGroups: [""]
    resources: ["pods", "namespaces"]
    verbs: ["get", "list", "watch"]

When permissions are insufficient, receivers often silently return empty data rather than throwing a clear exception. Always verify the received item count per receiver in the Collector's own metrics immediately after deployment.

Understanding the Two Directions of Backpressure

It's easy to lump these three components together as a "safety net," but in practice, they operate in opposite directions. memory_limiter works at the front of the pipeline, refusing reception toward the upstream (receivers) — a blocking direction. sending_queue and retry_on_failure work at the back of the pipeline, absorbing failures from the downstream (backend) — a buffering direction.

Summarizing the two directions:

retry_on_failure and sending_queue operate independently. Enabling only retry without a queue means retry-pending data exists only in memory; if the backend is down for an extended period, older data will eventually be dropped. Enable both explicitly, and attach file_storage to the queue whenever possible so data survives restarts — this is the battle-tested combination.

Tradeoffs and Common Pitfalls

Pros and Cons

Common Pitfalls in Practice

1. Wrong placement of memory_limiter

This is the most frequently seen mistake. With the ordering below, the batch processor bundles data and expands memory usage before memory_limiter fires, so the protection kicks in too late.

# Wrong order
processors: [k8sattributes, batch, memory_limiter]
 
# Correct order
processors: [memory_limiter, k8sattributes, batch]

2. Broken sharing due to incorrectly split component IDs

Sometimes people split k8sattributes or memory_limiter by signal type (e.g., k8sattributes/traces, k8sattributes/metrics) to tune them per signal, but end up with completely identical configurations. This instantiates them as separate instances, doubling memory and watcher connections. Only split IDs when you actually need different configurations per signal.

3. Blindly copying kubelet TLS verification bypass

The insecure_skip_verify: true in the example configuration is for development environments. In production, the principle is to mount the kubelet's server certificate via a CA bundle and disable this option. Even if inter-node traffic is isolated by network policies, leaving this option enabled keeps the attack surface open for kubelet endpoint spoofing.

Dashboard Migration After the January 2026 Release

After deploying the configuration above, you'll monitor pipeline health using the Collector's own metrics — and here, a recent version change can trip you up.

Starting with the January 2026 v1.49.0/v0.143.0 release, semantic conventions v1.38.0 were adopted, and the previously signal-separated processor metrics were unified into otelcol_processor_incoming_items and otelcol_processor_outgoing_items. The traces/metrics/logs distinction moved to the otel.signal attribute. If you import an existing Grafana dashboard as-is, the per-signal graphs may appear blank. It's worth reviewing the following items before upgrading:

• Processor throughput panels: Replace old metric names with the form otelcol_processor_incoming_items{otel.signal="traces"}
• Alert rules (e.g., drop rate spikes): Add otel.signal to label filters
• Dashboard variables: Instead of hardcoding signals, make otel.signal labels into variables

What to Check First Right After Deployment

If you paste in what you learned from documentation and call it done, something will go wrong within days. Here's what I recommend checking in order immediately after rollout:

1. Received item count per receiver: Verify otelcol_receiver_accepted_* is non-zero for each receiver. If it's zero, the culprit is almost always missing RBAC, a hostPath typo, or a failure to reach the kubelet endpoint.
2. Whether memory_limiter is firing: Look at otelcol_processor_refused_* together with process RSS. If it's already firing under normal load, it's time to readjust the container limit or GOMEMLIMIT.
3. Queue size trend: If otelcol_exporter_queue_size is trending upward, the backend is failing to keep up with ingestion. Consider scaling up the Gateway or adjusting batch parameters (send_batch_size, timeout).
4. file_storage directory usage: If node disk fills up faster than expected, recalculate the worst-case scenario by multiplying queue_size by the hourly ingest volume.
5. Drop counters: If otelcol_exporter_send_failed_* is continuously increasing, it may mean the retry_on_failure retry budget (max_elapsed_time: 300s) is shorter than the actual backend recovery time.

Putting just these five on a single dashboard page will do the most to prevent real incidents. For all other tuning, make decisions based on the direction these metrics move when they become unstable.

References

• OpenTelemetry Official: Kubernetes Getting Started
• OpenTelemetry Official: Kubernetes Collector Components
• OpenTelemetry Official: Collector Helm Chart
• OpenTelemetry Official: Resiliency (Persistent Queue, Retry)
• How to Deploy the OpenTelemetry Collector as a DaemonSet in Kubernetes – OneUptime
• How to Configure Backpressure-Aware Pipelines – OneUptime
• Batching, Queuing, and Retries in the OpenTelemetry Collector – Dash0
• Mastering the OpenTelemetry Memory Limiter Processor – Dash0
• OpenTelemetry Collector Backpressure: Fixing Drops with memory_limiter and Queues – Michal Drozd
• OpenTelemetry Collector Persistence and Retry Mechanisms Under the Hood – Axoflow
• OpenTelemetry Collector in Kubernetes: Architecture, Deployment & Best Practices – Groundcover
• OpenTelemetry Collector v1.49.0/v0.143.0: What's New – FusionReactor
• memory_limiter Processor GitHub README
• Improve OTel Collector Resilience using Persistent File Storage – Dynatrace Docs
• Managed Telemetry Platforms for Kubernetes Workloads Blueprint – OpenTelemetry