"Why Does It Slow Down After Deployment?" — Connecting Node.js Production FlameGraphs to Traces with Grafana Pyroscope and OTLP Profiles
When running a Node.js backend, you'll eventually encounter this situation: you spot a sudden spike in P95 latency for a specific endpoint in Grafana, open the trace in Tempo, and stare at the span timeline — but have no idea where that 1.2 seconds came from. It's not a DB query, it's not an external HTTP call — the Node process is clearly working hard at something, but there's no way to know what.
When that happens, you typically reach for --inspect with Chrome DevTools locally, or run clinic flame. If you can reproduce the issue, great — but production performance problems almost never reproduce locally or in staging. Traffic patterns differ, memory state differs, JIT optimization state differs.
Continuous Profiling tackles this problem head-on. It collects CPU usage and call stacks without interruption in production, letting you replay what happened "at that moment" as a FlameGraph after the fact. Grafana Pyroscope serves as that data store, and OTLP Profiles acts as the glue connecting profiles to the OTel ecosystem in a standardized way. This post covers the process of attaching this stack to a Node.js environment in practice, including Span Profiles integration for one-click drill-down from traces to FlameGraphs.
Core Concepts
Continuous Profiling vs One-Shot Profiling
When people first hear "Continuous Profiling," a common misconception is "isn't that just automated profiling?" But the difference is quite fundamental.
| Item | One-Shot Profiling | Continuous Profiling |
|---|---|---|
| Timing | Manually attached when an issue occurs | Always collecting, reviewed after the fact |
| Environment | Mainly local or staging | Production directly |
| Reproducibility | Can only analyze reproducible problems | Can retroactively analyze intermittent or one-off issues |
| Overhead | Unlimited (used only briefly) | Designed for under 2–3% CPU (real-world example) |
| Data Retention | Only during the session | Can be stored for weeks to months |
"Retroactively analyzing a past performance issue using historical profiles" — that single sentence captures the core value of Continuous Profiling. After an incident, you can replay a FlameGraph in a postmortem to see exactly which functions were eating CPU at that moment.
How to Read a FlameGraph
In a FlameGraph, the X-axis represents not time but CPU occupancy ratio (sample count). A wide block = ran a lot = bottleneck candidate. The Y-axis represents call stack depth; Pyroscope's default Icicle view places callers (root) at the top and callees (leaves) at the bottom.
┌─────────────────────────────────────────────────────────────────┐
│ handleRequest (73% of total) │
│ ├── parseBody (12%) │ processQuery (58%) │ sendResponse (3%) │
│ │ ├── buildSQL (8%) │
│ │ └── executeQuery (50%) ← bottleneck │
└─────────────────────────────────────────────────────────────────┘What you're looking for is not the widest block at the very top, but a wide and flat block — one that's wide without calling many child functions. That's where CPU is actually being burned.
OTLP Profiles — The Fourth OTel Signal
OpenTelemetry originally standardized three signals: Traces, Metrics, and Logs. OTLP Profiles is the fourth signal, which entered Public Alpha in March 2026, providing a specification for integrating profile data into the OTel ecosystem.
Its core structure is based on the pprof format, with OTel's Resource and InstrumentationScope context added, and it includes a field that can directly link to a trace's Span ID. This link is the foundation of Span Profiles — enabling "show me only the FlameGraph for while this span was executing."
One thing to note from a Node.js perspective: Go, Java, Python, and .NET SDKs can already output OTLP Profiles stably, but official Profiling support in the Node.js OTel SDK is currently in alpha (see the official support tracker at opentelemetry-js GitHub). For production use right now, the Pyroscope-specific @pyroscope/nodejs SDK is the more stable choice.
Pyroscope 2.0 Architecture — Single Write + Stateless Read
The key change in Pyroscope 2.0, which went GA in April 2026, is architectural simplification.
In 1.x, the write path handled replication directly, and the read-path Querier was pinned to specific storage replicas. 2.0 delegates replication to object storage (S3/GCS/Azure) and makes Queriers fully stateless. When query load increases, you only need to horizontally scale the Queriers, and storage costs are reduced by writing a single copy without replicas. Grafana itself processes 19.5PB of profile data using this architecture (Introducing Pyroscope 2.0).
Before Implementation — Choosing Between SDK and eBPF
All subsequent steps are explained using the SDK approach. Since the choice may vary by environment, it's worth comparing them first.
| Item | SDK Approach (@pyroscope/nodejs) |
eBPF Approach (pyroscope.ebpf) |
|---|---|---|
| Code changes | Required | Not required |
| Node.js call stack accuracy | High (reflects JIT optimization) | May miss V8 JIT frames |
| Span Profiles support | Supported (PyroscopeSpanProcessor) |
Not supported |
| Linux kernel requirement | None | 4.8 or higher |
| Container privileges | None | SYS_BPF required |
| System-wide profiling | Node.js process only | Entire system possible |
If you need accurate call stacks and Span Profiles for your Node.js application, choose the SDK approach. eBPF can conflict with Kubernetes PSS Restricted policies, and due to V8 JIT characteristics, stack frames can be dropped, which limits its usefulness for Node.js-specific diagnostics.
Practical Implementation
Step 0: Run the Pyroscope Server
All example code below assumes http://pyroscope:4040 is already running. To get started quickly locally or in staging, you can spin it up with Docker Compose.
# docker-compose.yml
services:
pyroscope:
image: grafana/pyroscope:latest
ports:
- "4040:4040"
volumes:
- pyroscope-data:/var/lib/pyroscope
volumes:
pyroscope-data:For production, you can find Helm chart configuration for connecting object storage (S3, etc.) as a backend in the Pyroscope official documentation.
Step 1: Node.js SDK Setup — @pyroscope/nodejs
npm install @pyroscope/nodejsInitialize at the very top of your application entry point. It must be loaded before any other modules for the V8 sampler to attach properly.
// profiler.js — ESM (Node 12+, package.json "type": "module") example
import Pyroscope from '@pyroscope/nodejs';
Pyroscope.init({
serverAddress: process.env.PYROSCOPE_SERVER_URL || 'http://localhost:4040',
appName: 'my-node-api',
tags: {
env: process.env.NODE_ENV,
version: process.env.APP_VERSION,
region: process.env.AWS_REGION,
},
});
Pyroscope.start();For CommonJS projects, replace import with const Pyroscope = require('@pyroscope/nodejs').default.
tags are labels you can filter with FlameQL queries. Putting high-cardinality values like requestId or userId as tags will severely degrade storage and query performance. Use only low-cardinality values at the level of env, version, region.
Step 2: Centralizing Collection with Grafana Alloy
While the SDK can push directly to the Pyroscope server, placing Grafana Alloy in between lets you manage authentication, label enrichment, and routing logic outside your application code. This is especially advantageous in multi-service environments.
// alloy.config
pyroscope.receive_http "node_apps" {
http {
listen_address = "0.0.0.0"
listen_port = 9999
}
forward_to = [pyroscope.write.default.receiver]
}
pyroscope.write "default" {
endpoint {
url = "http://pyroscope:4040"
// Add the block below when using Grafana Cloud
basic_auth {
username = env("GRAFANA_CLOUD_INSTANCE_ID")
password = env("GRAFANA_CLOUD_API_KEY")
}
}
}Then update the SDK's serverAddress to point to Alloy.
Pyroscope.init({
serverAddress: 'http://alloy:9999',
appName: 'my-node-api',
// ...
});Step 3: Span Profiles — Drilling Down from Traces to FlameGraphs
Span Profiles is the core value of this stack. You can filter FlameGraphs down to the execution window of a specific span, letting you drill directly into the exact code path of slow requests from the trace UI.
npm install @opentelemetry/sdk-node @opentelemetry/auto-instrumentations-node
npm install @pyroscope/otel// tracing.js — ESM example
import { NodeSDK } from '@opentelemetry/sdk-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { PyroscopeSpanProcessor } from '@pyroscope/otel';
const sdk = new NodeSDK({
traceExporter: new OTLPTraceExporter({
url: 'http://tempo:4318/v1/traces',
}),
spanProcessors: [
new PyroscopeSpanProcessor(), // injects active Span ID as a profile label
],
instrumentations: [getNodeAutoInstrumentations()],
});
sdk.start();PyroscopeSpanProcessor automatically injects OTel's currently active Span ID as a profile label. This attaches span_id and trace_id labels to the profile data, so when you click the "Profiles" link for a span in Grafana Tempo, you see a FlameGraph filtered to just that span's window.
The load order of the two files in the entry point matters.
// index.js
import './profiler.js'; // must be first line — attach V8 sampler first
import './tracing.js'; // then initialize OTel SDK
import express from 'express';
// ... rest of the codeThe actual drill-down flow looks like this:
Step 4: Grafana Datasource Integration Setup
After adding the Pyroscope datasource in Grafana, enable the "Trace to profiles" link in the Tempo datasource settings.
# Grafana provisioning datasources.yaml example
apiVersion: 1
datasources:
- name: Tempo
type: tempo
url: http://tempo:3200
jsonData:
tracesToProfiles:
datasourceUid: pyroscope-uid
profileTypeId: process_cpu:cpu:nanoseconds:cpu:nanoseconds
tags:
- key: service.name
value: appName
customQuery: false
- name: Pyroscope
uid: pyroscope-uid
type: grafana-pyroscope-datasource
url: http://pyroscope:4040profileTypeId is the profile type identifier in Pyroscope. The CPU profile that the Node.js SDK collects by default uses the format process_cpu:cpu:nanoseconds:cpu:nanoseconds. You can confirm this directly from the dropdown in Grafana Explore → Pyroscope datasource.
Common Issues
Points to check when things don't work as expected after setup.
When no profile data appears in Pyroscope at all
- Verify that
profiler.jsis imported on the first line ofindex.js. If SDK initialization runs after other modules, some functions will be excluded from profiling scope. - Check that the
PYROSCOPE_SERVER_URLenvironment variable orserverAddressvalue is an actually reachable address. - Confirm that
Pyroscope.start()is called afterPyroscope.init().
When the "Profiles" link doesn't appear in Tempo
- The
tracesToProfiles.datasourceUidindatasources.yamlmust exactly match the actualuidof the Pyroscope datasource. - The
profileTypeIdmust match the actually collected type shown in the Grafana Explore → Pyroscope datasource dropdown. The format may differ depending on the SDK version.
When FlameGraphs don't have a span_id label
- Confirm that
PyroscopeSpanProcessoris registered in the OTel SDK'sspanProcessorsarray. - Confirm that
tracing.jsis loaded afterprofiler.js. If the order is reversed, the Pyroscope sampler cannot read the OTel context and thespan_idlabel will be missing.
Pros and Cons Summary
Advantages
| Item | Details |
|---|---|
| Low overhead | Under 2–3% CPU overhead (source) — acceptable for always-on production use |
| Time-travel debugging | Retroactively analyze past performance issues using historical profiles |
| Trace integration | Span ID-based drill-down to identify the exact code path for slow requests |
| Grafana ecosystem integration | Connected with Tempo, Loki, and Mimir in a single UI — no separate dashboard setup needed |
| Open standard | OTLP Profiles enables data portability without vendor lock-in |
| Cost efficiency | Pyroscope 2.0 single write + stateless read reduces storage and compute costs |
| Metrics from Profiles | Converts profile aggregates into time-series metrics for CPU comparison across services and versions |
Disadvantages and Caveats
| Item | Details |
|---|---|
| Node.js OTel SDK immaturity | Official OTLP Profiles Node.js SDK is still in alpha. @pyroscope/nodejs is more stable for now |
| No official Span Profiles guide | No official documentation for Traces to Profiles for Node.js yet. Partial implementation possible via @pyroscope/otel |
| High-cardinality labels | Using requestId, userId, etc. as tags causes storage explosion — use only low-cardinality values |
| Missing stack symbolization | V8 JIT optimization can cause Node.js frame drops in eBPF mode — requires SDK approach as a workaround |
| Security and PII risks | Call stacks can expose function names and file paths. Access control for profile data and TLS encryption in transit are essential |
| Container permission issues | eBPF requires SYS_BPF privilege — may conflict with Kubernetes PSS Restricted policy |
Closing Thoughts
Traces tell you how slow something is; FlameGraphs tell you why it's slow. Connecting the two with Span Profiles lets you go from "slow span → exact CPU-consuming code path for that window" in a single click. Continuous Profiling makes this flow available in production at all times.
Pyroscope 2.0's single write + stateless read architecture is a far more realistic operational choice than before, and OTLP Profiles entering Public Alpha is also a signal that a vendor-independent standard pipeline will be achievable long-term.
If you're starting fresh, here's the recommended order:
- Attach the
@pyroscope/nodejsSDK to a staging environment first. The experience of seeing a FlameGraph with justinit+startis compelling, and this is also a good time to measure actual overhead. - Place Grafana Alloy in between, configure the Tempo datasource, and set up the Traces to Profiles link — the moment you drill down from a trace to a FlameGraph is the turning point for this stack.
- Use Pyroscope 2.0's Metrics from Profiles feature to compare CPU changes before and after a deployment, and performance improvements become measurable numbers.
References
- Grafana Pyroscope Official Documentation
- Node.js SDK Configuration Guide | Grafana Pyroscope
- Span Profiles (Traces to Profiles) Integration | Grafana Pyroscope
- Traces to Profiles View | Grafana Pyroscope
- Configure Traces to Profiles in Grafana | Grafana Documentation
- Introducing Pyroscope 2.0 | Grafana Labs Blog
- Pyroscope 2.0, Continuous Profiling at Scale | InfoQ (2026.05)
- Pyroscope 2.0 Architecture Deep Dive | tldrecap.tech
- OpenTelemetry Profiles Enters Public Alpha (2026.03)
- Profiles Signal Concepts | OpenTelemetry Official Documentation
- OTLP Profiles Protobuf Spec | opentelemetry-proto-profile GitHub
- OTel Profiles Specification | OpenTelemetry
- The State of Profiling | OpenTelemetry Blog (2024)
- OTel Profiles Alpha — Elastic's Contribution | Elastic Observability Labs
- Linking Traces with Continuous Profiling using Pyroscope | Infracloud
- Continuous profiling in production — real-world example | Grafana Labs Blog
- Performance Optimization with Pyroscope | Filigran Blog
- How to Use Grafana Pyroscope | oneuptime.com (2026.02)
- Exploring Tempo, Mimir, Loki, and Pyroscope with Grafana Alloy | oneuptime.com (2026.02)
- Grafana Pyroscope GitHub Repository
- OTel JS SDK Profiling Official Support Tracker | opentelemetry-js GitHub
- What is a FlameGraph | pyroscope.io
- Next-Gen FlameGraphs for Node.js | Platformatic Blog (2025.09)
- Pyroscope Pricing and Review 2026 | CubeAPM
- OTel Profiles Security — PII Leakage and Access Control | systemshardening.com