How to structure cross-service traces with OpenTelemetry Semantic Conventions and make them queryable in Tempo

When monitoring distributed systems, you'll eventually hit this wall: traces are piling up, but the moment you open Grafana Explore and want to see "only spans where the payment service returned a 500 error," you have no idea how to write the query. When I started, every team used http_method, httpMethod, or HTTP_METHOD however they liked, and some services had none of these at all — making end-to-end tracing practically impossible.

OpenTelemetry Semantic Conventions solves this problem through standardized attribute names. It's not merely about unifying names — it assigns a structured namespace across all three layers of traces, metrics, and logs, enabling backends like Tempo to understand that structure and index it efficiently. As of 2026, namespaces such as HTTP, DB, and Code have already been promoted to Stable, while GenAI-related namespaces remain Experimental and are actively evolving (check the official spec page for the latest status).

This article walks through how to structure inter-service traces, which attributes to place where so that TraceQL queries work correctly, and the common pitfalls encountered during convention migrations — all with real-world scenarios.

Understanding the attribute hierarchy makes design easier

Resource vs Span: separating the roles of two layers

In Semantic Conventions, attributes live in two main places.

Resource attributes describe who produced this telemetry. Values like service.name, service.version, and deployment.environment.name go here, and they are automatically attached to every span, metric, and log emitted by that process. Set it once and you're done.

Span attributes describe what happened in this individual operation. Values that change per request — like http.request.method, db.operation.name, and messaging.destination.name — go here.

Honestly, I didn't understand why this distinction mattered at first, but it becomes immediately clear once you start writing queries in Tempo.

{ resource.service.name = "payment-service" }
 
{ span.http.response.status_code >= 500 }
 
{ resource.service.name = "payment-service" && span.http.response.status_code >= 500 }

In TraceQL, the resource. prefix targets Resource attributes and the span. prefix targets Span attributes. Without these prefixes, queries will not behave as intended.

Span Kind: how to express relationships between services

There are five Span Kinds: SERVER, CLIENT, PRODUCER, CONSUMER, and INTERNAL. These are required for Tempo's service map to draw edge directions correctly.

The diagram below shows which kind of span is created at each hop within the same trace_id. Each service creates a SERVER span when it receives a request and a CLIENT span when it calls an external service.

When cart-service calls payment-service, a CLIENT span is created inside the cart-service process and a SERVER span is created on the payment-service receiving side. Both are recorded under the same trace_id, and this relationship forms the topology of the service map. At asynchronous boundaries like Kafka, PRODUCER/CONSUMER plays the same role as CLIENT/SERVER.

Stability levels: which attributes can you trust?

A practical approach is to use only Stable attributes in dashboards and alerting rules, and restrict Experimental attributes to local development or exploratory use.

Structuring a real inter-service trace

We'll use a payment flow in an e-commerce system as an example: a three-hop HTTP flow from cart-service → payment-service → bank-gateway. All code examples in this article use semconv version v1.26.0 (if you use a different version, helper function names may differ, so check the source for that version).

Step 1: Resource setup — stamping the service identity

Set the Resource once during SDK initialization. Here is a conceptual Go example.

// Conceptual example — opentelemetry-go SDK, semconv v1.26.0
import (
    "go.opentelemetry.io/otel/sdk/resource"
    semconv "go.opentelemetry.io/otel/semconv/v1.26.0"
)
 
res, _ := resource.New(ctx,
    resource.WithSchemaURL(semconv.SchemaURL),
    resource.WithAttributes(
        semconv.ServiceName("payment-service"),
        semconv.ServiceVersion("2.4.1"),
        // Whether the DeploymentEnvironmentName helper exists depends on the semconv version.
        // If the helper is absent, use the Key constant directly as shown below.
        semconv.DeploymentEnvironmentNameKey.String("production"),
    ),
)

Using the semconv package means you never have to type attribute names by hand. It prevents typos and accidentally using outdated names at compile time, making it effective for maintaining consistency across the whole team. Note that deployment.environment.name is a relatively recent attribute, so it's safest to check the source to confirm whether a helper function has been generated in the semconv release you're using.

Step 2: HTTP span attributes — based on Stable conventions

The HTTP convention was promoted to Stable around late 2023 (see the HTTP migration guide for the exact release). The example below highlights that the attributes required differ between CLIENT and SERVER spans.

// CLIENT span in payment-service calling bank-gateway
ctx, span := tracer.Start(ctx, "POST",
    trace.WithSpanKind(trace.SpanKindClient),
)
defer span.End()
 
// CLIENT spans use the url.* / server.* family of attributes.
span.SetAttributes(
    semconv.HTTPRequestMethodKey.String("POST"),
    semconv.URLFull("https://bank-gateway.internal/charge"),
    semconv.ServerAddress("bank-gateway.internal"),
    semconv.ServerPort(443),
)
 
// After receiving the response
span.SetAttributes(
    semconv.HTTPResponseStatusCode(statusCode),
)

A common mistake here is with http.route. http.route represents the route template matched by the server (/orders/{id}) and is a SERVER-only attribute. For CLIENT spans, use url.full or url.path along with server.address/server.port. On the receiving (SERVER) span of payment-service, attaching http.route like this is correct:

// SERVER span in payment-service receiving a request from cart-service
span.SetAttributes(
    semconv.HTTPRequestMethodKey.String("POST"),
    semconv.HTTPRoute("/checkout"),
    semconv.URLPath("/checkout"),
)

If you're still using the old names (http.method, http.url), the migration section below covers that.

Step 3: DB span attributes

The DB convention was refined over multiple releases and was recently promoted to Stable (see the DB migration guide for a list of key changes). Notable renames include db.system → db.system.name, db.statement → db.query.text, and db.name → db.namespace.

One thing worth noting: db.system.name uses single-identifier enum values like postgresql, mysql, redis, and mssql. It is not a two-part <vendor>.<product> structure. The Python conceptual example below reflects this.

# Conceptual example — Python OTel SDK, manual instrumentation
from opentelemetry import trace
 
tracer = trace.get_tracer(__name__)
 
with tracer.start_as_current_span(
    "SELECT orders",
    kind=trace.SpanKind.CLIENT,
) as span:
    # Use constants if they are included in your semconv release,
    # otherwise use the exact string literals from the spec.
    span.set_attribute("db.system.name", "postgresql")
    span.set_attribute("db.operation.name", "SELECT")
    span.set_attribute("db.collection.name", "orders")
    span.set_attribute("db.namespace", "shop")
    span.set_attribute("db.query.text", "SELECT * FROM orders WHERE id = ?")
 
    result = db.execute(query, [order_id])

Since we recommended using semconv helpers in the Go example, the same applies in Python — if your release includes constants like SpanAttributes.DB_SYSTEM_NAME, prefer those for consistency. If the constants aren't available yet, use the spec string literals directly and consider adding a thin wrapper within the team to catch typos.

Step 4: Connecting Kafka asynchronous boundaries

Unlike synchronous HTTP flows, message queues require explicit trace context propagation — the trace_id is carried in message headers.

# Conceptual example — Kafka producer, injecting context into headers
from opentelemetry.propagate import inject
 
headers = {}
inject(headers)  # adds traceparent and tracestate headers
 
producer.send(
    "order-events",
    value=payload,
    headers=list(headers.items()),
)
 
span.set_attribute("messaging.system", "kafka")
span.set_attribute("messaging.destination.name", "order-events")
span.set_attribute("messaging.operation.type", "publish")

On the consumer side, restoring context with extract(headers) links everything into a single trace across the asynchronous boundary. When you search by trace_id in Grafana Tempo, the entire flow from cart-service to notification-service appears on one screen.

Querying with TraceQL

With good structure in place, queries like these become possible. These align with the code examples above and cover both the /charge (CLIENT) and /checkout (SERVER) perspectives.

{ span.http.route = "/checkout" && span.http.request.method = "POST" && span:status = error }
 
{ resource.service.name = "payment-service" && span.http.response.status_code >= 500 }
 
{ span.db.system.name = "postgresql" && span:duration > 500ms }
 
{ span.messaging.system = "kafka" && span.messaging.operation.type = "receive" }

With Tempo's TraceQL Metrics, you can compute RED-style signals directly from traces without Prometheus.

{ resource.service.name = "payment-service" } | rate() by (span.http.response.status_code)
 
{ span:status = error } | rate() by (resource.service.name)

Convention migration: how to avoid common mistakes

From old to new: OTEL_SEMCONV_STABILITY_OPT_IN

When a large-scale rename occurs — as happened with the HTTP convention — switching all at once breaks existing dashboards. The OTEL_SEMCONV_STABILITY_OPT_IN environment variable enables a gradual transition. The key caveat is that the values for this variable are defined per convention. For HTTP, http/dup and http are defined; for other conventions, separate opt-in tokens are only available once they have been defined in the spec.

The diagram below uses the HTTP migration as an example.

http/dup mode emits both old and new attributes simultaneously. Storage costs increase temporarily, but the advantage is that services stay uninterrupted while the whole team migrates their queries. For DB or other convention migrations, you must first check the migration guide for that specific convention to find the supported opt-in tokens. HTTP-specific tokens do not automatically apply to DB.

Attribute transformation in the OTel Collector

There's also a pattern for handling renames in the Collector without touching application code.

# otel-collector-config.yaml
processors:
  attributes/http-migration:
    actions:
      - key: http.request.method
        from_attribute: http.method
        action: insert
      - key: http.response.status_code
        from_attribute: http.status_code
        action: insert
 
service:
  pipelines:
    traces:
      processors: [attributes/http-migration, batch]

This applies convention transformations without a code deployment, making it especially useful in environments with many legacy services.

Common tradeoffs when designing

Cardinality is a problem in both Resource and Span

It's well known that you shouldn't put high-cardinality values like user.id or order.id into Resource attributes. Resource is meant to express process-level identity, so only things with a small number of distinct values — like service.name or deployment.environment.name — belong there.

That said, moving them to Span attributes doesn't automatically make them safe. If you configure Tempo to index certain tags for fast search, and those tags are Span attributes containing high-cardinality values, index size and query overhead will grow together. In practice, the following approach is safe:

• Resource: low-cardinality values only (service.name, deployment.environment.name, etc.).
• Span: add as needed, but register only manageable-cardinality tags in Tempo's dedicated attribute index.
• For values that truly require individual request identification — like user.id or order.id — keep them on the span but default to looking them up through the trace body rather than the search index.

Tracking versions with schema_url

Attaching schema_url to Resource and Scope to declare the convention version gives backends a basis for handling conversions between versions.

// Attaching schema_url to Resource — semconv v1.26.0
res, _ := resource.New(ctx,
    resource.WithSchemaURL(semconv.SchemaURL),
    resource.WithAttributes(
        semconv.ServiceName("payment-service"),
    ),
)

Not all backends fully support automatic conversion yet, but declaring which convention version was used makes planning a migration significantly easier.

Closing: know in advance when this design breaks

Traces organized with Semantic Conventions work well — until they quietly fall apart at a few specific points. If any of the conditions below apply to your team's system right now, your dashboards may not be broken yet, but they soon could be.

• The same service is deployed with different service.name values across processes. The service map fragments and resource.service.name filters miss some traffic.
• http.route is attached to CLIENT spans without distinguishing CLIENT/SERVER. Aggregating by route in TraceQL mixes the caller side with the receiver side, inflating the numbers.
• A convention migration is in progress but only one side has been renamed without dup mode, so existing alert rules and new alert rules reference different names.
• A specific Span attribute is registered in Tempo's search index while high-cardinality values like user.id are flowing into that attribute.

Structuring traces well is ultimately about agreeing in advance on what name you'll use to find what when you query later. These four edge cases are the most common points where that agreement breaks down — a good starting point is to check them one by one in your next sprint.

References

• OpenTelemetry Semantic Conventions official documentation
• Trace Semantic Conventions — General
• Resource Semantic Conventions
• HTTP Semantic Convention Migration Guide
• Database Semantic Convention Migration Guide
• Grafana Tempo official documentation
• TraceQL query construction guide
• Tempo Metrics from Traces