Why KV Cache Hit Rate Drops to 0% When Scaling Out vLLM Pods, and How llm-d Solves It (Prefix-Aware Routing / Distributed KV Cache)

When operating an LLM service, you will eventually encounter this situation. When you had Automatic Prefix Caching (APC) enabled in vLLM and ran a multi-turn chatbot on a single server, you could clearly feel requests getting faster from the second one onward — but after scaling up to 3 or 5 pods to handle increased traffic, that benefit completely disappears. Cache hit rates plummet, TTFT slows down again, and the expectation that "adding more GPUs will make things faster" starts to fall apart. At first I thought it was a configuration problem and spent quite a while troubleshooting, but this is not a configuration issue — it is a structural problem where KV cache locality fundamentally breaks in a distributed environment.

llm-d is an open-source framework that tackles this problem head-on. It runs on Kubernetes and decides which pod to send a request to not by round-robin, but based on "which server already holds the KV cache for this request." After reading this article, you will understand how prefix-aware routing differs from single-server APC, and you will have the basis to decide whether to apply llm-d to your own cluster.

One thing worth mentioning upfront: llm-d is a project being built collaboratively by Red Hat, IBM, Google, and NVIDIA, and it has been accepted into the CNCF Sandbox. It is in its early stages, but ecosystem support is coming together quickly.

Core Concepts

How Single-Server APC Works

vLLM's APC is a fairly elegant mechanism. When a prompt comes in, it divides the token sequence into blocks of a fixed size and attaches a hash value to each block. When the same prefix appears again, it retrieves the already-computed KV cache blocks directly from GPU memory and only computes the changed trailing portion.

Request A: [System prompt 512 tokens] + [User message 1]
           └── KV block hash: abc123 → stored in GPU memory
 
Request B: [System prompt 512 tokens] + [User message 2]
           └── KV block hash: abc123 → Cache hit! No recomputation needed

The problem is that this entire process is valid only within the memory boundary of a single instance. If there are multiple pods, each holds its own cache and they do not share with each other.

Why Cache Locality Disappears During Horizontal Scaling

Consider an environment with 3 pods (A, B, C). With a round-robin load balancer, requests are distributed in A → B → C → A order. In a multi-turn conversation, this is what happens:

Turn 1: Pod A → Computes [system prompt + conversation history 1] → stored in A's cache
Turn 2: Pod B → Recomputes [system prompt + conversation history 1 + 2] from scratch
Turn 3: Pod C → Recomputes [system prompt + conversation history 1 + 2 + 3] from scratch

Cache hit rate converges to 0%, and the more pods you add, the greater the waste.

KV Cache (Key-Value Cache): A space that stores the Key and Value vectors for each token in transformer attention computations. Since already-processed tokens do not need to be recomputed, the reuse benefit grows with longer contexts. Since vLLM's default block size is 16 tokens, the shared prefix must be at least 16 tokens for any cache benefit.

llm-d's Three Core Components

llm-d tracks "which pod holds the KV cache for this request" at the cluster-wide level and uses that information for routing.

llm-d-router: Hashes the prompt prefix of an incoming request, queries the global indexer, and sends the request to the pod that holds the cache. If no cache exists, it selects the pod with the lowest load.

KV Cache Indexer: Tracks the KV cache block status of all pods in the cluster in near-real-time. Whenever a vLLM pod stores or evicts a block, it publishes BlockStored / BlockRemoved events via a gRPC stream, and the indexer receives these to update its index. Because it uses a gRPC stream the overhead is low, but if network delays or queue backlogs occur, the indexer's state can diverge from the actual pod cache state over time (this is called Staleness).

Scorer: Selects the optimal pod by summing the prefix cache hit weight and the pod load weight. Our team initially set the prefix weight too high and had one pod with a popular system prompt go down due to OOM. There is a reason why both metrics need to be considered together.

Final score = (prefix hit weight × hit status) + (load weight × pod headroom)

Staleness: The phenomenon where the state information managed by the cache indexer diverges from the actual pod state due to timing differences. It is caused by network event delays or processing queue backlogs, and there is a possibility that a request judged as a hit may actually be a miss.

Approximate vs Precise Routing

llm-d supports two routing strategies.

It started with the Approximate approach, but since v0.3 introduced Precise Prefix Cache Aware Routing, directly querying the actual cache state has become more preferred. If Staleness is a concern, switching to Precise mode is the practical response.

The Dual Hash Space Problem — Digging Deeper

The "dual hash space problem" that appears in the list of drawbacks is a pitfall that is hard to appreciate until you have actually operated it in production. Let me explain it in more detail.

The llm-d router hashes the request token ID sequence to index "which pod holds this prefix." Meanwhile, the vLLM internal engine hashes the actual KV tensor content to identify blocks. These two hashes operate in different spaces.

The issue arises with BlockRemoved events. The event that vLLM emits when evicting a cache block only contains the engine's internal key (content hash), not the token sequence-based key used by the router. From the indexer's perspective, it knows "this block was removed," but it cannot immediately find which entry to delete from the index the router references — so a reverse-mapping step is required. Without deep knowledge of vLLM's internal structure, this is where the index can fall out of sync and incorrect routing can occur.

Unless you are building a custom indexer yourself, it is recommended to use the official indexer provided by llm-d as-is. This reverse-mapping logic is already built in.

Prefill-Decode Disaggregation: The Next Level of Performance

Since 2025, an architecture that physically separates the prefill phase (prompt processing) and the decode phase (token generation) onto different servers has been gaining attention in the industry. It has been confirmed that Meta, LinkedIn, and HuggingFace have already deployed this in production, and seeing that led me to conclude "this is no longer experimental." NVIDIA also announced a dedicated framework, Dynamo, at GTC 2025. llm-d supports this pattern as well, with a structure that transfers KV cache computed in prefill pods to decode pods over an RDMA network.

The natural question arises of whether ordinary Ethernet will work. The issue is the size of KV tensors. For a Llama 70B model, the KV cache for a single request can reach several GB, and this must be fully transferred before the decode phase begins. With ordinary Ethernet (10–25 Gbps), the transfer latency becomes a bottleneck that actually increases TTFT. RDMA networks such as InfiniBand or RoCE provide bandwidth of 200 Gbps or more and latency on the order of microseconds, and enable direct memory-to-memory transfer without going through the CPU, so they do not interfere with GPU computation. In cloud environments, there are services offering equivalent capabilities such as AWS EFA and Google Cloud GPUDirect RDMA.

RDMA (Remote Direct Memory Access): A technology that directly accesses the memory of a remote server over the network without going through the CPU. It requires datacenter-grade NICs (InfiniBand HCAs or RoCE-capable Ethernet cards) and dedicated switches. If you are not using on-premises or high-performance cloud instance types, separate verification is needed.

When first adopting llm-d, you can see substantial benefits from prefix-aware routing alone without Prefill-Decode disaggregation. The practical approach is to start there and progressively move to a disaggregated configuration once RDMA infrastructure is secured.

Practical Application

Example 1: Reducing TTFT in Multi-Turn Conversation Services

This is the scenario with the most noticeable impact. In a chatbot where users exchange multiple messages, the cost of recomputing the previous context from scratch grows as conversation history lengthens. Applying llm-d routes subsequent requests to the pod that holds the KV cache of the previous conversation context, so the earlier computation can be skipped.

In the YAML example below, llm-d-epp is llm-d's Endpoint Picker Pod and must be pre-installed in the cluster before deploying the InferencePool. Installing via the official Helm chart creates it automatically.

# llm-d InferencePool configuration example
apiVersion: inference.networking.x-k8s.io/v1alpha2
kind: InferencePool
metadata:
  name: llama-chat-pool
spec:
  targetPortNumber: 8000
  selector:
    matchLabels:
      app: llama-chat
  extensionRef:
    name: llm-d-epp  # prefix-aware router object auto-created on Helm install
---
apiVersion: inference.networking.x-k8s.io/v1alpha2
kind: InferenceModel
metadata:
  name: llama-chat
spec:
  modelName: "meta-llama/Llama-3.1-8B-Instruct"
  criticality: Standard
  poolRef:
    name: llama-chat-pool

According to measured data from Red Hat's OpenShift AI environment, applying llm-d to multi-turn workloads reduced P99 TTFT from 18.3 seconds to 8 seconds — less than half. There is also a case with up to 57× reduction compared to round-robin in an 8-pod (16× H100) environment, but this was measured under conditions where conversation history was very long and workloads sharing the same system prompt were concentrated. The magnitude of the effect will vary depending on workload characteristics.

Example 2: Multi-Tenant Environment with a Shared Common System Prompt

Honestly, I was initially skeptical about this scenario — "how much difference can it really make?" — but when you actually apply it, the numbers come out quite dramatically. Multiple customers use the same platform, and there is a pattern where each shares a long system prompt (internal RAG context, codebase summaries, etc.). In this case, concentrating routing so that a specific pod maintains the KV cache for the same prefix eliminates the waste of repeating identical computations across the cluster.

For any cache benefit, the shared prefix must be at least the vLLM block size (default 16 tokens), but in cases like the one below with system prompts of several thousand tokens, the cache reuse effect is substantial.

import openai
 
client = openai.OpenAI(
    base_url="http://llm-d-gateway/v1",  # replace only base_url with llm-d gateway
    api_key="..."
)
 
# Common system prompt (RAG context of 3000+ tokens)
# This prefix serves as the basis for cache routing
SHARED_SYSTEM_PROMPT = """
You are an assistant that answers based on XXX company's internal documents.
[... document context totaling 3000 tokens ...]
"""
 
response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-70B-Instruct",
    messages=[
        {"role": "system", "content": SHARED_SYSTEM_PROMPT},
        {"role": "user", "content": user_message}
    ]
)
# llm-d-router automatically routes to the pod holding the cache
# based on the SHARED_SYSTEM_PROMPT prefix hash

Simply replacing base_url with the llm-d gateway address in existing vLLM OpenAI-compatible code will work as-is. The near-zero client code changes lowers the barrier to adoption.

Example 3: Tiered KV Cache Offloading

When GPU memory fills up, situations arise where cache must be evicted. Since v0.5, llm-d supports tiered offloading that moves KV cache blocks down in the order GPU → CPU → SSD → shared filesystem, with the router aware of the location of each tier for routing purposes.

The ConfigMap below is connected as a volume mount to the llm-d vLLM pod, and is used by mounting it at the /etc/llm-d/config.yaml path in the pod spec's volumeMounts.

# llm-d KV cache offloading configuration (mounted as /etc/llm-d/config.yaml in pod spec)
apiVersion: v1
kind: ConfigMap
metadata:
  name: llm-d-kv-config
data:
  config.yaml: |
    kv_cache:
      tiered_offload:
        enabled: true
        tiers:
          - type: gpu_memory
            capacity_gb: 40
          - type: cpu_memory
            capacity_gb: 200
          - type: local_ssd
            path: /mnt/nvme/kv-cache
            capacity_gb: 1000

Compared to running with GPU cache at 100% capacity, cases of 25% reduction in TTFT and 21% improvement in throughput have been reported.

Pros and Cons Analysis

Advantages

Disadvantages and Caveats

Most Common Mistakes in Practice

1. Leaving Scorer weights at their defaults: The prefix sharing ratio and per-pod load variance differ for each workload. If hit rates are low early on, try increasing the prefix weight; if a specific pod is observed to be overloaded, try increasing the load weight.

2. Leaving single-server APC enabled simultaneously and double-relying on it: If llm-d routes correctly, single-server APC will work alongside it, but APC itself does not compensate for llm-d routing. If routing is wrong, single APC is also useless.

3. Trying to introduce Prefill-Decode disaggregation from the start: Attempting a disaggregated configuration without RDMA infrastructure can actually degrade performance due to network bottlenecks. You can see significant benefits from prefix-aware routing alone, so a gradual approach is practical.

Closing Thoughts

In distributed LLM inference, adding more GPUs alone cannot improve cache efficiency — performance improves superlinearly only when KV cache locality is explicitly managed at the routing layer.

Here are 3 steps you can start with right now.

1. Set up a local environment with the llm-d quickstart: Use the command below to bring up a minimal configuration on a Kubernetes cluster and directly compare TTFT against a standalone vLLM deployment. This is the v0.5 installation command, so for later versions it is recommended to check the latest command in the official GitHub quickstart guide.

   bash

   helm install llm-d llm-d/llm-d --namespace llm-d  # as of v0.5

2. Design a multi-turn benchmark scenario: Create a request set that shares the same system prompt and conversation history, then measure the cache hit rate and TTFT difference between round-robin and prefix-aware routing numerically. Data is more persuasive than intuition and can also serve as justification for an adoption decision.

3. Start experimenting with Scorer weights: Analyze your production traffic patterns — if your workload has a high prefix sharing ratio, start with a higher prefix weight; if pod load balancing is more important, adjust the load weight to find the optimal balance point.

References

• KV-Cache Wins You Can See: From Prefix Caching in vLLM to Distributed Scheduling with llm-d | llm-d official blog
• Precise Prefix Cache Aware Routing official documentation | llm-d.ai
• Feature: Precise Prefix Cache Aware Routing | GitHub
• Master KV cache aware routing with llm-d for efficient AI inference | Red Hat Developer
• Accelerate multi-turn LLM workloads on OpenShift AI with llm-d | Red Hat Developer
• llm-d-kv-cache | GitHub
• llm-d main repository | GitHub
• llm-d 0.5: Sustaining Performance at Scale | llm-d official blog
• Automatic Prefix Caching | vLLM official documentation
• Prefix-aware routing | Ray Serve documentation
• Production-Grade LLM Inference at Scale with KServe, llm-d, and vLLM | llm-d blog
• Native KV Cache Offloading to Any Filesystem with llm-d | llm-d blog