SGLang RadixAttention: How to Boost RAG Pipeline Throughput 5x by Reusing KV Cache for Identical Document Blocks

If you've ever deployed a RAG pipeline to production, you've probably experienced this: you send dozens of requests with only the question changing while the same set of documents stays fixed, yet the GPU recomputes those same documents from scratch every single time, as if seeing them for the first time. If you've been staring at a cost monitoring dashboard thinking "something's wrong here," your instinct is right. I remember initially thinking "the KV cache takes care of it," then being genuinely shocked when I actually profiled it.

SGLang's RadixAttention can automatically reuse the KV cache for identical document blocks — with no code changes, just by structuring your prompts correctly — boosting throughput up to 5–6x. After reading this, you'll be able to check cache hit rates with just 3 lines of code and reduce TTFT (Time To First Token) noticeably by changing the order of one prompt element.

SGLang, developed by LMSYS, is an inference framework designed for structured LLM programs like agents, RAG, and multi-turn workflows, rather than simple single-prompt inference. While vLLM also has an Automatic Prefix Caching (APC) feature, the internal implementation differs from SGLang's RadixAttention. The differences and when to choose which will be covered in more depth in the next installment of this series.

Core Concepts

KV Cache, and Why Reuse Is Difficult

When a transformer model processes a request, it first runs through a phase called prefill, where it processes the entire input sequence at once. During this phase, it computes the Key-Value attention values for each token and loads them into memory. The KV cache is then read out during the decode phase, where tokens are generated one at a time.

The problem is that this cache disappears when the request ends. Even if the next request contains the same document block as the previous one, the conventional approach recomputes everything from scratch. When you think about the structure of a RAG pipeline, it's immediately clear how wasteful this is.

# Original approach — full prefill recomputation on every request
Request A: [System Prompt] + [Documents 1,2,3] + [Question A] → full computation
Request B: [System Prompt] + [Documents 1,2,3] + [Question B] → full recomputation (waste)
Request C: [System Prompt] + [Documents 1,2,3] + [Question C] → full recomputation (waste)
 
# RadixAttention — KV cache reuse for shared prefix
Request A: [System Prompt] + [Documents 1,2,3] + [Question A] → full computation + cache saved
Request B:  ↑ KV cache hit  ↑ KV cache hit  + [Question B] → only question portion computed
Request C:  ↑ KV cache hit  ↑ KV cache hit  + [Question C] → only question portion computed

The RAG pipeline structure — where retrieved documents are fixed and only the user question changes — maps perfectly onto this pattern.

Radix Tree — The Core of Automatic Cache Reuse

The "Radix" in RadixAttention comes from the Radix Tree, a trie variant data structure. The SGLang runtime manages processed sequences using this tree structure.

Radix Tree: A tree that compresses and stores common prefixes of strings (here, token sequences). "apple" and "application" share the same node up to "appl". In RadixAttention, token sequences act as strings, and each node has the KV cache pages for its corresponding segment attached to it.

When a new request arrives, the runtime traverses the tree to automatically find the Longest Common Prefix. The cache-hit segment skips prefill, and only the remaining segment is computed. When cache space runs low, older entries are evicted using an LRU (Least Recently Used) policy.

This may seem complex at first, but the important point from a developer's perspective is that this entire process is fully automatic. There is no need to manually manage cache keys or write code to check for cache hits.

HiCache — Extending the Cache Beyond the GPU

Introduced in SGLang v0.5+ in 2025, HiCache extends RadixAttention into a three-tier architecture, making it possible to secure a much larger cache pool beyond GPU memory limits.

There are measured results showing that when integrated with DeepSeek 3FS KVStore, the cache hit rate rose from 40% to 80% and the average session TTFT decreased by 56%. This is an option worth considering for cluster-scale RAG services.

Practical Application

The examples below are based on SGLang v0.3 and above. The @sgl.function decorator API may change across versions, so it's a good idea to check current version compatibility in the official documentation before applying it in practice.

Example 1: Multi-Query RAG — The Most Dramatic Scenario

This is the situation where multiple questions are asked against the same document set. Legal document analysis, product manual QA, and report-based chatbots all fall into this category.

import sglang as sgl
 
@sgl.function
def rag_query(s, documents: str, question: str):
    s += sgl.system("You are a document-based QA assistant.")
    s += sgl.user(f"Reference documents:\n{documents}\n\nQuestion: {question}")
    s += sgl.assistant(sgl.gen("answer", max_tokens=512))
 
runtime = sgl.Runtime(model_path="meta-llama/Llama-3.1-8B-Instruct")
sgl.set_default_backend(runtime)
 
shared_docs = """
Article 1 (Purpose of Contract) This contract sets forth the terms of software development services between Party A and Party B.
Article 2 (Contract Period) The contract period shall be from March 1, 2024 to December 31, 2024.
Article 3 (Termination Conditions) If either party breaches the contract, the other party may terminate it with 30 days' written notice.
"""
 
questions = [
    "What are the termination conditions of this contract?",
    "Please explain the penalty clause.",
    "How long is the contract period?",
]
 
# The KV cache for the shared_docs segment is saved on the first request,
# and subsequent requests reuse that segment
results = [rag_query(documents=shared_docs, question=q) for q in questions]
 
for q, r in zip(questions, results):
    print(f"Q: {q}")
    print(f"A: {r['answer']}\n")

Example 2: Few-Shot Classification Pipeline

This is the pattern of reusing the same example block across hundreds of requests. The longer the examples, the greater the RadixAttention effect — benchmarks have shown cache hit rates reaching up to 99%. Honestly, when I first saw this figure I thought it might be an exaggeration, but after fixing the few-shot block and measuring it myself, it made sense.

@sgl.function
def classify_document(s, few_shot_examples: str, target_text: str):
    s += sgl.system("Classify the document into one of the following categories: Contract, Report, Email, Other")
    s += sgl.user(
        f"Classification examples:\n{few_shot_examples}\n\n"
        f"Document to classify:\n{target_text}\n\n"
        f"Category:"
    )
    s += sgl.assistant(sgl.gen("category", max_tokens=10))
 
# few_shot_examples is fixed, only target_text changes
few_shots = """
[Example 1] "Party A and Party B enter into this agreement effective January 1, 2024..." → Contract
[Example 2] "Q3 revenue increased 12% year-over-year..." → Report
[Example 3] "Re: Please confirm your availability for next week's kickoff meeting..." → Email
"""
 
# List of documents to classify
documents_to_classify = [
    "This service agreement takes effect on January 1, 2025, with a contract term of one year...",
    "Operating profit this quarter declined 8% compared to the prior quarter, with the primary cause being...",
    "Hello, I wanted to check if you are available for a meeting this Friday at 3:00 PM...",
]
 
results = [
    classify_document(few_shot_examples=few_shots, target_text=doc)
    for doc in documents_to_classify
]

Example 3: Maintaining Multi-Turn Conversation Context

As conversation history grows longer, the KV cache for previous conversation segments can be reused to reduce response latency. There is one behavior worth knowing about in advance: because the conversation_history string grows with each turn, not the entire previous conversation is a complete cache hit. The history accumulated up to the previous turn reuses the cache, and only the newly added segment undergoes prefill computation. The structure means the proportion of reused segments grows as the conversation lengthens.

The code below is a simplified demo to illustrate the working principle. In a production environment, it's more natural to implement this as a FastAPI endpoint or an asynchronous handler.

@sgl.function
def chat_turn(s, history: str, new_message: str):
    s += sgl.system("You are a friendly customer support assistant.")
    if history:
        s += sgl.user(f"Previous conversation:\n{history}\n\nNew message: {new_message}")
    else:
        s += sgl.user(new_message)
    s += sgl.assistant(sgl.gen("response", max_tokens=256))
 
# Simplified demo loop (recommended to replace with an async handler in production)
conversation_history = ""
while True:
    user_input = input("User: ")
    if user_input.lower() == "quit":
        break
    result = chat_turn(history=conversation_history, new_message=user_input)
    response = result["response"]
    print(f"Assistant: {response}")
 
    # History accumulation — cache hit expected for previous segments
    conversation_history += f"User: {user_input}\nAssistant: {response}\n"

Checking Cache Hit Rate Directly

After running the code, you can immediately check how well the cache is actually working. Use the /metrics endpoint of the SGLang runtime.

import requests
 
# Check while the SGLang runtime is running
metrics_text = requests.get("http://localhost:30000/metrics").text
 
# Filter only cache_hit-related metrics
cache_lines = [line for line in metrics_text.split('\n') if 'cache_hit' in line.lower()]
for line in cache_lines:
    print(line)

If the hit rate is below 60%, first inspect the prompt structure (placement and ordering of dynamic values). If it's above 60% but performance improvement is marginal, consider introducing HiCache.

Pros and Cons Analysis

Advantages

• Throughput improvement: Up to 5–6x throughput improvement over baseline systems on prefix-sharing workloads
• TTFT reduction: Cache-hit segments skip prefill computation → significant reduction in time to first token
• Performance vs. vLLM: 29% higher throughput than vLLM on H100 (16,200 vs. 12,500 tok/s)
• Automatic management: Runtime automatically detects and reuses cache without manual cache key management
• Memory efficiency: KV cache reuse allows handling larger batch sizes
• Scalability: Hierarchical scaling from GPU → CPU → distributed storage via HiCache

Disadvantages and Caveats

There is a specific reason GIL contention is particularly problematic for ML workloads. CPU-based Python routing happens in parallel with GPU computation preparation, and as the number of concurrent requests grows, the GPU can sit idle while Python threads become the bottleneck.

The Most Common Mistakes in Practice

In practice, when you actually adopt SGLang, you encounter these situations more often than expected.

1. Inserting dynamic values in the middle of the prompt

SGLang's cache key is byte-for-byte identity of the token sequence. If values that change with every request — like timestamps, session IDs, or usernames — are embedded in the middle of the system prompt or document block, every token after that point becomes a new sequence. The difference between the two structures below makes this immediately clear.

# Bad example — timestamp inserted in the middle causes cache miss for everything after it
[System Prompt] + [Request Time: 2024-01-15 14:23:11] + [Documents 1,2,3] + [Question]
 
# Good example — dynamic values go at the end
[System Prompt] + [Documents 1,2,3] + [Question] + [Request Time: 2024-01-15 14:23:11]

2. Applying it unconditionally without analyzing the workload

If you apply SGLang to a service that sends completely different prompts on every request — just because you heard "SGLang is fast" — there will be almost no advantage over vLLM. It is far more efficient to first measure the cache hit rate with /metrics and then decide on the optimization direction.

3. Tuning other hyperparameters without first checking cache hits

Before adjusting parameters like batch size or chunk size, it helps to first verify how much the cache is actually being reused. If the hit rate is low, inspect the prompt structure first. If the hit rate is high but performance improvement is marginal, you can then look into GPU memory capacity or consider introducing HiCache.

Closing Thoughts

Automatically reusing the KV cache for repeated document blocks in a RAG pipeline — that is the core of what RadixAttention solves. Before changing the model or adding hardware, properly structuring your prompt alone can yield a meaningful difference in throughput and response speed.

Three steps you can start right now:

1. Install SGLang and verify basic operation: If you're starting fresh, try a basic pip install sglang installation first, or use a Docker image to get started quickly without build errors. sglang[all] includes Flash Attention, triton, and more, but may throw build errors depending on your environment. It's a lower barrier to entry to verify behavior with the base installation and then add the necessary extras afterward.

2. Review your prompt structure: It's worth checking whether your existing RAG prompt is fixed in the order [System Prompt] → [Retrieved Documents] → [User Question]. If dynamic values are appearing before or in the middle of the document block, simply moving them to the end can raise your cache hit rate.

3. Measure cache hit rate and make a decision: You can check the actual hit rate via the /metrics endpoint. If the hit rate is below 60%, inspect the prompt structure first. If it's 60% or above, it's time to consider introducing HiCache.

Have you ever measured this directly on your own RAG workload? It would be great if you could share in the comments which patterns gave you the best cache hit rates.

References

• Fast and Expressive LLM Inference with RadixAttention and SGLang | LMSYS Blog
• SGLang: Efficient Execution of Structured Language Model Programs (NeurIPS 2024)
• SGLang arXiv Paper (2312.07104)
• SGLang GitHub Repository
• SGLang HiCache: Fast Hierarchical KV Caching | LMSYS Blog
• HiCache System Design and Optimization | SGLang Docs
• SGLang Learning Series Part 1: Shared Prefix, KV Cache, and RadixAttention | Medium
• When to Choose SGLang Over vLLM: Multi-Turn Conversations and KV Cache Reuse | RunPod
• SGLang vs vLLM in 2026: Benchmarks, Architecture, and When to Use Each | Particula
• RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation | ACM
• Welcome to SGLang | Official Documentation