Prefix Cache vs KV Cache in LLM Inference
Distinguish the attention state stored during generation from reuse of that state across requests.
On this page
Problem
KV cache and prefix caching are often discussed together, leading to the mistaken expectation that enabling prefix reuse reduces every phase of generation.
Conclusion
KV cache stores previously computed key and value state. Prefix caching reuses eligible cached state when a new request shares a previously processed prefix. These describe related but different responsibilities.
Environment
Record the engine and version, model revision, prompt tokenization, prefix-caching settings, cache state and workload order. Use both repeated-prefix and unrelated-prefix requests.
Symptoms
Repeated long prefixes may benefit while unrelated prompts do not. A warm-cache test and a cold-cache test can produce different results despite using the same visible request text.
Cause
Reuse avoids recomputing eligible shared-prefix state. It does not eliminate generation of new output tokens. Cache capacity, eviction and exact prefix matching affect observed behavior.
Solution
Design two workload groups. One reuses a stable system prompt or document prefix; the other uses distinct prefixes. Keep output-length controls and arrival conditions consistent.
# Workload sketch: no performance measurements are implied.
shared_prefix = "A stable document or system prompt\n"
questions = ["Summarize the constraints.", "List the assumptions."]
requests = [shared_prefix + question for question in questions]
Measure the first cold request separately from subsequent warm requests. Capture TTFT and decode metrics independently. Repeat with a working set larger than the retained cache to understand eviction behavior.
Verification
This article provides a verification procedure; no hardware measurements are claimed.
A useful report states whether cache state was warm or cold, which prefixes were repeated and which phase improved. Do not attribute every throughput change to prefix reuse.
Caveats
Exact cache matching rules and isolation behavior belong to the serving engine. Applications with multiple tenants should review the engine's cache-isolation controls and their security requirements.
Two requests, one reusable prefix
Consider a document-questioning service. Put the stable document before the question, and keep the tokenization and chat template fixed. Request A asks for a summary; request B asks for the document's assumptions. Reusing the prefix is possible only for eligible state that remains available. A timestamp inserted at the beginning can change the prefix even if most of the visible text is identical.
| Observation | Next check | Avoid concluding |
|---|---|---|
| The second request starts faster | Compare cold and warm TTFT under the same load | Decode is faster too |
| No improvement with similar text | Inspect tokenized prefix and cache eviction | Caching is broken |
| Improvement disappears under load | Record working-set size and request order | The model became slower |
An illustrative time budget
Suppose prefill takes 400 ms and decode takes 1,600 ms in a hypothetical workload. Even eliminating all prefill would reduce total time from 2,000 to 1,600 ms: a 20% reduction, or 1.25× speedup. Real caching still has overhead and usually reuses only part of the work. This arithmetic explains why a large TTFT improvement can coexist with a modest end-to-end improvement; it is not an engine benchmark.
For your comparison, record queue time separately where possible. A faster second request may reflect reduced contention rather than prefix reuse. Use the benchmark recording sheet to retain the workload and cache state with each result.
References
Revision note
September 14, 2026: added a conceptual diagram, a worked time budget and a cache troubleshooting table. Original publication date retained.