Understanding FP8 KV Cache in vLLM
Understand what KV cache quantization changes and build a compatibility and quality check before enabling it.
On this page
Problem
Long-context or concurrent requests can put pressure on cache memory even when the model weights fit on the GPU.
Conclusion
FP8 KV cache changes the precision used for stored key and value tensors. It is separate from weight quantization. Check the attention backend, hardware and scale handling supported by your exact vLLM release before enabling it.
Environment
Record the vLLM release, attention backend, GPU, CUDA runtime, model revision and cache dtype. Compare against the same model and workload with the baseline cache configuration.
Symptoms
Cache allocation can limit the available request capacity. A change in cache dtype may also change numerical behavior or backend selection. No error output from an actual run is presented here.
Cause
The cache retains attention state for previously processed tokens. Lower-precision storage reduces the bytes used for eligible tensors, but the overall process also needs weights, activations and other allocations.
Solution
Read the quantized-cache documentation for your pinned release. The following fragment illustrates the option name; it requires a compatible installed environment and a model you have validated.
# Replace MODEL_ID with your validated checkpoint.
vllm serve "$MODEL_ID" --kv-cache-dtype fp8
Review how cache scales are obtained. Do not assume every checkpoint supplies appropriate scales. Run a short correctness test first, then compare representative long prompts, generation quality and memory behavior against the baseline.
Verification
This article provides a verification procedure; no hardware measurements are claimed.
Record peak memory, accepted request capacity and application-specific quality checks. A smaller cache representation does not by itself prove lower latency or higher useful throughput.
Caveats
Supported FP8 formats, scale options and attention behavior vary by backend and release. Avoid treating an old compatibility restriction as current without checking the versioned documentation.
Work through the memory arithmetic
For a conventional full-attention model, an unsharded cache estimate in bytes is:
2 × attention layers × KV heads × head dimension
× cached tokens × concurrent sequences × bytes per element
The factor of two counts keys and values. Use KV heads, not query heads, for grouped-query attention. This estimate does not apply unchanged to hybrid, sliding-window or latent-attention models.
For an illustrative architecture with 32 attention layers, 8 KV heads and a head dimension of 128, each token requires 131,072 bytes at two bytes per element. At 8,192 cached tokens, that is 1 GiB per independent sequence. One-byte storage reduces the raw tensor estimate to 0.5 GiB; four independent sequences need 4 GiB or 2 GiB respectively. Scales, allocation granularity and the rest of the process are excluded.
Try different inputs in the memory estimator. These are calculations, not measured GPU allocations.
Make a quality check that can fail
Choose prompts where a wrong answer matters: extracting a number from a long document, following a structured-output schema, or recalling a detail near the beginning of a long input. Keep the model, template and generation settings fixed. Define a pass criterion before looking at results; for example, exact agreement on a labeled extraction set rather than whether an answer looks fluent.
Record baseline and candidate outputs, startup warnings, the selected attention backend and actual cache dtype. If the configuration fails compatibility checks, return to the baseline rather than assuming a successful process start proves the intended precision is active. Memory savings and useful throughput are separate outcomes.
References
Revision note
September 14, 2026: added a worked decision framework and explicit verification limits. Original publication date retained.