Tools
Small, focused utilities for planning LLM deployments.
Weight and KV cache estimator
For conventional full-attention MHA/GQA models. The defaults are an illustrative architecture, not a named checkpoint.
Weights only 14.90 GiB
KV cache only 1.00 GiB
Subtotal 15.90 GiB
Formula and limitations
Weights = parameters × bits ÷ 8. KV bytes = 2 × layers × KV heads × head dimension × cached tokens × concurrent requests × bytes per element. Divide bytes by 2³⁰ for GiB.
This excludes activations, quantization scales, CUDA graphs, allocator overhead and communication buffers. It assumes independent requests with no prefix sharing. It does not model sliding windows, MLA, recurrent or hybrid architectures, or GPU sharding. A subtotal that fits is not proof that a deployment will fit.
Read the cache precision guideUse the estimate to design a test
- Read layer count, KV heads and head dimension from the exact model configuration.
- Use input plus expected output tokens for the cache length.
- Start with one request, then estimate your target concurrency.
- Measure actual device memory after loading and during a representative workload.
Do not apply this standard attention formula to hybrid or latent-attention architectures. Hugging Face explains different cache strategies.