INFERENCE LAB / BENCHMARKS
Benchmark methods
Performance claims need a workload, an environment and the raw evidence.
Build a benchmark you can audit.
Use this protocol and the downloadable recording sheet to compare serving configurations. No measured scores are presented here.
What every report will include
| Measurement | Reporting requirement |
|---|---|
| TTFT | Time to first token, including measurement boundaries |
| TPOT | Time per output token; report the distribution |
| Prefill / decode throughput | Separate input processing from output generation |
| VRAM usage | Peak and steady-state memory, with measurement method |
| Workload | Concurrency, context length, output lengths and arrival pattern |
| Environment | Model, GPU, framework, quantization, TP, versions and commands |
A repeatable run sequence
- Freeze a workload file and record its checksum.
- Check outputs before measuring speed.
- Separate startup, warm-up and measurement windows.
- Run cold-prefix and warm-prefix cases separately.
- Repeat each configuration and record errors, latency distributions and output tokens.
Download a blank benchmark recording sheet (CSV)
Compare like with like
Record the same model, GPU, precision, tensor parallelism, context and concurrency for each comparison. Report failed requests alongside successful ones.
Read our engine evaluation method →