Prefill and decode timeline
Cached decoding1. Prompt prefill
Process 2,048 prompt positions once. Create prompt keys and values for every layer and KV head.
→
2. Cached token decoding
Process one new query position. Append one new key and value, then reuse earlier cache entries.
Queries
Keys
Values
Step controls
Paused1 / 256
Generated token 1 · sequence length 2,049
KV-cache shape
[batch, layers, 2, kv_heads, sequence_length, head_dimension][1, 32, 2, 32, 2049, 128]
Keys[1, 32, 32, 2049, 128]Stored for every prior position
Values[1, 32, 32, 2049, 128]Stored with a separate factor of 1
Memory
Binary units onlyPrompt-only cache memory
Current cache memory
Final cache memory
Cache growth per generated token
KV caching reduces repeated compute but consumes additional memory. It does not reduce KV-cache memory.
Attention-work comparison
Attention score multiplicationsPrefill attention work
Cached work at current step
Uncached work at current step
Work saved
Total cached attention work
Total uncached attention work
Work reduction
Uncached-to-cached ratio
MHA, GQA, and MQA comparison
All parameters stay fixed except KV-head count| Mode | KV heads | Final KV-cache memory | Growth per token | Relative to MHA | Attention-score work |
|---|
Charts
Exact formulas, sampled only for drawingKV-cache memory vs sequence lengthLinear
Cache memory
Work per generation stepLinear vs quadratic
CachedUncached
Cumulative attention workCached includes one prefill
Cached totalUncached total
State explanation
Cached decodingThis tool models KV-cache memory and attention-score multiplications only. It excludes MLPs, projections, softmax, memory transfer, kernel launch, quantization, sampling, tokenization, and network latency.