Prefill & decode timeline
Segment widths are schematic (compressed for readability); the labelled token counts are exact.
KV-cache shape
Memory
KV caching reduces repeated compute but consumes additional memory — it never reduces KV-cache memory. The cache grows linearly with sequence length.
Attention-score multiplications
Counts attention-score multiplications (Q·Kᵀ dot-product multiplies) only — no projections, MLP, softmax, memory traffic, or measured latency/FLOPs.
Cache memory vs sequence length
linear scale
Work per generation step
cached vs uncached · log-10 y-scale
Cumulative work
cached (incl. prefill) vs uncached · log-10 y-scale