KV Cache Debugger

Models KV-cache memory and attention-score multiplications only · seq_len(t) = prompt + t · not full runtime simulation

Configuration

Prompt tokens 2048

Generated tokens 256

Batch size 1

Transformer layers 32

Query attention heads 32

KV heads 32

Invalid head combination

Head dimension 128

Data type BF16

KV cache

Enabled — cached decoding

Attention mode: Multi-Head Attention

Presets

Prefill & decode timeline Prefill + cached decode

Prompt: 2048 Generated: 256 Decode step t: 1 Sequence length: 2049 Processed now: —

Prompt (prefill / full recompute) Cached keys/values (reused) New token (this step) Already generated Not yet generated

Speed

t = 1 / 256 · seq = 2049

KV-cache shape

[1, 32, 2, 32, 2049, 128]

batch

layers

K&V

kv_heads

seq_len

head_dim

Keys

[B, L, Hkv, S, D]

Values

[B, L, Hkv, S, D]

Shape = [batch, layers, 2, kv_heads, sequence_length, head_dimension]. Factor 2 = keys + values. Memory uses KV heads, not query heads.

Memory panel

KV caching reduces repeated compute but consumes additional memory. It does not reduce KV-cache memory — the cache stores keys and values for every prior token.

Attention-work comparison

Metric: Attention score multiplications only. Not measured latency or FLOPs. Work uses query heads; independent of KV-head count.

Charts

KV-cache memory vs sequence length

Work per generation step (cached vs uncached)

Cumulative work (cached vs uncached)

MHA / GQA / MQA comparison

Same config except KV-head count. Attention-score work stays constant; cache memory scales with KV heads.

Mode	KV heads	Final cache	Growth / token	vs MHA	Total cached work