Ashish Pandey

The KV cache: why generation is fast at all

Ashish Pandey — Sat, 19 Sep 2026 18:30:00 GMT

The model stops redoing work it has already done. Nothing is approximated — it just keeps notes in the margin instead of re-reading the whole chart every time.

Doing it all again, every time

Generation is one word at a time, and each word attends to everything before it.

To write word 500, the model considers all 499 words before it. To write word 501, it does all 500 again. Then 501 again for the next one. The same arithmetic, over and over.

A doctor writing a long note, who re-reads the patient’s entire chart before adding each new line. Twenty lines in, she’s read the chart twenty times. Or — she could jot the key facts in the margin as she goes, and only read the new line.

What the trick is worth

I measured both approaches on the same model, generating the same text:

Approach	Time taken
Re-reading everything each time (no cache)	2.97 seconds
Keeping notes (KV cache)	0.18 seconds

Sixteen times faster — and the output was word-for-word identical. Nothing was approximated or skipped. It simply stopped doing the same work twice.

That’s the rare kind of optimisation: free. The bill arrives as memory.

Why K and V, but never Q

From the attention chapter, every token produces a query, a key and a value. When the model generates a new token:

The new token’s query is compared against the keys of every token so far.
Those keys and values have not changed since they were computed, and never will.
The new query, once used, is thrown away. It’s never needed again.

So you cache K and V, not Q. That’s the entire idea, and it’s why it’s called the KV cache.

Without it, generating tokens costs work overall. With it, — each step computes one new K and V and attends over the stored ones.

It grows with every token

where the 2 is K and V, is layers, is key/value heads, is head dimension, and is tokens so far. It grows linearly with every token generated, for every concurrent user.

Play with it — these defaults are roughly an 8B-class model in fp16:

viewof layers = Inputs.range([8, 128], {value: 32, step: 1, label: "Layers"})
viewof kvheads = Inputs.range([1, 64], {value: 8, step: 1, label: "KV heads (8 = GQA, 32 = full MHA)"})
viewof headdim = Inputs.range([32, 256], {value: 128, step: 32, label: "Head dimension"})
viewof ctx = Inputs.range([1000, 128000], {value: 8000, step: 1000, label: "Context length (tokens)"})
viewof bits = Inputs.radio([16, 8], {value: 16, label: "Cache precision (bits)"})

perUserGB = 2 * layers * kvheads * headdim * ctx * (bits / 8) / 1e9
freeGB = 80 - 16            // an 80GB GPU, minus ~16GB of model weights
users = Math.max(0, Math.floor(freeGB / perUserGB))
html`
  Cache per user
    ${perUserGB.toFixed(2)} GB
  Users on one 80GB GPU
    ${users < 8 ? "#e76f51" : "#2a9d8f"}">${users}
    after ~16GB of weights
`

Push the context to 128k and watch how few users fit. This is why long context is expensive to serve, not just to compute.

Two phases, two bottlenecks

Phase	What happens	Limited by
Prefill	The whole prompt is processed at once, filling the cache	Compute — it’s one big matrix multiply
Decode	One token at a time, each step dragging the entire cache through memory	Memory bandwidth

This is the single most useful mental model for inference performance. Time-to-first-token is a prefill problem; tokens-per-second after that is a bandwidth problem. Adding compute rarely fixes slow decoding — during decode a GPU spends most of its time waiting on memory, not calculating.

How the cache is made smaller

GQA / MQA. Several query heads share one set of keys and values. Multi-query attention is the extreme (one KV head). Grouped-query attention is the common middle ground — it’s why the slider above defaults to 8 KV heads rather than 32, and it shrinks the cache by that same 4×.
Sliding-window attention. Only keep the last tokens’ keys and values, capping the cache no matter how long the conversation runs. You trade away exact long-range recall.
Quantised cache. Store K and V in 8 bits instead of 16. Halves the memory for a small quality cost — try the toggle above.
PagedAttention (vLLM). Don’t shrink the cache; stop wasting it. Allocate in small pages like virtual memory instead of one contiguous block per request, and fragmentation mostly disappears.
Prefix sharing. Many requests start with the same system prompt. Compute that prefix’s cache once and share it across requests.

The same trick, one level up

Prompt caching in a hosted API is this idea at the product layer: a long, unchanging system prompt gets its cache computed once and reused, so you’re billed a fraction for those tokens on later calls. Same principle — stop redoing old work — different altitude.

Where it breaks

Memory, not compute, is usually your serving limit. Batch size is capped by cache size. People buy faster GPUs when they needed more memory or a smaller cache.
The cache is per-conversation. A long chat history is re-billed and re-carried on every turn. Trim aggressively.
Change anything early in the prompt and the cache dies. Cached prefixes are only valid while the prefix is byte-identical. A timestamp at the top of a system prompt quietly destroys every prefix hit.
Sliding windows forget. If a fact left the window, no amount of attention brings it back. It’s not “compressed” — it’s gone.

Whiteboard check

Marker in hand, someone watching. Could you get through these without notes?

What exactly is stored in a KV cache, and why not the queries?

The key and value vectors for every past token, at every layer and every KV head. Past keys and values are reused at every future step and never change. A query is used once, at the step that created it, and is then irrelevant.

Estimate the KV cache for a 32-layer model, 8 KV heads, head dim 128, fp16, at 8k context.

bytes ≈ 1.05 GB per request. On an 80GB GPU with ~16GB of weights, that’s around 60 concurrent users before memory runs out — which is why batch size, not FLOPs, usually sets your throughput.

Your time-to-first-token is fine but tokens-per-second is poor. What do you investigate?

That’s decode, so suspect memory bandwidth, not compute: cache size per request, batch size, GQA vs MHA, cache precision, and whether paged attention is in use. Bigger GPUs with the same bandwidth won’t help; larger batches (amortising weight reads) usually will.

Does a KV cache change the model’s output?

No. It’s exact — identical tokens, identical probabilities, just without recomputation. That’s what separates it from sliding-window or quantised variants, which do trade quality for memory.

TL;DR

Cache the keys and values of past tokens; queries are single-use.
16× faster, identical output. Pure win on compute, paid for in memory.
Cache size grows linearly with tokens and users: .
Prefill is compute-bound; decode is bandwidth-bound. Diagnose them separately.
GQA, sliding windows, quantisation and paged attention all exist to fight the same bill.

← Attention

Attention: how a model decides what matters

Ashish Pandey — Sat, 19 Sep 2026 18:30:00 GMT

Attention repaints every word using the words around it, taking most of its paint from the words that matter most.

The one job

Underneath every chatbot, assistant and clever demo, there is a single operation:

Given the words so far, guess what comes next.

That’s it. Everything else is that trick, repeated: write a word, add it to the sentence, guess again.

And it doesn’t give one answer. It scores every word it knows — tens of thousands of them. I gave a model “The patient has”:

Next word	Confidence
been	20.1%
a	8.3%
not	4.6%
no	3.9%
had	2.2%

Look carefully: there’s no medicine in that list. No fever, no diabetes, no chest pain. The model knows *“The patient has ___“* needs a verb. It has no idea it’s standing in a hospital.

To make a good guess, it needs one more thing: a way to work out which of the earlier words actually matter right now. That’s attention.

Mixing paint

Take this half-finished sentence:

Patient has severe chest ___

Which earlier words matter for the next word?

Earlier word	How much it matters
severe	85%
patient	10%
has	2%

Every word is a colour. The word chest starts out plain grey — on its own it could be a chest of drawers, a treasure chest, or a chest X-ray. The word severe is red: urgent, intense. Attention mixes the paint, stirring 85% of severe’s red into chest’s grey. So chest comes out reddish.

Drag the slider and watch the word change colour — and change meaning:

viewof att = Inputs.range([0, 100], {value: 85, step: 1, label: "attention paid to “severe”"})

mixed = d3.interpolateRgb("#9aa0a6", "#e63946")(att / 100)
html`
  chest
  +
  severe · ${att}%
  →
  ${mixed};color:#fff;padding:.55rem 1rem;border-radius:8px;font-weight:600">chest
  next word likely ${att > 50 ? "“pain”" : "“of drawers”"}
`

In “chest of drawers”, drawers is brown, so chest comes out brownish instead. Same word in, different colour out, depending on its neighbours.

And here’s the part people miss: when the model makes its guess, it doesn’t look at the word. It looks at the colour the word ended up as. Reddish chest leads to pain. Brownish chest leads to drawers.

Queries, keys and values

Each word produces three different vectors, by multiplying its embedding with three learned weight matrices:

	Question it answers	In the analogy
Query (Q)	“What am I looking for?”	chest asking: which word tells me what kind of chest I am?
Key (K)	“What do I offer?”	severe advertising: I’m an intensity word
Value (V)	“What do I pass on?”	the actual red paint that gets mixed in

The mechanism is three steps:

Score. Compare every query against every key with a dot product. High score = this word is relevant to me.
Normalise. Divide by , then softmax. That turns raw scores into the percentages you saw: 85%, 10%, 2%. They always sum to 100%.
Mix. Take a weighted average of the values using those percentages. The result replaces the word’s representation.

That’s the whole formula, and you already know every piece of it. The paint being mixed is V. The percentages come from Q·K.

Why divide by ?

Dot products of long vectors grow large. Feed a large number into softmax and it saturates: one word gets 99.9% and the rest get nothing, so gradients vanish and learning stalls. Dividing by keeps the scores in a sane range. It’s a numerical fix, not a deep idea — but interviewers love asking about it.

It cannot look forward

While generating text, a word may only attend to words before it. Future positions are set to before the softmax, so they get exactly 0% of the paint.

Without that mask, predicting the next word would be trivial: the model would have already seen the answer. This is the causal mask, and it’s the difference between GPT-style decoders and BERT-style encoders (which do see both directions, which is why BERT is good at understanding text but can’t generate it well).

toks = ["Patient", "has", "severe", "chest", "___"]
weights = [
  [1.00, 0,    0,    0,    0],
  [0.30, 0.70, 0,    0,    0],
  [0.15, 0.10, 0.75, 0,    0],
  [0.10, 0.02, 0.60, 0.28, 0],
  [0.10, 0.02, 0.85, 0.03, 0]
]
cells = weights.flatMap((row, i) => row.map((w, j) => ({q: toks[i], k: toks[j], w, masked: j > i})))
Plot.plot({
  marginLeft: 80, marginBottom: 60, height: 300, width: 480,
  x: {label: "attends to →", domain: toks, tickRotate: -30},
  y: {label: "word doing the attending", domain: toks},
  color: {scheme: "Greens", domain: [0, 1], legend: true, label: "share of the paint"},
  marks: [
    Plot.cell(cells.filter(d => !d.masked), {x: "k", y: "q", fill: "w", inset: 1}),
    Plot.cell(cells.filter(d => d.masked), {x: "k", y: "q", fill: "#f1f1f1", inset: 1}),
    Plot.text(cells.filter(d => !d.masked), {x: "k", y: "q", text: d => d.w ? d.w.toFixed(2) : "", fill: d => d.w > 0.5 ? "white" : "black", fontSize: 11}),
    Plot.text(cells.filter(d => d.masked), {x: "k", y: "q", text: () => "—", fill: "#bbb"})
  ]
})

Read a row: it’s one word choosing where to get its paint. The grey cells are the future, blocked out. Notice the last row — the word being predicted takes 85% of its paint from severe, which is exactly the table from earlier.

(Weights here are illustrative, to show the shape of a real attention matrix.)

Many heads, not one

One attention head mixes paint one way. Real models run dozens in parallel — each with its own Q, K and V matrices, each free to learn a different kind of relationship. One head might track which adjective modifies which noun; another, who she refers to; another, which bracket closes which.

Each head works in a smaller space (), their outputs are concatenated, and a final matrix mixes them back together. Same total compute, many more relationships captured.

Attention has no sense of order

Look at the formula again: it’s a weighted sum. Shuffle the words and you’d get the same answer. Attention on its own has no idea about word order — “dog bites man” and “man bites dog” would be identical.

Order is added separately, as positional information baked into the vectors (modern models mostly use RoPE, which rotates Q and K by an angle that depends on position). That’s its own chapter.

Why long context gets expensive

Every word attends to every other word. words means scores.

Tokens	Attention scores
1,000	1 million
10,000	100 million
100,000	10 billion

Double the text, quadruple the work. This one line explains most of what long-context pricing looks like, and most of the research into making attention cheaper (FlashAttention makes it memory-efficient without changing the maths; sliding-window and sparse attention change the maths to look at less).

Where it breaks

Attention weights are not explanations. A high weight shows where the paint came from, not why the model answered as it did. Treating attention maps as interpretability is a known trap — Jain & Wallace (2019) made this case sharply.
The middle gets lost. With long inputs, models reliably use the beginning and end more than the middle. Put what matters at the edges of the prompt.
More context isn’t more understanding. Attention dilutes: with 100k tokens competing, each relevant one gets a thinner slice of the paint.
The paint analogy has a limit. Mixing colours is lossy and commutative; attention operates in hundreds of dimensions where “mixtures” stay separable. It’s a picture of the weighting, not of the geometry.

Whiteboard check

Marker in hand, someone watching. Could you get through these without notes?

Explain self-attention to someone non-technical, in under a minute.

Every word starts with a rough meaning. Attention lets each word look at the others and ask “which of you matters to me?”, then blends in a bit of each one in proportion to the answer. “Chest” next to “severe” ends up meaning something different from “chest” next to “drawers”. The model then predicts from the blended meaning, not the original word.

Why scale by the square root of d_k?

Dot products grow with dimension; large inputs push softmax into saturation, where one position takes almost all the weight and gradients vanish. Scaling by keeps the variance of the scores roughly constant so training stays stable.

What breaks if you remove the causal mask during training of a decoder model?

The model can see the token it’s meant to predict, so loss drops to near zero and it learns nothing useful. At inference the future doesn’t exist, so the distribution it faces is completely different and generation falls apart.

Why multiple heads instead of one big head?

A single softmax-weighted average forces one relationship per position. Multiple heads in lower-dimensional subspaces let the model attend to several kinds of relationship at once — syntactic, coreference, positional — at the same total cost.

Where does the quadratic cost come from, and how do people avoid it?

Every query is scored against every key: time and, naively, memory. FlashAttention keeps the exact result while avoiding materialising the full matrix; sliding window, sparse and linear attention approximate by restricting or reformulating who can attend to whom.

TL;DR

Attention = weighted averaging. Queries and keys set the weights, values are what gets mixed.
Softmax makes weights sum to 1; keeps them numerically sane.
The causal mask stops the model seeing the future — that’s what makes generation possible.
Many heads capture many kinds of relationship at once.
Cost is quadratic in sequence length. That’s the whole long-context story.

← Tokens The KV cache →

Tokens: how text becomes numbers

Ashish Pandey — Sat, 19 Sep 2026 18:30:00 GMT

A machine cannot read words. Text is chopped into pieces, and each piece becomes a number. Everything downstream — cost, memory, even accuracy — is decided by how well that chopping goes.

A box of LEGO bricks

Imagine you can only speak using a fixed box of LEGO bricks. A common word like weather has its own brick: you just grab it. An unusual word has no brick of its own, so you build it out of several smaller ones.

That box is the model’s vocabulary, typically 32,000 to 200,000 bricks. Every piece of text you send gets built out of those bricks, and each brick has a number.

Here’s what actually happened when I fed two words into a well-known model:

Word	Pieces
`weather`	`weather`	one piece
`hyperlipidemia`	`hyper` · `lip` · `id` · `emia`	four pieces

Notice something odd. A person would split that word as hyper–lipid–emia. The machine split it as hyper–lip–id–emia. That lip is the lip on your face — nothing to do with fats in the blood.

The machine didn’t choose pieces that carry meaning. It chose pieces that are common on the internet. And the internet talks about lips far more than lipids.

Count what it costs

Specialist language shatters into more pieces than everyday language. I measured pieces per word on the same model, for two kinds of text:

viewof words = Inputs.range([50, 2000], {value: 400, step: 50, label: "Length of the document (words)"})
viewof kind = Inputs.radio(["Everyday English", "Clinical notes"], {value: "Clinical notes", label: "Kind of text"})

rates = ({"Everyday English": {general: 1.08, medical: 1.17}, "Clinical notes": {general: 1.97, medical: 1.37}})
gen = Math.round(words * rates[kind].general)
med = Math.round(words * rates[kind].medical)
Plot.plot({
  height: 150, marginLeft: 110, marginRight: 60,
  x: {label: "Tokens the model actually processes →", domain: [0, Math.max(gen, med) * 1.15], grid: true},
  y: {label: null},
  color: {legend: false},
  marks: [
    Plot.barX([{m: "General model", v: gen}, {m: "Medical model", v: med}],
      {y: "m", x: "v", fill: d => d.m === "General model" ? "#e76f51" : "#2a9d8f"}),
    Plot.text([{m: "General model", v: gen}, {m: "Medical model", v: med}],
      {y: "m", x: "v", text: d => d.v.toLocaleString(), dx: 8, textAnchor: "start", fontWeight: "bold"})
  ]
})

md`A **${words}-word** ${kind.toLowerCase()} document costs **${gen.toLocaleString()} tokens**
on a general model and **${med.toLocaleString()}** on a medical one —
a **${Math.round((gen / med - 1) * 100)}%** difference for exactly the same text.`

Nobody designed this vocabulary

The vocabulary is learned from a corpus, usually with byte-pair encoding (BPE): start with individual characters, then repeatedly merge the most frequent adjacent pair into a new token. Do that 50,000 times and you have a vocabulary.

The consequence is the whole chapter in one line: frequency in the training corpus, not meaning, decides what gets its own token.

What specialist language costs

Same two models, two kinds of text:

	Everyday English	Clinical notes	Penalty for medicine
General model	1.08	1.97	+82%
Medical model	1.17	1.37	+17%

A medical-domain tokenizer cuts the cost of clinical text by about 30%, and pays for it by being slightly worse at everyday English. That’s not a flaw — it’s a choice about which world the model lives in.

thrombocytopenia
  general model  →  th · rom · b · ocy · top · en · ia     (7 pieces)
  medical model  →  thrombocytopenia                        (1 piece)

Seven fragments, or one whole idea. That’s a dangerously low platelet count — exactly the kind of thing you want a medical system to understand as one concept.

I looked inside both vocabularies. The general model reserved a slot for harbaugh, an American football coach’s surname. The medical model spent its slot on leishmaniasis, a tropical parasitic disease. Neither is wrong. But you wouldn’t want the football dictionary reading your blood report.

The three bills

Money. These systems bill per token. A 400-word clinical note is ~790 tokens instead of ~430. Every note, forever.

Memory. A model holds a fixed number of tokens at once. Whatever that limit is, you fit roughly half as much patient history when the text is specialist.

Meaning. This is the one that hurts. The model isn’t reading “hyperlipidemia” — it’s reading four unrelated fragments and rebuilding the idea from scratch, every single time it appears.

Where it breaks

You can’t swap in a better tokenizer. The model’s entire learned memory is indexed by those token numbers. Change the numbering and every word points at the wrong meaning. A specialist vocabulary means retraining — a real project, not a setting.
Numbers and code tokenize badly. 2026 may be one token or three; that’s part of why models fumble arithmetic.
Non-English text pays a tax. Languages under-represented in the training corpus need far more tokens per word — sometimes 2–3×, for identical meaning. The same API call costs more in Hindi than in English.
Token boundaries hide characters. “How many r’s in strawberry?” is hard partly because the model sees two or three chunks, not eleven letters.

Whiteboard check

Marker in hand, someone watching. Could you get through these without notes?

What is BPE and why is it used instead of word-level or character-level tokenization?

Byte-pair encoding starts from characters and repeatedly merges the most frequent adjacent pair. Word-level vocabularies can’t handle unseen words and get enormous; character-level sequences are far too long for attention, which costs quadratic time. BPE sits in between: a fixed-size vocabulary, no out-of-vocabulary words (anything can be spelled out from smaller pieces), and sequences short enough to process.

Your RAG system over clinical documents is slower and pricier than expected. Where do you look first?

Tokens per word. Specialist text can nearly double the token count of the same content, which inflates embedding cost, retrieval chunk sizes and prompt length all at once. Measure the ratio on your own corpus before assuming the model or the retriever is at fault.

Would a domain-specific tokenizer fix a model that’s bad at medical text?

It fixes the representation problem, not the knowledge problem, and only if you retrain or continue pretraining — the vocabulary can’t be swapped into an existing model. Expect a ~30% reduction in tokens for clinical text, and slightly worse performance on general text.

TL;DR

Text becomes pieces, and pieces become numbers. Pieces are chosen by frequency, not meaning.
Specialist language shatters worst: +82% tokens for clinical text on a general model.
A domain tokenizer cuts that to +17%, but it must be baked in from the start.
Cost, context limits and accuracy all trace back to this one step.

Attention: how a model decides what matters →