Fourth Deep Dive post, and another different layer: FlashAttention made one attention computation cheaper, PagedAttention changed how KV cache memory gets allocated. Speculative decoding attacks something else entirely — the number of times you have to run the full model at all.

Why decoding is the expensive part

Generating a response token by token means running a full forward pass through the model for every single token. During that forward pass, the GPU has to move the model's weights (and the growing KV cache) from memory to the compute units — and for a large model, that transfer dominates the time far more than the actual arithmetic does. Decoding one token at a time is memory-bandwidth-bound, not compute-bound.

That has a strange consequence: scoring several candidate tokens in one forward pass costs barely more than scoring one, because the expensive part — loading the weights — happens exactly once either way. Speculative decoding exists specifically to exploit that.

The idea

Use a small, cheap model (the "draft" model) to quickly propose several tokens ahead, generating them one at a time the normal slow way — but since it's small, that's fast. Then feed all of those candidates into the big model (the "target" model) at once, checking them all in a single forward pass instead of one sequential pass per token.

Diagram showing a small draft model proposing four candidate tokens sequentially, the large target model verifying all four in one parallel pass, and the resulting accept/reject/resample outcome

The part that makes this exact, not approximate

The obvious worry: doesn't checking a smaller model's guesses risk changing what actually gets generated? It doesn't — and the reason is a specific rejection-sampling rule, from Leviathan et al. and Chen et al. (both 2023):

Each draft token y is accepted with probability

β(y) = min(1, p_target(y) / p_draft(y))

If the target model was at least as likely to produce that token as the draft model was, it's accepted outright. If the target model liked it less, it's accepted only sometimes — proportional to how much less. When a token is rejected, the algorithm doesn't just fall back to greedy decoding — it samples a replacement from the residual distribution:

r(·) ∝ max(p_target(·) − p_draft(·), 0)

— the probability mass the target model wanted to assign that the draft model under-weighted. Every draft token after the rejection point gets thrown away, since the sequence has now diverged. The proof (in both papers) shows this combination makes the final output distributed identically to sampling from the target model alone — speculative decoding is a wall-clock optimization, not a quality tradeoff.

What it costs when it works

Both papers report solid, not dramatic, speedups: Leviathan et al. measured 1.6–2.8x on T5, and Chen et al. measured 2–2.5x decoding speedup on Chinchilla (70B) in a distributed setup. The ceiling depends entirely on the acceptance rate — how often the draft model's guesses actually match what the target model would have picked. A draft model that's too different from the target gets rejected constantly, and all that speculative generation is wasted work; a draft model too close in cost to the target eats into the savings even when it is accepted. The technique only pays off in the middle: cheap enough to speculate fast, aligned enough to be right often.

Where this sits in the stack

This is the same pattern the PagedAttention post ended on, from one more angle: FlashAttention makes each attention computation cheaper, PagedAttention makes memory allocation nearly waste-free, and speculative decoding reduces how many sequential target-model steps a response takes in the first place — three completely different bottlenecks, all shipping in the same modern serving stacks (vLLM supports speculative decoding directly), because none of them make the others unnecessary.

References

Flow diagram above is original artwork made for this post.