GPU architecture, CUDA, and low-level performance optimization.
A small draft model guesses several tokens ahead. The real model checks all of them in one pass, for roughly the cost of generating a single token. The output distribution doesn't change — only how many decode steps it takes to get there.
Existing LLM serving systems were wasting 60–80% of KV cache memory before a single token got generated. PagedAttention borrows a 60-year-old idea from operating systems — virtual memory paging — to bring that down to under 4%.
FP16 lets you feed one WGMMA's output straight into the next — the layouts already match. FP8 doesn't get that for free: the accumulator and the next operand disagree on which thread owns which value, and fixing that costs real shuffle instructions.
WGMMA showed up unexplained in two earlier posts — as a row in a diagram, then as a building block of FlashAttention-3. Here's what it actually is: an async, 128-thread-wide matrix multiply, and the register layout that makes it usable.
Three papers, three years, three different bottlenecks. FlashAttention wasn't 'made faster' three times in a row — each version targeted something the previous one left on the table.
Every CuTe layout is just a pair — a Shape and a Stride. That one idea, applied recursively, is what lets CUTLASS 3.x describe a whole GEMM's tiling — from the full matrix down to a single thread's registers — with one abstraction instead of a class per level.
cuBLAS gives you a fast, opaque matrix multiply. CUTLASS gives you the same speed as composable C++ building blocks, so you can fuse, customize, and specialize — matched to the GPU's own execution and memory hierarchy.
Tensor Cores in Volta, sparsity in Ampere, a Transformer Engine in Hopper, FP4 in Blackwell — the architectural changes that actually mattered, one generation at a time.
Works on My GPU