How the Transformer Actually Works left one sentence unexplained: most modern LLMs moved on from sinusoidal position encoding to something called RoPE. The trick behind it is almost mechanical — rotate the query and key vectors by an angle proportional to position, and relative position falls out of the dot product for free.
Static batching forces a GPU to wait for the slowest sequence in a batch before it can serve anything new. Orca's iteration-level scheduling — the idea vLLM calls continuous batching — schedules at the granularity of a single decoding step instead, and the throughput gap is not small.
The last three posts built a text encoder and an image encoder separately. CLIP's entire contribution is refusing to keep them separate: train both at once so a caption and its matching image land at the same point in a shared embedding space, and classification falls out for free.
Split an image into patches, treat each patch as a token, and feed the sequence into the exact same Transformer encoder from the NLP posts on this blog. No convolutions, no vision-specific architecture — just enough data to make up for what CNNs get for free.
The previous post started from 'token embeddings' without saying where tokens come from. Byte-Pair Encoding builds a vocabulary by repeatedly merging the most frequent adjacent pair — a small, mechanical rule that ends up deciding how expensive every prompt is.
Self-attention, in one formula: softmax(QKᵀ/√d_k)V. Everything else in the Transformer — multi-head splitting, positional encoding, the residual stack — exists to make that one computation usable at scale.
A small draft model guesses several tokens ahead. The real model checks all of them in one pass, for roughly the cost of generating a single token. The output distribution doesn't change — only how many decode steps it takes to get there.
Existing LLM serving systems were wasting 60–80% of KV cache memory before a single token got generated. PagedAttention borrows a 60-year-old idea from operating systems — virtual memory paging — to bring that down to under 4%.
FP16 lets you feed one WGMMA's output straight into the next — the layouts already match. FP8 doesn't get that for free: the accumulator and the next operand disagree on which thread owns which value, and fixing that costs real shuffle instructions.
WGMMA showed up unexplained in two earlier posts — as a row in a diagram, then as a building block of FlashAttention-3. Here's what it actually is: an async, 128-thread-wide matrix multiply, and the register layout that makes it usable.
Three papers, three years, three different bottlenecks. FlashAttention wasn't 'made faster' three times in a row — each version targeted something the previous one left on the table.
Every CuTe layout is just a pair — a Shape and a Stride. That one idea, applied recursively, is what lets CUTLASS 3.x describe a whole GEMM's tiling — from the full matrix down to a single thread's registers — with one abstraction instead of a class per level.
cuBLAS gives you a fast, opaque matrix multiply. CUTLASS gives you the same speed as composable C++ building blocks, so you can fuse, customize, and specialize — matched to the GPU's own execution and memory hierarchy.
Tensor Cores in Volta, sparsity in Ampere, a Transformer Engine in Hopper, FP4 in Blackwell — the architectural changes that actually mattered, one generation at a time.
Works on My GPU