LLMs, NLP, computer vision, and general ML research.
How the Transformer Actually Works left one sentence unexplained: most modern LLMs moved on from sinusoidal position encoding to something called RoPE. The trick behind it is almost mechanical — rotate the query and key vectors by an angle proportional to position, and relative position falls out of the dot product for free.
The last three posts built a text encoder and an image encoder separately. CLIP's entire contribution is refusing to keep them separate: train both at once so a caption and its matching image land at the same point in a shared embedding space, and classification falls out for free.
Split an image into patches, treat each patch as a token, and feed the sequence into the exact same Transformer encoder from the NLP posts on this blog. No convolutions, no vision-specific architecture — just enough data to make up for what CNNs get for free.
The previous post started from 'token embeddings' without saying where tokens come from. Byte-Pair Encoding builds a vocabulary by repeatedly merging the most frequent adjacent pair — a small, mechanical rule that ends up deciding how expensive every prompt is.
Self-attention, in one formula: softmax(QKᵀ/√d_k)V. Everything else in the Transformer — multi-head splitting, positional encoding, the residual stack — exists to make that one computation usable at scale.
Works on My GPU