Lab / Transformers
Every attention variant keeps the same engine: queries are scored against keys, the scores become weights, and the weights mix the values. The names differ in which of seven questions they answer.
softmax(QKᵀ / √d) VInteractive field guide
Change the mask, share the key-value cache, then assemble choices across all seven parts of the design.
01 / Connectivity
02 / Heads and KV cache
03 / Seven questions
A real model may use one choice from every row. The resulting combination is a design description, not a new species requiring a press release.
Field guide
Use these to place any attention name. FlashAttention is execution, RoPE is position, GQA is heads and cache.
The rule that scores a query against a key. Scaled dot product maps cleanly onto matrix multiplication; additive attention uses a small learned network; cosine attention compares direction only.
The mask decides which pairs exist. Full attention fills the square, causal attention is a triangle that hides the future, local attention is a band near the diagonal. Cross-attention takes keys and values from another sequence.
MHA gives every query head its own key and value head. MQA shares one across all of them, GQA shares one per group. MLA compresses keys and values into a smaller latent instead. Each trades KV-cache memory against variety.
Full attention compares n × n pairs, so doubling the input can quadruple the work. Local and sparse patterns compare fewer pairs; linear attention keeps a running state instead of the full matrix, with different recall behaviour.
FlashAttention computes exact attention but tiles the work so it stays in fast on-chip memory. PagedAttention manages the serving KV cache in blocks. Neither changes which token attends to which.
Plain attention sees a set. Sinusoidal and learned embeddings add a vector per position; RoPE rotates query and key features so their dot product carries relative position.
Text is a sequence, images are patch grids, video adds time, graphs define neighbours by edges. The mask and position scheme should respect that structure.
Read a model
A model name rarely maps to one attention type. Read it as one answer per axis.
A decoder language model. Its choices sit on three different axes, connectivity, heads and position, so they combine rather than compete.
MLA shrinks the key-value representation used during inference. That is mainly a heads-and-cache decision inside a much larger architecture.
Recurrent Kimi Delta Attention layers mixed with full MLA layers. It changes how work and memory grow at long context while keeping some full attention.
Which scoring rule changed? Which connections are allowed? How many heads, and what is cached? How does cost grow? Is it maths, a kernel or memory management? Where does position enter? What structure is the input?