llm-architectures-explained
← /learn · 01

The attention family

MHA, GQA, MQA, MLA, sliding windows, sparse attention, and linear and state-space hybrids.

Concept

The token mixer is the part of a block that lets a position read earlier positions. Since 2017 it has been the place where architectures differ most, and nearly every change has the same target: the cache a model keeps per sequence while it generates, and the work each new token does against it.

  • Multi-head attention (MHA), the original Transformer's: every query head has its own key and value head, so each layer caches two vectors per head per token.
  • Multi-query attention (MQA) keeps all the query heads but shares one key and one value head between them (Shazeer, 2019). PaLM used it.
  • Grouped-query attention (GQA) sits between the two: query heads in groups, one key/value head per group (Ainslie et al., 2023). Llama 3 8B has 32 query heads and 8 key/value heads.
  • Multi-head latent attention (MLA), from DeepSeek-V2, caches one compressed latent vector per token per layer, plus a small separate key for positions, and rebuilds every head's keys and values from it.
  • Sliding windows make most layers attend only to the most recent W tokens. Gemma 3 interleaves five local layers (a 1,024-token window) with one global layer.
  • Sparse attention keeps the whole cache but lets a cheap indexer pick which cached tokens each query reads (DeepSeek-V3.2 picks 2,048).
  • Linear attention and state-space layers (Gated DeltaNet, Lightning attention, Mamba) replace the growing cache with a fixed-size state. They are almost always mixed with some full-attention layers: Qwen3-Next uses three Gated DeltaNet layers per attention layer, Jamba seven Mamba layers per attention layer.

The interactive puts all eight on one body: Llama 3 8B's width, depth, feed-forward blocks and vocabulary, with only the token mixer swapped. So every difference you see is the mixer's alone. At a 128K-token context in BF16, GQA's cache is 16 GiB per sequence; MHA's is 4× that, 64 GiB; MQA's 2 GiB; MLA's 4.5 GiB. The sliding-window mix holds 2.61 GiB, and the Mamba-2 hybrid 2.06 GiB, almost all of it in its four global layers.

Loading the interactive…

Concept

Things to try.

  • Look at the chart from 1K to 1M. Every line ends up rising in step with the context, because every variant keeps some layers whose cache grows. The window, linear and state-space hybrids start almost flat, where their fixed-size part is all there is, and then rise from a much lower level, because only one layer in four, six or eight keeps a growing cache.
  • Compare decode FLOPs at 128K. The attention term is proportional to the tokens each query reads, so the sliding-window mix needs 26.2 GFLOP per token against GQA's 83.7. MLA needs more (101 GFLOP) in this cost model, which counts its attention in the expanded form: MLA saves memory, not arithmetic.
  • Switch the cache to FP8. Every cache halves; the recurrent state is counted at its own precision.
GQA: 86 models in the data set

Grouped-query attention: several query heads share one key/value head. Newest first; each links to its sourced page.

MLA: 24 models in the data set

Multi-head latent attention: keys and values cached as one low-rank latent vector. Newest first; each links to its sourced page.

Sliding window: 35 models in the data set

Some layers attend only to the most recent W tokens. Newest first; each links to its sourced page.

Sparse attention: 10 models in the data set

An indexer picks which cached tokens each query reads. Newest first; each links to its sourced page.

Hybrid: 27 models in the data set

More than one kind of token mixer in the stack. Newest first; each links to its sourced page.

Maths

Per token, summed over ll layers, a softmax-attention cache holds

MixerValues per token
MHA2 nh dh l2\,n_h\,d_h\,l
GQA2 nkv dh l2\,n_{kv}\,d_h\,l
MQA2 dh l2\,d_h\,l
MLA(dc+dhR) l(d_c + d_h^R)\,l
Window WW2 nkv dh l2\,n_{kv}\,d_h\,l, for the last WW tokens only

for nhn_h query heads of dimension dhd_h, nkvn_{kv} key/value heads, a compressed latent of dcd_c values and a decoupled rotary key of dhRd_h^R. For the Llama 3 8B body (l=32l = 32, nh=32n_h = 32, nkv=8n_{kv} = 8, dh=128d_h = 128) in BF16 that is 512 KiB per token for MHA, 128 KiB for GQA, 16 KiB for MQA and, with DeepSeek-V3's latent sizes (dc=512d_c = 512, dhR=64d_h^R = 64), 36 KiB for MLA.

A linear-attention or state-space layer instead keeps a state of fixed size: h⋅dk⋅dvh \cdot d_k \cdot d_v for a Gated DeltaNet layer with hh value heads, h⋅dh⋅Nh \cdot d_h \cdot N for a Mamba-2 layer with state size NN, plus a few tokens of convolution history. That is why the hybrids' curves flatten.

Decode FLOPs per token at context tt are 2Pactive2P_{\text{active}} for the matrix multiplications, plus, per attention layer, 2nh(dk+dv)2 n_h (d_k + d_v) for each cached token the query reads: tt for full attention, min⁡(t,W)\min(t, W) in a window, and the indexer's top-kk for sparse attention (whose indexer still scores all tt tokens with 2hIdI2 h_I d_I FLOPs each).

Code

// src/lib/arch/costModel.ts — the cache one layer adds per token (excerpt)
  if (t === "attn") {
    const kv = m.kv_heads!;
    const hd = m.head_dim!;
    const vd = m.v_head_dim || m.head_dim!;
    let e = m.k_eq_v ? kv * hd : kv * (hd + vd);

The variants are built in reference/chapter_model.py (attention_variants) and costed by the cost model in Python and in TypeScript; tests/unit/chapters/model.test.ts asserts that every number in the table is identical in the two.