llm-architectures-explained

/models

Chinchilla 70B

Google DeepMind · Chinchilla · closed weights

Facts and where they come from

Released2022-03paperpaperarXiv v1, March 2022
Licenceproprietarylabpaperweights not published
Total parameters70BpaperpaperTable 4 / abstract: Chinchilla, 70B parameters
Active parametersnot disclosednot disclosed
Context length2K tokenspaperpaperGopher paper: 2048-token context
Norm placementprecodemodelling codeGopher paper §3: RMSNorm instead of LayerNorm
Norm typeRMSNormcodemodelling codeGopher paper §3: RMSNorm instead of LayerNorm
QK-normnocodemodelling codeno q/k normalisation in the attention block
Positional encodingrelative position biaspaperpaperGopher paper §3: relative positional encoding
Parallel attention and MLPnocodemodelling codeGopher paper §3: RMSNorm instead of LayerNorm

Architecture, drawn from the data

MHA 64q/64kv. Each column is one layer: its token mixer above, its feed-forward block below. Paler columns reuse another layer’s keys and values.

Chinchilla 70B: layer stack and blockslayers (80)mixer / FFNlayer 0: MHA: 64 query / 64 KV heads · head 128layer 0: MLP: 32768layer 1: MHA: 64 query / 64 KV heads · head 128layer 1: MLP: 32768layer 2: MHA: 64 query / 64 KV heads · head 128layer 2: MLP: 32768layer 3: MHA: 64 query / 64 KV heads · head 128layer 3: MLP: 32768layer 4: MHA: 64 query / 64 KV heads · head 128layer 4: MLP: 32768layer 5: MHA: 64 query / 64 KV heads · head 128layer 5: MLP: 32768layer 6: MHA: 64 query / 64 KV heads · head 128layer 6: MLP: 32768layer 7: MHA: 64 query / 64 KV heads · head 128layer 7: MLP: 32768layer 8: MHA: 64 query / 64 KV heads · head 128layer 8: MLP: 32768layer 9: MHA: 64 query / 64 KV heads · head 128layer 9: MLP: 32768layer 10: MHA: 64 query / 64 KV heads · head 128layer 10: MLP: 32768layer 11: MHA: 64 query / 64 KV heads · head 128layer 11: MLP: 32768layer 12: MHA: 64 query / 64 KV heads · head 128layer 12: MLP: 32768layer 13: MHA: 64 query / 64 KV heads · head 128layer 13: MLP: 32768layer 14: MHA: 64 query / 64 KV heads · head 128layer 14: MLP: 32768layer 15: MHA: 64 query / 64 KV heads · head 128layer 15: MLP: 32768layer 16: MHA: 64 query / 64 KV heads · head 128layer 16: MLP: 32768layer 17: MHA: 64 query / 64 KV heads · head 128layer 17: MLP: 32768layer 18: MHA: 64 query / 64 KV heads · head 128layer 18: MLP: 32768layer 19: MHA: 64 query / 64 KV heads · head 128layer 19: MLP: 32768layer 20: MHA: 64 query / 64 KV heads · head 128layer 20: MLP: 32768layer 21: MHA: 64 query / 64 KV heads · head 128layer 21: MLP: 32768layer 22: MHA: 64 query / 64 KV heads · head 128layer 22: MLP: 32768layer 23: MHA: 64 query / 64 KV heads · head 128layer 23: MLP: 32768layer 24: MHA: 64 query / 64 KV heads · head 128layer 24: MLP: 32768layer 25: MHA: 64 query / 64 KV heads · head 128layer 25: MLP: 32768layer 26: MHA: 64 query / 64 KV heads · head 128layer 26: MLP: 32768layer 27: MHA: 64 query / 64 KV heads · head 128layer 27: MLP: 32768layer 28: MHA: 64 query / 64 KV heads · head 128layer 28: MLP: 32768layer 29: MHA: 64 query / 64 KV heads · head 128layer 29: MLP: 32768layer 30: MHA: 64 query / 64 KV heads · head 128layer 30: MLP: 32768layer 31: MHA: 64 query / 64 KV heads · head 128layer 31: MLP: 32768layer 32: MHA: 64 query / 64 KV heads · head 128layer 32: MLP: 32768layer 33: MHA: 64 query / 64 KV heads · head 128layer 33: MLP: 32768layer 34: MHA: 64 query / 64 KV heads · head 128layer 34: MLP: 32768layer 35: MHA: 64 query / 64 KV heads · head 128layer 35: MLP: 32768layer 36: MHA: 64 query / 64 KV heads · head 128layer 36: MLP: 32768layer 37: MHA: 64 query / 64 KV heads · head 128layer 37: MLP: 32768layer 38: MHA: 64 query / 64 KV heads · head 128layer 38: MLP: 32768layer 39: MHA: 64 query / 64 KV heads · head 128layer 39: MLP: 32768layer 40: MHA: 64 query / 64 KV heads · head 128layer 40: MLP: 32768layer 41: MHA: 64 query / 64 KV heads · head 128layer 41: MLP: 32768layer 42: MHA: 64 query / 64 KV heads · head 128layer 42: MLP: 32768layer 43: MHA: 64 query / 64 KV heads · head 128layer 43: MLP: 32768layer 44: MHA: 64 query / 64 KV heads · head 128layer 44: MLP: 32768layer 45: MHA: 64 query / 64 KV heads · head 128layer 45: MLP: 32768layer 46: MHA: 64 query / 64 KV heads · head 128layer 46: MLP: 32768layer 47: MHA: 64 query / 64 KV heads · head 128layer 47: MLP: 32768layer 48: MHA: 64 query / 64 KV heads · head 128layer 48: MLP: 32768layer 49: MHA: 64 query / 64 KV heads · head 128layer 49: MLP: 32768layer 50: MHA: 64 query / 64 KV heads · head 128layer 50: MLP: 32768layer 51: MHA: 64 query / 64 KV heads · head 128layer 51: MLP: 32768layer 52: MHA: 64 query / 64 KV heads · head 128layer 52: MLP: 32768layer 53: MHA: 64 query / 64 KV heads · head 128layer 53: MLP: 32768layer 54: MHA: 64 query / 64 KV heads · head 128layer 54: MLP: 32768layer 55: MHA: 64 query / 64 KV heads · head 128layer 55: MLP: 32768layer 56: MHA: 64 query / 64 KV heads · head 128layer 56: MLP: 32768layer 57: MHA: 64 query / 64 KV heads · head 128layer 57: MLP: 32768layer 58: MHA: 64 query / 64 KV heads · head 128layer 58: MLP: 32768layer 59: MHA: 64 query / 64 KV heads · head 128layer 59: MLP: 32768layer 60: MHA: 64 query / 64 KV heads · head 128layer 60: MLP: 32768layer 61: MHA: 64 query / 64 KV heads · head 128layer 61: MLP: 32768layer 62: MHA: 64 query / 64 KV heads · head 128layer 62: MLP: 32768layer 63: MHA: 64 query / 64 KV heads · head 128layer 63: MLP: 32768layer 64: MHA: 64 query / 64 KV heads · head 128layer 64: MLP: 32768layer 65: MHA: 64 query / 64 KV heads · head 128layer 65: MLP: 32768layer 66: MHA: 64 query / 64 KV heads · head 128layer 66: MLP: 32768layer 67: MHA: 64 query / 64 KV heads · head 128layer 67: MLP: 32768layer 68: MHA: 64 query / 64 KV heads · head 128layer 68: MLP: 32768layer 69: MHA: 64 query / 64 KV heads · head 128layer 69: MLP: 32768layer 70: MHA: 64 query / 64 KV heads · head 128layer 70: MLP: 32768layer 71: MHA: 64 query / 64 KV heads · head 128layer 71: MLP: 32768layer 72: MHA: 64 query / 64 KV heads · head 128layer 72: MLP: 32768layer 73: MHA: 64 query / 64 KV heads · head 128layer 73: MLP: 32768layer 74: MHA: 64 query / 64 KV heads · head 128layer 74: MLP: 32768layer 75: MHA: 64 query / 64 KV heads · head 128layer 75: MLP: 32768layer 76: MHA: 64 query / 64 KV heads · head 128layer 76: MLP: 32768layer 77: MHA: 64 query / 64 KV heads · head 128layer 77: MLP: 32768layer 78: MHA: 64 query / 64 KV heads · head 128layer 78: MLP: 32768layer 79: MHA: 64 query / 64 KV heads · head 128layer 79: MLP: 3276804079× 80normMHA: 64 query / 64 KV heads · head 128+normMLP: 32768+full attentiondense FFN

Modelled costs

From the cost model, batch size 1. Totals the lab states are in the table above; differences come from rounding, from what a lab counts, or from parts the model does not describe (listed on the about page).

Parameters (modelled)65B
Active per token (modelled)65B
Without embeddings and output head64.4B total, 64.4B active
KV cache per token, BF16 (layers that grow with context)2.5 MiB
KV cache + state at 2K tokens, BF165 GiB
Decode FLOPs per token at 4K context140 GFLOP
Prefill FLOPs for a 4K prompt550 TFLOP

KV cache against context

Chinchilla 70B: KV cache bytes against context length101001,0009.5 MiB95 MiB950 MiB9.3 GiBcontext (tokens)KV cache + state (BF16)Chinchilla 70B

Compare with other models →

Every architecture field

FieldValueSource
d_model8,192paperpaperTable 4: d_model 8,192
vocab32,000paperpaperGopher paper §3: SentencePiece vocabulary of 32,000
tied_embeddingsfalsepaperpapernot stated; untied assumed
mixers.full.typeattncodemodelling codeattention block
mixers.full.heads64paperpaperTable 4: 64 heads
mixers.full.kv_heads64paperpaperdata/transcribed/chinchilla-70b.json: num_key_value_heads
mixers.full.head_dim128paperpaperTable 4: key/value size 128
ffns.dense.typedensecodemodelling codeMLP block
ffns.dense.d_ff32,768paperpaperGopher paper (arXiv 2112.11446) §3: feed-forward size always 4 x d_model
ffns.dense.gatedfalsecodemodelling codeMLP: two matrices, no gate
layout80× full/densepaperpaperTable 4: layers 80

Sources