llm-architectures-explained

/models

PaLM 540B

Google · PaLM · closed weights

Facts and where they come from

Released2022-04paperpaperarXiv v1, April 2022
Licenceproprietarylabpaperweights not published
Total parameters540BpaperpaperTable 1: 540.35B parameters
Active parametersnot disclosednot disclosed
Context length2K tokenspaperpaper§3: sequence length 2048
Norm placementparallelcodemodelling codePaLM paper §2: parallel layers
Norm typeLayerNormcodemodelling codePaLM paper §2: parallel layers
QK-normnocodemodelling codeno q/k normalisation in the attention block
Positional encodingRoPEpaperpaper§2: RoPE embeddings
Parallel attention and MLPyescodemodelling codePaLM paper §2: parallel layers

Architecture, drawn from the data

MQA 48q/1kv. Each column is one layer: its token mixer above, its feed-forward block below. Paler columns reuse another layer’s keys and values.

PaLM 540B: layer stack and blockslayers (118)mixer / FFNlayer 0: MQA: 48 query / 1 KV heads · head 256layer 0: Gated MLP: 73728layer 1: MQA: 48 query / 1 KV heads · head 256layer 1: Gated MLP: 73728layer 2: MQA: 48 query / 1 KV heads · head 256layer 2: Gated MLP: 73728layer 3: MQA: 48 query / 1 KV heads · head 256layer 3: Gated MLP: 73728layer 4: MQA: 48 query / 1 KV heads · head 256layer 4: Gated MLP: 73728layer 5: MQA: 48 query / 1 KV heads · head 256layer 5: Gated MLP: 73728layer 6: MQA: 48 query / 1 KV heads · head 256layer 6: Gated MLP: 73728layer 7: MQA: 48 query / 1 KV heads · head 256layer 7: Gated MLP: 73728layer 8: MQA: 48 query / 1 KV heads · head 256layer 8: Gated MLP: 73728layer 9: MQA: 48 query / 1 KV heads · head 256layer 9: Gated MLP: 73728layer 10: MQA: 48 query / 1 KV heads · head 256layer 10: Gated MLP: 73728layer 11: MQA: 48 query / 1 KV heads · head 256layer 11: Gated MLP: 73728layer 12: MQA: 48 query / 1 KV heads · head 256layer 12: Gated MLP: 73728layer 13: MQA: 48 query / 1 KV heads · head 256layer 13: Gated MLP: 73728layer 14: MQA: 48 query / 1 KV heads · head 256layer 14: Gated MLP: 73728layer 15: MQA: 48 query / 1 KV heads · head 256layer 15: Gated MLP: 73728layer 16: MQA: 48 query / 1 KV heads · head 256layer 16: Gated MLP: 73728layer 17: MQA: 48 query / 1 KV heads · head 256layer 17: Gated MLP: 73728layer 18: MQA: 48 query / 1 KV heads · head 256layer 18: Gated MLP: 73728layer 19: MQA: 48 query / 1 KV heads · head 256layer 19: Gated MLP: 73728layer 20: MQA: 48 query / 1 KV heads · head 256layer 20: Gated MLP: 73728layer 21: MQA: 48 query / 1 KV heads · head 256layer 21: Gated MLP: 73728layer 22: MQA: 48 query / 1 KV heads · head 256layer 22: Gated MLP: 73728layer 23: MQA: 48 query / 1 KV heads · head 256layer 23: Gated MLP: 73728layer 24: MQA: 48 query / 1 KV heads · head 256layer 24: Gated MLP: 73728layer 25: MQA: 48 query / 1 KV heads · head 256layer 25: Gated MLP: 73728layer 26: MQA: 48 query / 1 KV heads · head 256layer 26: Gated MLP: 73728layer 27: MQA: 48 query / 1 KV heads · head 256layer 27: Gated MLP: 73728layer 28: MQA: 48 query / 1 KV heads · head 256layer 28: Gated MLP: 73728layer 29: MQA: 48 query / 1 KV heads · head 256layer 29: Gated MLP: 73728layer 30: MQA: 48 query / 1 KV heads · head 256layer 30: Gated MLP: 73728layer 31: MQA: 48 query / 1 KV heads · head 256layer 31: Gated MLP: 73728layer 32: MQA: 48 query / 1 KV heads · head 256layer 32: Gated MLP: 73728layer 33: MQA: 48 query / 1 KV heads · head 256layer 33: Gated MLP: 73728layer 34: MQA: 48 query / 1 KV heads · head 256layer 34: Gated MLP: 73728layer 35: MQA: 48 query / 1 KV heads · head 256layer 35: Gated MLP: 73728layer 36: MQA: 48 query / 1 KV heads · head 256layer 36: Gated MLP: 73728layer 37: MQA: 48 query / 1 KV heads · head 256layer 37: Gated MLP: 73728layer 38: MQA: 48 query / 1 KV heads · head 256layer 38: Gated MLP: 73728layer 39: MQA: 48 query / 1 KV heads · head 256layer 39: Gated MLP: 73728layer 40: MQA: 48 query / 1 KV heads · head 256layer 40: Gated MLP: 73728layer 41: MQA: 48 query / 1 KV heads · head 256layer 41: Gated MLP: 73728layer 42: MQA: 48 query / 1 KV heads · head 256layer 42: Gated MLP: 73728layer 43: MQA: 48 query / 1 KV heads · head 256layer 43: Gated MLP: 73728layer 44: MQA: 48 query / 1 KV heads · head 256layer 44: Gated MLP: 73728layer 45: MQA: 48 query / 1 KV heads · head 256layer 45: Gated MLP: 73728layer 46: MQA: 48 query / 1 KV heads · head 256layer 46: Gated MLP: 73728layer 47: MQA: 48 query / 1 KV heads · head 256layer 47: Gated MLP: 73728layer 48: MQA: 48 query / 1 KV heads · head 256layer 48: Gated MLP: 73728layer 49: MQA: 48 query / 1 KV heads · head 256layer 49: Gated MLP: 73728layer 50: MQA: 48 query / 1 KV heads · head 256layer 50: Gated MLP: 73728layer 51: MQA: 48 query / 1 KV heads · head 256layer 51: Gated MLP: 73728layer 52: MQA: 48 query / 1 KV heads · head 256layer 52: Gated MLP: 73728layer 53: MQA: 48 query / 1 KV heads · head 256layer 53: Gated MLP: 73728layer 54: MQA: 48 query / 1 KV heads · head 256layer 54: Gated MLP: 73728layer 55: MQA: 48 query / 1 KV heads · head 256layer 55: Gated MLP: 73728layer 56: MQA: 48 query / 1 KV heads · head 256layer 56: Gated MLP: 73728layer 57: MQA: 48 query / 1 KV heads · head 256layer 57: Gated MLP: 73728layer 58: MQA: 48 query / 1 KV heads · head 256layer 58: Gated MLP: 73728layer 59: MQA: 48 query / 1 KV heads · head 256layer 59: Gated MLP: 73728layer 60: MQA: 48 query / 1 KV heads · head 256layer 60: Gated MLP: 73728layer 61: MQA: 48 query / 1 KV heads · head 256layer 61: Gated MLP: 73728layer 62: MQA: 48 query / 1 KV heads · head 256layer 62: Gated MLP: 73728layer 63: MQA: 48 query / 1 KV heads · head 256layer 63: Gated MLP: 73728layer 64: MQA: 48 query / 1 KV heads · head 256layer 64: Gated MLP: 73728layer 65: MQA: 48 query / 1 KV heads · head 256layer 65: Gated MLP: 73728layer 66: MQA: 48 query / 1 KV heads · head 256layer 66: Gated MLP: 73728layer 67: MQA: 48 query / 1 KV heads · head 256layer 67: Gated MLP: 73728layer 68: MQA: 48 query / 1 KV heads · head 256layer 68: Gated MLP: 73728layer 69: MQA: 48 query / 1 KV heads · head 256layer 69: Gated MLP: 73728layer 70: MQA: 48 query / 1 KV heads · head 256layer 70: Gated MLP: 73728layer 71: MQA: 48 query / 1 KV heads · head 256layer 71: Gated MLP: 73728layer 72: MQA: 48 query / 1 KV heads · head 256layer 72: Gated MLP: 73728layer 73: MQA: 48 query / 1 KV heads · head 256layer 73: Gated MLP: 73728layer 74: MQA: 48 query / 1 KV heads · head 256layer 74: Gated MLP: 73728layer 75: MQA: 48 query / 1 KV heads · head 256layer 75: Gated MLP: 73728layer 76: MQA: 48 query / 1 KV heads · head 256layer 76: Gated MLP: 73728layer 77: MQA: 48 query / 1 KV heads · head 256layer 77: Gated MLP: 73728layer 78: MQA: 48 query / 1 KV heads · head 256layer 78: Gated MLP: 73728layer 79: MQA: 48 query / 1 KV heads · head 256layer 79: Gated MLP: 73728layer 80: MQA: 48 query / 1 KV heads · head 256layer 80: Gated MLP: 73728layer 81: MQA: 48 query / 1 KV heads · head 256layer 81: Gated MLP: 73728layer 82: MQA: 48 query / 1 KV heads · head 256layer 82: Gated MLP: 73728layer 83: MQA: 48 query / 1 KV heads · head 256layer 83: Gated MLP: 73728layer 84: MQA: 48 query / 1 KV heads · head 256layer 84: Gated MLP: 73728layer 85: MQA: 48 query / 1 KV heads · head 256layer 85: Gated MLP: 73728layer 86: MQA: 48 query / 1 KV heads · head 256layer 86: Gated MLP: 73728layer 87: MQA: 48 query / 1 KV heads · head 256layer 87: Gated MLP: 73728layer 88: MQA: 48 query / 1 KV heads · head 256layer 88: Gated MLP: 73728layer 89: MQA: 48 query / 1 KV heads · head 256layer 89: Gated MLP: 73728layer 90: MQA: 48 query / 1 KV heads · head 256layer 90: Gated MLP: 73728layer 91: MQA: 48 query / 1 KV heads · head 256layer 91: Gated MLP: 73728layer 92: MQA: 48 query / 1 KV heads · head 256layer 92: Gated MLP: 73728layer 93: MQA: 48 query / 1 KV heads · head 256layer 93: Gated MLP: 73728layer 94: MQA: 48 query / 1 KV heads · head 256layer 94: Gated MLP: 73728layer 95: MQA: 48 query / 1 KV heads · head 256layer 95: Gated MLP: 73728layer 96: MQA: 48 query / 1 KV heads · head 256layer 96: Gated MLP: 73728layer 97: MQA: 48 query / 1 KV heads · head 256layer 97: Gated MLP: 73728layer 98: MQA: 48 query / 1 KV heads · head 256layer 98: Gated MLP: 73728layer 99: MQA: 48 query / 1 KV heads · head 256layer 99: Gated MLP: 73728layer 100: MQA: 48 query / 1 KV heads · head 256layer 100: Gated MLP: 73728layer 101: MQA: 48 query / 1 KV heads · head 256layer 101: Gated MLP: 73728layer 102: MQA: 48 query / 1 KV heads · head 256layer 102: Gated MLP: 73728layer 103: MQA: 48 query / 1 KV heads · head 256layer 103: Gated MLP: 73728layer 104: MQA: 48 query / 1 KV heads · head 256layer 104: Gated MLP: 73728layer 105: MQA: 48 query / 1 KV heads · head 256layer 105: Gated MLP: 73728layer 106: MQA: 48 query / 1 KV heads · head 256layer 106: Gated MLP: 73728layer 107: MQA: 48 query / 1 KV heads · head 256layer 107: Gated MLP: 73728layer 108: MQA: 48 query / 1 KV heads · head 256layer 108: Gated MLP: 73728layer 109: MQA: 48 query / 1 KV heads · head 256layer 109: Gated MLP: 73728layer 110: MQA: 48 query / 1 KV heads · head 256layer 110: Gated MLP: 73728layer 111: MQA: 48 query / 1 KV heads · head 256layer 111: Gated MLP: 73728layer 112: MQA: 48 query / 1 KV heads · head 256layer 112: Gated MLP: 73728layer 113: MQA: 48 query / 1 KV heads · head 256layer 113: Gated MLP: 73728layer 114: MQA: 48 query / 1 KV heads · head 256layer 114: Gated MLP: 73728layer 115: MQA: 48 query / 1 KV heads · head 256layer 115: Gated MLP: 73728layer 116: MQA: 48 query / 1 KV heads · head 256layer 116: Gated MLP: 73728layer 117: MQA: 48 query / 1 KV heads · head 256layer 117: Gated MLP: 73728059117× 118normMQA: 48 query / 1 KV heads · head 256Gated MLP: 73728+full attentiondense FFN

Modelled costs

From the cost model, batch size 1. Totals the lab states are in the table above; differences come from rounding, from what a lab counts, or from parts the model does not describe (listed on the about page).

Parameters (modelled)540B
Active per token (modelled)540B
Without embeddings and output head536B total, 536B active
KV cache per token, BF16 (layers that grow with context)118 KiB
KV cache + state at 2K tokens, BF16236 MiB
Decode FLOPs per token at 4K context1.1 TFLOP
Prefill FLOPs for a 4K prompt4.44 PFLOP

KV cache against context

PaLM 540B: KV cache bytes against context length101001,000980 KiB9.5 MiB95 MiBcontext (tokens)KV cache + state (BF16)PaLM 540B

Compare with other models →

Every architecture field

FieldValueSource
d_model18,432paperpaperTable 1: d_model 18432
vocab256,000paperpaper§2: SentencePiece vocabulary with 256k tokens (256,000 assumed)
tied_embeddingstruepaperpaper§2: shared input-output embeddings
mixers.full.typeattncodemodelling codeattention block
mixers.full.heads48paperpaperTable 1: 48 heads
mixers.full.kv_heads1paperpaper§2: multi-query attention (one key/value head)
mixers.full.head_dim256paperpaperTable 1 caption: head size always 256
ffns.dense.typedensecodemodelling codeMLP block
ffns.dense.d_ff73,728paperpaperTable 1 caption: d_ff always 4 x d_model
ffns.dense.gatedtruecodemodelling codeMLP: gated (SwiGLU/GeGLU)
layout118× full/densepaperpaperTable 1: layers 118
norms_per_layer1codemodelling codeparallel block: one norm feeds attention and MLP

Sources