llm-architectures-explained

/models

Llama 3.1 405B

Meta · Llama 3 · open weights

Facts and where they come from

Released2024-07paperarXiv 2407.21783arXiv v1, July 2024
Licencenot disclosednot disclosed
Total parameters405BpaperarXiv 2407.21783abstract: a dense Transformer with 405B parameters
Active parametersnot disclosednot disclosed
Context length128K tokenslabmeta-llama/llama-models sku_list.py (pinned)Llama 3 paper (arXiv 2407.21783): 128K context
Norm placementprecodemodelling codetransformers 5.18.0 / repo modelling code for llama: input_layernorm before attention, post_attention_layernorm before the MLP
Norm typeRMSNormcodemodelling codetransformers 5.18.0 / repo modelling code for llama: input_layernorm before attention, post_attention_layernorm before the MLP
QK-normnocodemodelling codeno q/k normalisation in the attention block
Positional encodingRoPEcodemodelling coderotary on the full head (default)
Parallel attention and MLPnocodemodelling codetransformers 5.18.0 / repo modelling code for llama: input_layernorm before attention, post_attention_layernorm before the MLP

Architecture, drawn from the data

GQA 128q/8kv. Each column is one layer: its token mixer above, its feed-forward block below. Paler columns reuse another layer’s keys and values.

Llama 3.1 405B: layer stack and blockslayers (126)mixer / FFNlayer 0: GQA: 128 query / 8 KV heads · head 128layer 0: Gated MLP: 53248layer 1: GQA: 128 query / 8 KV heads · head 128layer 1: Gated MLP: 53248layer 2: GQA: 128 query / 8 KV heads · head 128layer 2: Gated MLP: 53248layer 3: GQA: 128 query / 8 KV heads · head 128layer 3: Gated MLP: 53248layer 4: GQA: 128 query / 8 KV heads · head 128layer 4: Gated MLP: 53248layer 5: GQA: 128 query / 8 KV heads · head 128layer 5: Gated MLP: 53248layer 6: GQA: 128 query / 8 KV heads · head 128layer 6: Gated MLP: 53248layer 7: GQA: 128 query / 8 KV heads · head 128layer 7: Gated MLP: 53248layer 8: GQA: 128 query / 8 KV heads · head 128layer 8: Gated MLP: 53248layer 9: GQA: 128 query / 8 KV heads · head 128layer 9: Gated MLP: 53248layer 10: GQA: 128 query / 8 KV heads · head 128layer 10: Gated MLP: 53248layer 11: GQA: 128 query / 8 KV heads · head 128layer 11: Gated MLP: 53248layer 12: GQA: 128 query / 8 KV heads · head 128layer 12: Gated MLP: 53248layer 13: GQA: 128 query / 8 KV heads · head 128layer 13: Gated MLP: 53248layer 14: GQA: 128 query / 8 KV heads · head 128layer 14: Gated MLP: 53248layer 15: GQA: 128 query / 8 KV heads · head 128layer 15: Gated MLP: 53248layer 16: GQA: 128 query / 8 KV heads · head 128layer 16: Gated MLP: 53248layer 17: GQA: 128 query / 8 KV heads · head 128layer 17: Gated MLP: 53248layer 18: GQA: 128 query / 8 KV heads · head 128layer 18: Gated MLP: 53248layer 19: GQA: 128 query / 8 KV heads · head 128layer 19: Gated MLP: 53248layer 20: GQA: 128 query / 8 KV heads · head 128layer 20: Gated MLP: 53248layer 21: GQA: 128 query / 8 KV heads · head 128layer 21: Gated MLP: 53248layer 22: GQA: 128 query / 8 KV heads · head 128layer 22: Gated MLP: 53248layer 23: GQA: 128 query / 8 KV heads · head 128layer 23: Gated MLP: 53248layer 24: GQA: 128 query / 8 KV heads · head 128layer 24: Gated MLP: 53248layer 25: GQA: 128 query / 8 KV heads · head 128layer 25: Gated MLP: 53248layer 26: GQA: 128 query / 8 KV heads · head 128layer 26: Gated MLP: 53248layer 27: GQA: 128 query / 8 KV heads · head 128layer 27: Gated MLP: 53248layer 28: GQA: 128 query / 8 KV heads · head 128layer 28: Gated MLP: 53248layer 29: GQA: 128 query / 8 KV heads · head 128layer 29: Gated MLP: 53248layer 30: GQA: 128 query / 8 KV heads · head 128layer 30: Gated MLP: 53248layer 31: GQA: 128 query / 8 KV heads · head 128layer 31: Gated MLP: 53248layer 32: GQA: 128 query / 8 KV heads · head 128layer 32: Gated MLP: 53248layer 33: GQA: 128 query / 8 KV heads · head 128layer 33: Gated MLP: 53248layer 34: GQA: 128 query / 8 KV heads · head 128layer 34: Gated MLP: 53248layer 35: GQA: 128 query / 8 KV heads · head 128layer 35: Gated MLP: 53248layer 36: GQA: 128 query / 8 KV heads · head 128layer 36: Gated MLP: 53248layer 37: GQA: 128 query / 8 KV heads · head 128layer 37: Gated MLP: 53248layer 38: GQA: 128 query / 8 KV heads · head 128layer 38: Gated MLP: 53248layer 39: GQA: 128 query / 8 KV heads · head 128layer 39: Gated MLP: 53248layer 40: GQA: 128 query / 8 KV heads · head 128layer 40: Gated MLP: 53248layer 41: GQA: 128 query / 8 KV heads · head 128layer 41: Gated MLP: 53248layer 42: GQA: 128 query / 8 KV heads · head 128layer 42: Gated MLP: 53248layer 43: GQA: 128 query / 8 KV heads · head 128layer 43: Gated MLP: 53248layer 44: GQA: 128 query / 8 KV heads · head 128layer 44: Gated MLP: 53248layer 45: GQA: 128 query / 8 KV heads · head 128layer 45: Gated MLP: 53248layer 46: GQA: 128 query / 8 KV heads · head 128layer 46: Gated MLP: 53248layer 47: GQA: 128 query / 8 KV heads · head 128layer 47: Gated MLP: 53248layer 48: GQA: 128 query / 8 KV heads · head 128layer 48: Gated MLP: 53248layer 49: GQA: 128 query / 8 KV heads · head 128layer 49: Gated MLP: 53248layer 50: GQA: 128 query / 8 KV heads · head 128layer 50: Gated MLP: 53248layer 51: GQA: 128 query / 8 KV heads · head 128layer 51: Gated MLP: 53248layer 52: GQA: 128 query / 8 KV heads · head 128layer 52: Gated MLP: 53248layer 53: GQA: 128 query / 8 KV heads · head 128layer 53: Gated MLP: 53248layer 54: GQA: 128 query / 8 KV heads · head 128layer 54: Gated MLP: 53248layer 55: GQA: 128 query / 8 KV heads · head 128layer 55: Gated MLP: 53248layer 56: GQA: 128 query / 8 KV heads · head 128layer 56: Gated MLP: 53248layer 57: GQA: 128 query / 8 KV heads · head 128layer 57: Gated MLP: 53248layer 58: GQA: 128 query / 8 KV heads · head 128layer 58: Gated MLP: 53248layer 59: GQA: 128 query / 8 KV heads · head 128layer 59: Gated MLP: 53248layer 60: GQA: 128 query / 8 KV heads · head 128layer 60: Gated MLP: 53248layer 61: GQA: 128 query / 8 KV heads · head 128layer 61: Gated MLP: 53248layer 62: GQA: 128 query / 8 KV heads · head 128layer 62: Gated MLP: 53248layer 63: GQA: 128 query / 8 KV heads · head 128layer 63: Gated MLP: 53248layer 64: GQA: 128 query / 8 KV heads · head 128layer 64: Gated MLP: 53248layer 65: GQA: 128 query / 8 KV heads · head 128layer 65: Gated MLP: 53248layer 66: GQA: 128 query / 8 KV heads · head 128layer 66: Gated MLP: 53248layer 67: GQA: 128 query / 8 KV heads · head 128layer 67: Gated MLP: 53248layer 68: GQA: 128 query / 8 KV heads · head 128layer 68: Gated MLP: 53248layer 69: GQA: 128 query / 8 KV heads · head 128layer 69: Gated MLP: 53248layer 70: GQA: 128 query / 8 KV heads · head 128layer 70: Gated MLP: 53248layer 71: GQA: 128 query / 8 KV heads · head 128layer 71: Gated MLP: 53248layer 72: GQA: 128 query / 8 KV heads · head 128layer 72: Gated MLP: 53248layer 73: GQA: 128 query / 8 KV heads · head 128layer 73: Gated MLP: 53248layer 74: GQA: 128 query / 8 KV heads · head 128layer 74: Gated MLP: 53248layer 75: GQA: 128 query / 8 KV heads · head 128layer 75: Gated MLP: 53248layer 76: GQA: 128 query / 8 KV heads · head 128layer 76: Gated MLP: 53248layer 77: GQA: 128 query / 8 KV heads · head 128layer 77: Gated MLP: 53248layer 78: GQA: 128 query / 8 KV heads · head 128layer 78: Gated MLP: 53248layer 79: GQA: 128 query / 8 KV heads · head 128layer 79: Gated MLP: 53248layer 80: GQA: 128 query / 8 KV heads · head 128layer 80: Gated MLP: 53248layer 81: GQA: 128 query / 8 KV heads · head 128layer 81: Gated MLP: 53248layer 82: GQA: 128 query / 8 KV heads · head 128layer 82: Gated MLP: 53248layer 83: GQA: 128 query / 8 KV heads · head 128layer 83: Gated MLP: 53248layer 84: GQA: 128 query / 8 KV heads · head 128layer 84: Gated MLP: 53248layer 85: GQA: 128 query / 8 KV heads · head 128layer 85: Gated MLP: 53248layer 86: GQA: 128 query / 8 KV heads · head 128layer 86: Gated MLP: 53248layer 87: GQA: 128 query / 8 KV heads · head 128layer 87: Gated MLP: 53248layer 88: GQA: 128 query / 8 KV heads · head 128layer 88: Gated MLP: 53248layer 89: GQA: 128 query / 8 KV heads · head 128layer 89: Gated MLP: 53248layer 90: GQA: 128 query / 8 KV heads · head 128layer 90: Gated MLP: 53248layer 91: GQA: 128 query / 8 KV heads · head 128layer 91: Gated MLP: 53248layer 92: GQA: 128 query / 8 KV heads · head 128layer 92: Gated MLP: 53248layer 93: GQA: 128 query / 8 KV heads · head 128layer 93: Gated MLP: 53248layer 94: GQA: 128 query / 8 KV heads · head 128layer 94: Gated MLP: 53248layer 95: GQA: 128 query / 8 KV heads · head 128layer 95: Gated MLP: 53248layer 96: GQA: 128 query / 8 KV heads · head 128layer 96: Gated MLP: 53248layer 97: GQA: 128 query / 8 KV heads · head 128layer 97: Gated MLP: 53248layer 98: GQA: 128 query / 8 KV heads · head 128layer 98: Gated MLP: 53248layer 99: GQA: 128 query / 8 KV heads · head 128layer 99: Gated MLP: 53248layer 100: GQA: 128 query / 8 KV heads · head 128layer 100: Gated MLP: 53248layer 101: GQA: 128 query / 8 KV heads · head 128layer 101: Gated MLP: 53248layer 102: GQA: 128 query / 8 KV heads · head 128layer 102: Gated MLP: 53248layer 103: GQA: 128 query / 8 KV heads · head 128layer 103: Gated MLP: 53248layer 104: GQA: 128 query / 8 KV heads · head 128layer 104: Gated MLP: 53248layer 105: GQA: 128 query / 8 KV heads · head 128layer 105: Gated MLP: 53248layer 106: GQA: 128 query / 8 KV heads · head 128layer 106: Gated MLP: 53248layer 107: GQA: 128 query / 8 KV heads · head 128layer 107: Gated MLP: 53248layer 108: GQA: 128 query / 8 KV heads · head 128layer 108: Gated MLP: 53248layer 109: GQA: 128 query / 8 KV heads · head 128layer 109: Gated MLP: 53248layer 110: GQA: 128 query / 8 KV heads · head 128layer 110: Gated MLP: 53248layer 111: GQA: 128 query / 8 KV heads · head 128layer 111: Gated MLP: 53248layer 112: GQA: 128 query / 8 KV heads · head 128layer 112: Gated MLP: 53248layer 113: GQA: 128 query / 8 KV heads · head 128layer 113: Gated MLP: 53248layer 114: GQA: 128 query / 8 KV heads · head 128layer 114: Gated MLP: 53248layer 115: GQA: 128 query / 8 KV heads · head 128layer 115: Gated MLP: 53248layer 116: GQA: 128 query / 8 KV heads · head 128layer 116: Gated MLP: 53248layer 117: GQA: 128 query / 8 KV heads · head 128layer 117: Gated MLP: 53248layer 118: GQA: 128 query / 8 KV heads · head 128layer 118: Gated MLP: 53248layer 119: GQA: 128 query / 8 KV heads · head 128layer 119: Gated MLP: 53248layer 120: GQA: 128 query / 8 KV heads · head 128layer 120: Gated MLP: 53248layer 121: GQA: 128 query / 8 KV heads · head 128layer 121: Gated MLP: 53248layer 122: GQA: 128 query / 8 KV heads · head 128layer 122: Gated MLP: 53248layer 123: GQA: 128 query / 8 KV heads · head 128layer 123: Gated MLP: 53248layer 124: GQA: 128 query / 8 KV heads · head 128layer 124: Gated MLP: 53248layer 125: GQA: 128 query / 8 KV heads · head 128layer 125: Gated MLP: 53248063125× 126normGQA: 128 query / 8 KV heads · head 128+normGated MLP: 53248+full attentiondense FFN

Modelled costs

From the cost model, batch size 1. Totals the lab states are in the table above; differences come from rounding, from what a lab counts, or from parts the model does not describe (listed on the about page).

Parameters (modelled)406B
Active per token (modelled)406B
Without embeddings and output head402B total, 402B active
KV cache per token, BF16 (layers that grow with context)504 KiB
KV cache + state at 128K tokens, BF1663 GiB
Decode FLOPs per token at 4K context841 GFLOP
Prefill FLOPs for a 4K prompt3.36 PFLOP

KV cache against context

Llama 3.1 405B: KV cache bytes against context length101001,00010,000100,000980 KiB9.5 MiB95 MiB950 MiB9.3 GiBcontext (tokens)KV cache + state (BF16)Llama 3.1 405B

Compare with other models →

Every architecture field

FieldValueSource
d_model16,384labmeta-llama/llama-models sku_list.py (pinned)sku_list.py llama3_1_405b: dim
vocab128,256labmeta-llama/llama-models sku_list.py (pinned)sku_list.py: LLAMA3_VOCAB_SIZE
tied_embeddingsfalselabmeta-llama/llama-models sku_list.py (pinned)llama3/model.py: separate output projection
mixers.full.typeattncodemodelling codeattention block
mixers.full.heads128labmeta-llama/llama-models sku_list.py (pinned)sku_list.py llama3_1_405b: n_heads
mixers.full.kv_heads8labmeta-llama/llama-models sku_list.py (pinned)sku_list.py llama3_1_405b: n_kv_heads
mixers.full.head_dim128codemodelling codetransformers 5.18.0: head_dim = hidden_size / num_attention_heads
ffns.dense.typedensecodemodelling codeMLP block
ffns.dense.d_ff53,248labmeta-llama/llama-models sku_list.py (pinned)llama3/model.py FeedForward: int(2*4*dim/3), x ffn_dim_multiplier 1.2, rounded up to multiple_of 4096
ffns.dense.gatedtruecodemodelling codeMLP: gated (SwiGLU/GeGLU)
layout126× full/denselabmeta-llama/llama-models sku_list.py (pinned)sku_list.py llama3_1_405b: n_layers

Sources