llm-architectures-explained

/models

Llama 2 70B

Meta · Llama · open weights

Facts and where they come from

Released2023-07paperarXiv 2307.09288arXiv v1, July 2023
Licencellama2config.jsonconfig.jsonREADME metadata: license
Total parameters70Blabmodel cardmodel name Llama-2-70b
Active parametersnot disclosednot disclosed
Context length4K tokenslabmeta-llama/llama-models sku_list.py (pinned)Llama 2 paper (arXiv 2307.09288) Table 1: context length 4k
Norm placementprecodemodelling codetransformers 5.18.0 / repo modelling code for llama: input_layernorm before attention, post_attention_layernorm before the MLP
Norm typeRMSNormcodemodelling codetransformers 5.18.0 / repo modelling code for llama: input_layernorm before attention, post_attention_layernorm before the MLP
QK-normnocodemodelling codeno q/k normalisation in the attention block
Positional encodingRoPEcodemodelling coderotary on the full head (default)
Parallel attention and MLPnocodemodelling codetransformers 5.18.0 / repo modelling code for llama: input_layernorm before attention, post_attention_layernorm before the MLP

Architecture, drawn from the data

GQA 64q/8kv. Each column is one layer: its token mixer above, its feed-forward block below. Paler columns reuse another layer’s keys and values.

Llama 2 70B: layer stack and blockslayers (80)mixer / FFNlayer 0: GQA: 64 query / 8 KV heads · head 128layer 0: Gated MLP: 28672layer 1: GQA: 64 query / 8 KV heads · head 128layer 1: Gated MLP: 28672layer 2: GQA: 64 query / 8 KV heads · head 128layer 2: Gated MLP: 28672layer 3: GQA: 64 query / 8 KV heads · head 128layer 3: Gated MLP: 28672layer 4: GQA: 64 query / 8 KV heads · head 128layer 4: Gated MLP: 28672layer 5: GQA: 64 query / 8 KV heads · head 128layer 5: Gated MLP: 28672layer 6: GQA: 64 query / 8 KV heads · head 128layer 6: Gated MLP: 28672layer 7: GQA: 64 query / 8 KV heads · head 128layer 7: Gated MLP: 28672layer 8: GQA: 64 query / 8 KV heads · head 128layer 8: Gated MLP: 28672layer 9: GQA: 64 query / 8 KV heads · head 128layer 9: Gated MLP: 28672layer 10: GQA: 64 query / 8 KV heads · head 128layer 10: Gated MLP: 28672layer 11: GQA: 64 query / 8 KV heads · head 128layer 11: Gated MLP: 28672layer 12: GQA: 64 query / 8 KV heads · head 128layer 12: Gated MLP: 28672layer 13: GQA: 64 query / 8 KV heads · head 128layer 13: Gated MLP: 28672layer 14: GQA: 64 query / 8 KV heads · head 128layer 14: Gated MLP: 28672layer 15: GQA: 64 query / 8 KV heads · head 128layer 15: Gated MLP: 28672layer 16: GQA: 64 query / 8 KV heads · head 128layer 16: Gated MLP: 28672layer 17: GQA: 64 query / 8 KV heads · head 128layer 17: Gated MLP: 28672layer 18: GQA: 64 query / 8 KV heads · head 128layer 18: Gated MLP: 28672layer 19: GQA: 64 query / 8 KV heads · head 128layer 19: Gated MLP: 28672layer 20: GQA: 64 query / 8 KV heads · head 128layer 20: Gated MLP: 28672layer 21: GQA: 64 query / 8 KV heads · head 128layer 21: Gated MLP: 28672layer 22: GQA: 64 query / 8 KV heads · head 128layer 22: Gated MLP: 28672layer 23: GQA: 64 query / 8 KV heads · head 128layer 23: Gated MLP: 28672layer 24: GQA: 64 query / 8 KV heads · head 128layer 24: Gated MLP: 28672layer 25: GQA: 64 query / 8 KV heads · head 128layer 25: Gated MLP: 28672layer 26: GQA: 64 query / 8 KV heads · head 128layer 26: Gated MLP: 28672layer 27: GQA: 64 query / 8 KV heads · head 128layer 27: Gated MLP: 28672layer 28: GQA: 64 query / 8 KV heads · head 128layer 28: Gated MLP: 28672layer 29: GQA: 64 query / 8 KV heads · head 128layer 29: Gated MLP: 28672layer 30: GQA: 64 query / 8 KV heads · head 128layer 30: Gated MLP: 28672layer 31: GQA: 64 query / 8 KV heads · head 128layer 31: Gated MLP: 28672layer 32: GQA: 64 query / 8 KV heads · head 128layer 32: Gated MLP: 28672layer 33: GQA: 64 query / 8 KV heads · head 128layer 33: Gated MLP: 28672layer 34: GQA: 64 query / 8 KV heads · head 128layer 34: Gated MLP: 28672layer 35: GQA: 64 query / 8 KV heads · head 128layer 35: Gated MLP: 28672layer 36: GQA: 64 query / 8 KV heads · head 128layer 36: Gated MLP: 28672layer 37: GQA: 64 query / 8 KV heads · head 128layer 37: Gated MLP: 28672layer 38: GQA: 64 query / 8 KV heads · head 128layer 38: Gated MLP: 28672layer 39: GQA: 64 query / 8 KV heads · head 128layer 39: Gated MLP: 28672layer 40: GQA: 64 query / 8 KV heads · head 128layer 40: Gated MLP: 28672layer 41: GQA: 64 query / 8 KV heads · head 128layer 41: Gated MLP: 28672layer 42: GQA: 64 query / 8 KV heads · head 128layer 42: Gated MLP: 28672layer 43: GQA: 64 query / 8 KV heads · head 128layer 43: Gated MLP: 28672layer 44: GQA: 64 query / 8 KV heads · head 128layer 44: Gated MLP: 28672layer 45: GQA: 64 query / 8 KV heads · head 128layer 45: Gated MLP: 28672layer 46: GQA: 64 query / 8 KV heads · head 128layer 46: Gated MLP: 28672layer 47: GQA: 64 query / 8 KV heads · head 128layer 47: Gated MLP: 28672layer 48: GQA: 64 query / 8 KV heads · head 128layer 48: Gated MLP: 28672layer 49: GQA: 64 query / 8 KV heads · head 128layer 49: Gated MLP: 28672layer 50: GQA: 64 query / 8 KV heads · head 128layer 50: Gated MLP: 28672layer 51: GQA: 64 query / 8 KV heads · head 128layer 51: Gated MLP: 28672layer 52: GQA: 64 query / 8 KV heads · head 128layer 52: Gated MLP: 28672layer 53: GQA: 64 query / 8 KV heads · head 128layer 53: Gated MLP: 28672layer 54: GQA: 64 query / 8 KV heads · head 128layer 54: Gated MLP: 28672layer 55: GQA: 64 query / 8 KV heads · head 128layer 55: Gated MLP: 28672layer 56: GQA: 64 query / 8 KV heads · head 128layer 56: Gated MLP: 28672layer 57: GQA: 64 query / 8 KV heads · head 128layer 57: Gated MLP: 28672layer 58: GQA: 64 query / 8 KV heads · head 128layer 58: Gated MLP: 28672layer 59: GQA: 64 query / 8 KV heads · head 128layer 59: Gated MLP: 28672layer 60: GQA: 64 query / 8 KV heads · head 128layer 60: Gated MLP: 28672layer 61: GQA: 64 query / 8 KV heads · head 128layer 61: Gated MLP: 28672layer 62: GQA: 64 query / 8 KV heads · head 128layer 62: Gated MLP: 28672layer 63: GQA: 64 query / 8 KV heads · head 128layer 63: Gated MLP: 28672layer 64: GQA: 64 query / 8 KV heads · head 128layer 64: Gated MLP: 28672layer 65: GQA: 64 query / 8 KV heads · head 128layer 65: Gated MLP: 28672layer 66: GQA: 64 query / 8 KV heads · head 128layer 66: Gated MLP: 28672layer 67: GQA: 64 query / 8 KV heads · head 128layer 67: Gated MLP: 28672layer 68: GQA: 64 query / 8 KV heads · head 128layer 68: Gated MLP: 28672layer 69: GQA: 64 query / 8 KV heads · head 128layer 69: Gated MLP: 28672layer 70: GQA: 64 query / 8 KV heads · head 128layer 70: Gated MLP: 28672layer 71: GQA: 64 query / 8 KV heads · head 128layer 71: Gated MLP: 28672layer 72: GQA: 64 query / 8 KV heads · head 128layer 72: Gated MLP: 28672layer 73: GQA: 64 query / 8 KV heads · head 128layer 73: Gated MLP: 28672layer 74: GQA: 64 query / 8 KV heads · head 128layer 74: Gated MLP: 28672layer 75: GQA: 64 query / 8 KV heads · head 128layer 75: Gated MLP: 28672layer 76: GQA: 64 query / 8 KV heads · head 128layer 76: Gated MLP: 28672layer 77: GQA: 64 query / 8 KV heads · head 128layer 77: Gated MLP: 28672layer 78: GQA: 64 query / 8 KV heads · head 128layer 78: Gated MLP: 28672layer 79: GQA: 64 query / 8 KV heads · head 128layer 79: Gated MLP: 2867204079× 80normGQA: 64 query / 8 KV heads · head 128+normGated MLP: 28672+full attentiondense FFN

Modelled costs

From the cost model, batch size 1. Totals the lab states are in the table above; differences come from rounding, from what a lab counts, or from parts the model does not describe (listed on the about page).

Parameters (modelled)69B
Active per token (modelled)69B
Without embeddings and output head68.5B total, 68.5B active
Published weights (Hugging Face count)69B
KV cache per token, BF16 (layers that grow with context)320 KiB
KV cache + state at 4K tokens, BF161.25 GiB
Decode FLOPs per token at 4K context148 GFLOP
Prefill FLOPs for a 4K prompt583 TFLOP

KV cache against context

Llama 2 70B: KV cache bytes against context length101001,000980 KiB9.5 MiB95 MiB950 MiBcontext (tokens)KV cache + state (BF16)Llama 2 70B

Compare with other models →

Every architecture field

FieldValueSource
d_model8,192labmeta-llama/llama-models sku_list.py (pinned)sku_list.py llama2_70b: dim
vocab32,000labmeta-llama/llama-models sku_list.py (pinned)sku_list.py: LLAMA2_VOCAB_SIZE
tied_embeddingsfalselabmeta-llama/llama-models sku_list.py (pinned)llama3/model.py: separate output projection
mixers.full.typeattncodemodelling codeattention block
mixers.full.heads64labmeta-llama/llama-models sku_list.py (pinned)sku_list.py llama2_70b: n_heads
mixers.full.kv_heads8labmeta-llama/llama-models sku_list.py (pinned)sku_list.py llama2_70b: n_kv_heads
mixers.full.head_dim128codemodelling codetransformers 5.18.0: head_dim = hidden_size / num_attention_heads
ffns.dense.typedensecodemodelling codeMLP block
ffns.dense.d_ff28,672labmeta-llama/llama-models sku_list.py (pinned)llama3/model.py FeedForward: int(2*4*dim/3), x ffn_dim_multiplier 1.3, rounded up to multiple_of 4096
ffns.dense.gatedtruecodemodelling codeMLP: gated (SwiGLU/GeGLU)
layout80× full/denselabmeta-llama/llama-models sku_list.py (pinned)sku_list.py llama2_70b: n_layers

Sources