llm-architectures-explained

/models

GPT-NeoX-20B

EleutherAI · GPT-NeoX · open weights

Facts and where they come from

Released2022-04paperarXiv 2204.06745arXiv v1, April 2022
Licenceapache-2.0config.jsonconfig.jsonREADME metadata: license
Total parameters20Blabmodel cardREADME: a 20 billion parameter autoregressive language model
Active parametersnot disclosednot disclosed
Context length2K tokensconfig.jsonconfig.jsonmax_position_embeddings
Norm placementparallelcodemodelling codetransformers 5.18.0 gpt_neox: use_parallel_residual, separate input and post-attention norms
Norm typeLayerNormcodemodelling codetransformers 5.18.0 gpt_neox: use_parallel_residual, separate input and post-attention norms
QK-normnocodemodelling codeno q/k normalisation in the attention block
Positional encodingRoPE on 25% of each headconfig.jsonconfig.jsonrotary_pct
Parallel attention and MLPyescodemodelling codetransformers 5.18.0 gpt_neox: use_parallel_residual, separate input and post-attention norms

Architecture, drawn from the data

MHA 64q/64kv. Each column is one layer: its token mixer above, its feed-forward block below. Paler columns reuse another layer’s keys and values.

GPT-NeoX-20B: layer stack and blockslayers (44)mixer / FFNlayer 0: MHA: 64 query / 64 KV heads · head 96layer 0: MLP: 24576layer 1: MHA: 64 query / 64 KV heads · head 96layer 1: MLP: 24576layer 2: MHA: 64 query / 64 KV heads · head 96layer 2: MLP: 24576layer 3: MHA: 64 query / 64 KV heads · head 96layer 3: MLP: 24576layer 4: MHA: 64 query / 64 KV heads · head 96layer 4: MLP: 24576layer 5: MHA: 64 query / 64 KV heads · head 96layer 5: MLP: 24576layer 6: MHA: 64 query / 64 KV heads · head 96layer 6: MLP: 24576layer 7: MHA: 64 query / 64 KV heads · head 96layer 7: MLP: 24576layer 8: MHA: 64 query / 64 KV heads · head 96layer 8: MLP: 24576layer 9: MHA: 64 query / 64 KV heads · head 96layer 9: MLP: 24576layer 10: MHA: 64 query / 64 KV heads · head 96layer 10: MLP: 24576layer 11: MHA: 64 query / 64 KV heads · head 96layer 11: MLP: 24576layer 12: MHA: 64 query / 64 KV heads · head 96layer 12: MLP: 24576layer 13: MHA: 64 query / 64 KV heads · head 96layer 13: MLP: 24576layer 14: MHA: 64 query / 64 KV heads · head 96layer 14: MLP: 24576layer 15: MHA: 64 query / 64 KV heads · head 96layer 15: MLP: 24576layer 16: MHA: 64 query / 64 KV heads · head 96layer 16: MLP: 24576layer 17: MHA: 64 query / 64 KV heads · head 96layer 17: MLP: 24576layer 18: MHA: 64 query / 64 KV heads · head 96layer 18: MLP: 24576layer 19: MHA: 64 query / 64 KV heads · head 96layer 19: MLP: 24576layer 20: MHA: 64 query / 64 KV heads · head 96layer 20: MLP: 24576layer 21: MHA: 64 query / 64 KV heads · head 96layer 21: MLP: 24576layer 22: MHA: 64 query / 64 KV heads · head 96layer 22: MLP: 24576layer 23: MHA: 64 query / 64 KV heads · head 96layer 23: MLP: 24576layer 24: MHA: 64 query / 64 KV heads · head 96layer 24: MLP: 24576layer 25: MHA: 64 query / 64 KV heads · head 96layer 25: MLP: 24576layer 26: MHA: 64 query / 64 KV heads · head 96layer 26: MLP: 24576layer 27: MHA: 64 query / 64 KV heads · head 96layer 27: MLP: 24576layer 28: MHA: 64 query / 64 KV heads · head 96layer 28: MLP: 24576layer 29: MHA: 64 query / 64 KV heads · head 96layer 29: MLP: 24576layer 30: MHA: 64 query / 64 KV heads · head 96layer 30: MLP: 24576layer 31: MHA: 64 query / 64 KV heads · head 96layer 31: MLP: 24576layer 32: MHA: 64 query / 64 KV heads · head 96layer 32: MLP: 24576layer 33: MHA: 64 query / 64 KV heads · head 96layer 33: MLP: 24576layer 34: MHA: 64 query / 64 KV heads · head 96layer 34: MLP: 24576layer 35: MHA: 64 query / 64 KV heads · head 96layer 35: MLP: 24576layer 36: MHA: 64 query / 64 KV heads · head 96layer 36: MLP: 24576layer 37: MHA: 64 query / 64 KV heads · head 96layer 37: MLP: 24576layer 38: MHA: 64 query / 64 KV heads · head 96layer 38: MLP: 24576layer 39: MHA: 64 query / 64 KV heads · head 96layer 39: MLP: 24576layer 40: MHA: 64 query / 64 KV heads · head 96layer 40: MLP: 24576layer 41: MHA: 64 query / 64 KV heads · head 96layer 41: MLP: 24576layer 42: MHA: 64 query / 64 KV heads · head 96layer 42: MLP: 24576layer 43: MHA: 64 query / 64 KV heads · head 96layer 43: MLP: 2457602243× 44normMHA: 64 query / 64 KV heads · head 96MLP: 24576+full attentiondense FFN

Modelled costs

From the cost model, batch size 1. Totals the lab states are in the table above; differences come from rounding, from what a lab counts, or from parts the model does not describe (listed on the about page).

Parameters (modelled)20.6B
Active per token (modelled)20.6B
Without embeddings and output head19.9B total, 19.9B active
Published weights (Hugging Face count)20.7B (packed low-bit tensors, so not comparable)
KV cache per token, BF16 (layers that grow with context)1.03 MiB
KV cache + state at 2K tokens, BF162.06 GiB
Decode FLOPs per token at 4K context44.9 GFLOP
Prefill FLOPs for a 4K prompt172 TFLOP

KV cache against context

GPT-NeoX-20B: KV cache bytes against context length101001,0009.5 MiB95 MiB950 MiBcontext (tokens)KV cache + state (BF16)GPT-NeoX-20B

Compare with other models →

Every architecture field

FieldValueSource
d_model6,144config.jsonconfig.jsonhidden_size
vocab50,432config.jsonconfig.jsonvocab_size
tied_embeddingsfalseconfig.jsonconfig.jsontie_word_embeddings
mixers.full.typeattncodemodelling codeattention block
mixers.full.heads64config.jsonconfig.jsonnum_attention_heads
mixers.full.kv_heads64config.jsonconfig.jsonnum_attention_heads
mixers.full.head_dim96codemodelling codetransformers 5.18.0: head_dim = hidden_size / num_attention_heads
mixers.full.biastruecodemodelling codeqkv biases
ffns.dense.typedensecodemodelling codeMLP block
ffns.dense.d_ff24,576config.jsonconfig.jsonintermediate_size
ffns.dense.gatedfalsecodemodelling codeMLP: two matrices, no gate
ffns.dense.biastruecodemodelling codeMLP has biases
layout44× full/denseconfig.jsonconfig.jsonnum_hidden_layers

Sources