GPT-2 XL
OpenAI · GPT-2 · open weights
- MHA
- Learned absolute
- Pre-norm
Facts and where they come from
| Released | 2019-02 | labannouncementGPT-2 paper, February 2019 |
|---|---|---|
| Licence | mit | config.jsonconfig.jsonREADME metadata: license |
| Total parameters | 1.5B | labannouncementGPT-2 paper (cdn.openai.com), Table 2: 1542M parameters for the largest model |
| Active parameters | not disclosed | not disclosed |
| Context length | 1K tokens | config.jsonconfig.jsonn_positions |
| Norm placement | pre | codemodelling codetransformers 5.18.0 gpt2: ln_1 before attention, ln_2 before the MLP |
| Norm type | LayerNorm | codemodelling codetransformers 5.18.0 gpt2: ln_1 before attention, ln_2 before the MLP |
| QK-norm | no | codemodelling codeno q/k normalisation in the attention block |
| Positional encoding | learned absolute | codemodelling codetransformers 5.18.0 gpt2: learned absolute position embeddings |
| Parallel attention and MLP | no | codemodelling codetransformers 5.18.0 gpt2: ln_1 before attention, ln_2 before the MLP |
Architecture, drawn from the data
MHA 25q/25kv. Each column is one layer: its token mixer above, its feed-forward block below. Paler columns reuse another layer’s keys and values.
Modelled costs
From the cost model, batch size 1. Totals the lab states are in the table above; differences come from rounding, from what a lab counts, or from parts the model does not describe (listed on the about page).
| Parameters (modelled) | 1.56B |
|---|---|
| Active per token (modelled) | 1.56B |
| Without embeddings and output head | 1.48B total, 1.48B active |
| Published weights (Hugging Face count) | 1.61B |
| KV cache per token, BF16 (layers that grow with context) | 300 KiB |
| KV cache + state at 1K tokens, BF16 | 300 MiB |
| Decode FLOPs per token at 4K context | 4.37 GFLOP |
| Prefill FLOPs for a 4K prompt | 14.7 TFLOP |
KV cache against context
Every architecture field
| Field | Value | Source |
|---|---|---|
| d_model | 1,600 | config.jsonconfig.jsonn_embd |
| vocab | 50,257 | config.jsonconfig.jsonvocab_size |
| tied_embeddings | true | codemodelling codetransformers 5.18.0 gpt2: lm_head tied to wte |
| mixers.full.type | attn | codemodelling codeattention block |
| mixers.full.heads | 25 | config.jsonconfig.jsonn_head |
| mixers.full.kv_heads | 25 | config.jsonconfig.jsonn_head |
| mixers.full.head_dim | 64 | codemodelling codetransformers 5.18.0: head_dim = hidden_size / n_head |
| mixers.full.bias | true | codemodelling codec_attn/c_proj have biases |
| ffns.dense.type | dense | codemodelling codeMLP |
| ffns.dense.d_ff | 6,400 | codemodelling codetransformers 5.18.0 gpt2: n_inner defaults to 4 * n_embd |
| ffns.dense.gated | false | codemodelling codeGELU MLP, two matrices |
| ffns.dense.bias | true | codemodelling codeMLP biases |
| layout | 48× full/dense | config.jsonconfig.jsonn_layer |
| extra_embedding_params | 1,638,400 | codemodelling codelearned position embeddings: n_positions x n_embd |
Sources
- config.json @ 15ea56d
- model card
- announcement
- modelling code · transformers 5.18.0 modelling code, or the model repository's own modelling file at the pinned revision
Listed in the LLM Architecture Gallery checklist as “GPT-2 XL (1.5B)” (name only; see about).