llm-architectures-explained

/models

gpt-oss-120b

OpenAI · gpt-oss · open weights

Facts and where they come from

Released2025-08config.jsonconfig.jsonHugging Face repository creation date (api.createdAt)
Licenceapache-2.0config.jsonconfig.jsonREADME metadata: license
Total parameters117Blabmodel cardREADME: 117B parameters with 5.1B active
Active parameters5.1Blabmodel cardREADME: 5.1B active
Context length128K tokensconfig.jsonconfig.jsonmax_position_embeddings
Norm placementprecodemodelling codetransformers 5.18.0 / repo modelling code for gpt_oss: input_layernorm before attention, post_attention_layernorm before the MLP
Norm typeRMSNormcodemodelling codetransformers 5.18.0 / repo modelling code for gpt_oss: input_layernorm before attention, post_attention_layernorm before the MLP
QK-normnocodemodelling codeno q/k normalisation in the attention block
Positional encodingRoPEcodemodelling coderotary on the full head (default)
Parallel attention and MLPnocodemodelling codetransformers 5.18.0 / repo modelling code for gpt_oss: input_layernorm before attention, post_attention_layernorm before the MLP

Architecture, drawn from the data

18× GQA 64q/8kv, window 128 + 18× GQA 64q/8kv. Each column is one layer: its token mixer above, its feed-forward block below. Paler columns reuse another layer’s keys and values.

gpt-oss-120b: layer stack and blockslayers (36)mixer / FFNlayer 0: GQA: 64 query / 8 KV heads · head 64 · window 128layer 0: MoE: 128 experts, 4 active · expert 2880layer 1: GQA: 64 query / 8 KV heads · head 64layer 1: MoE: 128 experts, 4 active · expert 2880layer 2: GQA: 64 query / 8 KV heads · head 64 · window 128layer 2: MoE: 128 experts, 4 active · expert 2880layer 3: GQA: 64 query / 8 KV heads · head 64layer 3: MoE: 128 experts, 4 active · expert 2880layer 4: GQA: 64 query / 8 KV heads · head 64 · window 128layer 4: MoE: 128 experts, 4 active · expert 2880layer 5: GQA: 64 query / 8 KV heads · head 64layer 5: MoE: 128 experts, 4 active · expert 2880layer 6: GQA: 64 query / 8 KV heads · head 64 · window 128layer 6: MoE: 128 experts, 4 active · expert 2880layer 7: GQA: 64 query / 8 KV heads · head 64layer 7: MoE: 128 experts, 4 active · expert 2880layer 8: GQA: 64 query / 8 KV heads · head 64 · window 128layer 8: MoE: 128 experts, 4 active · expert 2880layer 9: GQA: 64 query / 8 KV heads · head 64layer 9: MoE: 128 experts, 4 active · expert 2880layer 10: GQA: 64 query / 8 KV heads · head 64 · window 128layer 10: MoE: 128 experts, 4 active · expert 2880layer 11: GQA: 64 query / 8 KV heads · head 64layer 11: MoE: 128 experts, 4 active · expert 2880layer 12: GQA: 64 query / 8 KV heads · head 64 · window 128layer 12: MoE: 128 experts, 4 active · expert 2880layer 13: GQA: 64 query / 8 KV heads · head 64layer 13: MoE: 128 experts, 4 active · expert 2880layer 14: GQA: 64 query / 8 KV heads · head 64 · window 128layer 14: MoE: 128 experts, 4 active · expert 2880layer 15: GQA: 64 query / 8 KV heads · head 64layer 15: MoE: 128 experts, 4 active · expert 2880layer 16: GQA: 64 query / 8 KV heads · head 64 · window 128layer 16: MoE: 128 experts, 4 active · expert 2880layer 17: GQA: 64 query / 8 KV heads · head 64layer 17: MoE: 128 experts, 4 active · expert 2880layer 18: GQA: 64 query / 8 KV heads · head 64 · window 128layer 18: MoE: 128 experts, 4 active · expert 2880layer 19: GQA: 64 query / 8 KV heads · head 64layer 19: MoE: 128 experts, 4 active · expert 2880layer 20: GQA: 64 query / 8 KV heads · head 64 · window 128layer 20: MoE: 128 experts, 4 active · expert 2880layer 21: GQA: 64 query / 8 KV heads · head 64layer 21: MoE: 128 experts, 4 active · expert 2880layer 22: GQA: 64 query / 8 KV heads · head 64 · window 128layer 22: MoE: 128 experts, 4 active · expert 2880layer 23: GQA: 64 query / 8 KV heads · head 64layer 23: MoE: 128 experts, 4 active · expert 2880layer 24: GQA: 64 query / 8 KV heads · head 64 · window 128layer 24: MoE: 128 experts, 4 active · expert 2880layer 25: GQA: 64 query / 8 KV heads · head 64layer 25: MoE: 128 experts, 4 active · expert 2880layer 26: GQA: 64 query / 8 KV heads · head 64 · window 128layer 26: MoE: 128 experts, 4 active · expert 2880layer 27: GQA: 64 query / 8 KV heads · head 64layer 27: MoE: 128 experts, 4 active · expert 2880layer 28: GQA: 64 query / 8 KV heads · head 64 · window 128layer 28: MoE: 128 experts, 4 active · expert 2880layer 29: GQA: 64 query / 8 KV heads · head 64layer 29: MoE: 128 experts, 4 active · expert 2880layer 30: GQA: 64 query / 8 KV heads · head 64 · window 128layer 30: MoE: 128 experts, 4 active · expert 2880layer 31: GQA: 64 query / 8 KV heads · head 64layer 31: MoE: 128 experts, 4 active · expert 2880layer 32: GQA: 64 query / 8 KV heads · head 64 · window 128layer 32: MoE: 128 experts, 4 active · expert 2880layer 33: GQA: 64 query / 8 KV heads · head 64layer 33: MoE: 128 experts, 4 active · expert 2880layer 34: GQA: 64 query / 8 KV heads · head 64 · window 128layer 34: MoE: 128 experts, 4 active · expert 2880layer 35: GQA: 64 query / 8 KV heads · head 64layer 35: MoE: 128 experts, 4 active · expert 288001835× 18normGQA: 64 query / 8 KV heads · head 64 · window 128+normMoE: 128 experts, 4 active · expert 2880+× 18normGQA: 64 query / 8 KV heads · head 64+normMoE: 128 experts, 4 active · expert 2880+sliding windowfull attentionMoE FFN

Modelled costs

From the cost model, batch size 1. Totals the lab states are in the table above; differences come from rounding, from what a lab counts, or from parts the model does not describe (listed on the about page).

Parameters (modelled)117B
Active per token (modelled)5.71B
Without embeddings and output head116B total, 4.55B active
Published weights (Hugging Face count)117B (packed low-bit tensors, so not comparable)
KV cache per token, BF16 (layers that grow with context)36 KiB
KV cache + state at 128K tokens, BF164.5 GiB
Decode FLOPs per token at 4K context11.5 GFLOP
Prefill FLOPs for a 4K prompt39.9 TFLOP

KV cache against context

gpt-oss-120b: KV cache bytes against context length101001,00010,000100,000980 KiB9.5 MiB95 MiB950 MiBcontext (tokens)KV cache + state (BF16)gpt-oss-120b

Compare with other models →

Every architecture field

FieldValueSource
d_model2,880config.jsonconfig.jsonhidden_size
vocab201,088config.jsonconfig.jsonvocab_size
tied_embeddingsfalseconfig.jsonconfig.jsontie_word_embeddings
mixers.full.typeattncodemodelling codeattention block
mixers.full.heads64config.jsonconfig.jsonnum_attention_heads
mixers.full.kv_heads8config.jsonconfig.jsonnum_key_value_heads
mixers.full.head_dim64config.jsonconfig.jsonhead_dim
mixers.full.biastruecodemodelling codeattention_bias: true
mixers.sliding.typeattncodemodelling codeattention block
mixers.sliding.heads64config.jsonconfig.jsonnum_attention_heads
mixers.sliding.kv_heads8config.jsonconfig.jsonnum_key_value_heads
mixers.sliding.head_dim64config.jsonconfig.jsonhead_dim
mixers.sliding.window128config.jsonconfig.jsonsliding_window
mixers.sliding.biastruecodemodelling codeattention_bias: true
ffns.moe.typemoecodemodelling codeMoE block
ffns.moe.experts128config.jsonconfig.jsonnum_local_experts
ffns.moe.active4config.jsonconfig.jsonnum_experts_per_tok
ffns.moe.d_expert2,880config.jsonconfig.jsonintermediate_size
ffns.moe.gatedtruecodemodelling codeexperts are gated MLPs
layout1× sliding/moe · 1× full/moe · 1× sliding/moe · 1× full/moe · 1× sliding/moe · 1× full/moe · 1× sliding/moe · 1× full/moe · 1× sliding/moe · 1× full/moe · 1× sliding/moe · 1× full/moe · 1× sliding/moe · 1× full/moe · 1× sliding/moe · 1× full/moe · 1× sliding/moe · 1× full/moe · 1× sliding/moe · 1× full/moe · 1× sliding/moe · 1× full/moe · 1× sliding/moe · 1× full/moe · 1× sliding/moe · 1× full/moe · 1× sliding/moe · 1× full/moe · 1× sliding/moe · 1× full/moe · 1× sliding/moe · 1× full/moe · 1× sliding/moe · 1× full/moe · 1× sliding/moe · 1× full/moeconfig.jsonconfig.jsonlayer_types

Sources

Listed in the LLM Architecture Gallery checklist as “GPT-OSS (120B)” (name only; see about).