llm-architectures-explained

/models

Gemma 4 12B

Google · Gemma 4 · open weights · multimodal (text stack modelled)

Facts and where they come from

Released2026-05config.jsonconfig.jsonHugging Face repository creation date (api.createdAt)
Licenceapache-2.0config.jsonconfig.jsonREADME metadata: license
Total parameters11.9Blabmodel cardREADME: Total Parameters 11.95B
Active parametersnot disclosednot disclosed
Context length256K tokenslabmodel cardREADME: 256K tokens
Norm placementsandwichcodemodelling codetransformers 5.18.0 gemma4: pre and post norms around attention and MLP
Norm typeRMSNormcodemodelling codetransformers 5.18.0 gemma4: pre and post norms around attention and MLP
QK-normyescodemodelling codetransformers 5.18.0 gemma4: q_norm and k_norm
Positional encodingRoPE on 25% of each headconfig.jsonconfig.jsontext_config.rope_parameters.full_attention.partial_rotary_factor (global layers)
Parallel attention and MLPnocodemodelling codetransformers 5.18.0 gemma4: pre and post norms around attention and MLP

Architecture, drawn from the data

40× GQA 16q/8kv, window 1024 + 8× MQA 16q/1kv. Each column is one layer: its token mixer above, its feed-forward block below. Paler columns reuse another layer’s keys and values.

Gemma 4 12B: layer stack and blockslayers (48)mixer / FFNlayer 0: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 0: Gated MLP: 15360layer 1: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 1: Gated MLP: 15360layer 2: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 2: Gated MLP: 15360layer 3: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 3: Gated MLP: 15360layer 4: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 4: Gated MLP: 15360layer 5: MQA: 16 query / 1 KV heads · head 512 · K = Vlayer 5: Gated MLP: 15360layer 6: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 6: Gated MLP: 15360layer 7: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 7: Gated MLP: 15360layer 8: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 8: Gated MLP: 15360layer 9: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 9: Gated MLP: 15360layer 10: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 10: Gated MLP: 15360layer 11: MQA: 16 query / 1 KV heads · head 512 · K = Vlayer 11: Gated MLP: 15360layer 12: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 12: Gated MLP: 15360layer 13: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 13: Gated MLP: 15360layer 14: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 14: Gated MLP: 15360layer 15: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 15: Gated MLP: 15360layer 16: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 16: Gated MLP: 15360layer 17: MQA: 16 query / 1 KV heads · head 512 · K = Vlayer 17: Gated MLP: 15360layer 18: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 18: Gated MLP: 15360layer 19: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 19: Gated MLP: 15360layer 20: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 20: Gated MLP: 15360layer 21: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 21: Gated MLP: 15360layer 22: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 22: Gated MLP: 15360layer 23: MQA: 16 query / 1 KV heads · head 512 · K = Vlayer 23: Gated MLP: 15360layer 24: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 24: Gated MLP: 15360layer 25: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 25: Gated MLP: 15360layer 26: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 26: Gated MLP: 15360layer 27: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 27: Gated MLP: 15360layer 28: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 28: Gated MLP: 15360layer 29: MQA: 16 query / 1 KV heads · head 512 · K = Vlayer 29: Gated MLP: 15360layer 30: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 30: Gated MLP: 15360layer 31: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 31: Gated MLP: 15360layer 32: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 32: Gated MLP: 15360layer 33: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 33: Gated MLP: 15360layer 34: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 34: Gated MLP: 15360layer 35: MQA: 16 query / 1 KV heads · head 512 · K = Vlayer 35: Gated MLP: 15360layer 36: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 36: Gated MLP: 15360layer 37: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 37: Gated MLP: 15360layer 38: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 38: Gated MLP: 15360layer 39: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 39: Gated MLP: 15360layer 40: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 40: Gated MLP: 15360layer 41: MQA: 16 query / 1 KV heads · head 512 · K = Vlayer 41: Gated MLP: 15360layer 42: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 42: Gated MLP: 15360layer 43: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 43: Gated MLP: 15360layer 44: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 44: Gated MLP: 15360layer 45: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 45: Gated MLP: 15360layer 46: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 46: Gated MLP: 15360layer 47: MQA: 16 query / 1 KV heads · head 512 · K = Vlayer 47: Gated MLP: 1536002447× 40normGQA: 16 query / 8 KV heads · head 256 · window 1,024norm+normGated MLP: 15360norm+× 8normMQA: 16 query / 1 KV heads · head 512 · K = Vnorm+normGated MLP: 15360norm+sliding windowfull attentiondense FFN

Modelled costs

From the cost model, batch size 1. Totals the lab states are in the table above; differences come from rounding, from what a lab counts, or from parts the model does not describe (listed on the about page).

Parameters (modelled)11.9B
Active per token (modelled)11.9B
Without embeddings and output head10.9B total, 10.9B active
Published weights (Hugging Face count)12B
KV cache per token, BF16 (layers that grow with context)8 KiB
KV cache + state at 256K tokens, BF162.31 GiB
Decode FLOPs per token at 4K context25.6 GFLOP
Prefill FLOPs for a 4K prompt93.9 TFLOP

KV cache against context

Gemma 4 12B: KV cache bytes against context length101001,00010,000100,000980 KiB9.5 MiB95 MiB950 MiBcontext (tokens)KV cache + state (BF16)Gemma 4 12B

Compare with other models →

Every architecture field

FieldValueSource
d_model3,840config.jsonconfig.jsontext_config.hidden_size
vocab262,144config.jsonconfig.jsontext_config.vocab_size
tied_embeddingstruecodemodelling codetransformers 5.18.0 gemma4: tie_word_embeddings default true
mixers.full.typeattncodemodelling codeattention
mixers.full.heads16config.jsonconfig.jsontext_config.num_attention_heads
mixers.full.kv_heads1config.jsonconfig.jsontext_config.num_global_key_value_heads
mixers.full.head_dim512config.jsonconfig.jsontext_config.global_head_dim
mixers.full.qk_normtruecodemodelling codetransformers 5.18.0 gemma4: q_norm and k_norm
mixers.full.k_eq_vtrueconfig.jsonconfig.jsontext_config.attention_k_eq_v
mixers.sliding.typeattncodemodelling codeattention block
mixers.sliding.heads16config.jsonconfig.jsontext_config.num_attention_heads
mixers.sliding.kv_heads8config.jsonconfig.jsontext_config.num_key_value_heads
mixers.sliding.head_dim256config.jsonconfig.jsontext_config.head_dim
mixers.sliding.window1,024config.jsonconfig.jsontext_config.sliding_window
mixers.sliding.qk_normtruecodemodelling codetransformers 5.18.0 gemma4: q_norm and k_norm
ffns.dense.typedensecodemodelling codeMLP block
ffns.dense.d_ff15,360config.jsonconfig.jsontext_config.intermediate_size
ffns.dense.gatedtruecodemodelling codeMLP: gated (SwiGLU/GeGLU)
layout5× sliding/dense · 1× full/dense · 5× sliding/dense · 1× full/dense · 5× sliding/dense · 1× full/dense · 5× sliding/dense · 1× full/dense · 5× sliding/dense · 1× full/dense · 5× sliding/dense · 1× full/dense · 5× sliding/dense · 1× full/dense · 5× sliding/dense · 1× full/denseconfig.jsonconfig.jsontext_config.layer_types, num_kv_shared_layers
norms_per_layer4codemodelling codetransformers 5.18.0 gemma4: sandwich norms (pre and post around attention and MLP)

Sources

Listed in the LLM Architecture Gallery checklist as “Gemma 4 (12B)” (name only; see about).