llm-architectures-explained

/models

Gemma 4 26B-A4B

Google · Gemma 4 · open weights · multimodal (text stack modelled)

Facts and where they come from

Released2026-03config.jsonconfig.jsonHugging Face repository creation date (api.createdAt)
Licenceapache-2.0config.jsonconfig.jsonREADME metadata: license
Total parameters26Blabmodel cardmodel name gemma-4-26B-A4B
Active parameters4Blabmodel cardmodel name ...-A4B
Context length256K tokenslabmodel cardREADME: 256K tokens
Norm placementsandwichcodemodelling codetransformers 5.18.0 gemma4: pre and post norms around attention and MLP
Norm typeRMSNormcodemodelling codetransformers 5.18.0 gemma4: pre and post norms around attention and MLP
QK-normyescodemodelling codetransformers 5.18.0 gemma4: q_norm and k_norm
Positional encodingRoPE on 25% of each headconfig.jsonconfig.jsontext_config.rope_parameters.full_attention.partial_rotary_factor (global layers)
Parallel attention and MLPnocodemodelling codetransformers 5.18.0 gemma4: pre and post norms around attention and MLP

Architecture, drawn from the data

25× GQA 16q/8kv, window 1024 + 5× GQA 16q/2kv. Each column is one layer: its token mixer above, its feed-forward block below. Paler columns reuse another layer’s keys and values.

Gemma 4 26B-A4B: layer stack and blockslayers (30)mixer / FFNlayer 0: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 0: MoE: 128 experts, 8 active · expert 704 · + dense 2112layer 1: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 1: MoE: 128 experts, 8 active · expert 704 · + dense 2112layer 2: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 2: MoE: 128 experts, 8 active · expert 704 · + dense 2112layer 3: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 3: MoE: 128 experts, 8 active · expert 704 · + dense 2112layer 4: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 4: MoE: 128 experts, 8 active · expert 704 · + dense 2112layer 5: GQA: 16 query / 2 KV heads · head 512 · K = Vlayer 5: MoE: 128 experts, 8 active · expert 704 · + dense 2112layer 6: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 6: MoE: 128 experts, 8 active · expert 704 · + dense 2112layer 7: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 7: MoE: 128 experts, 8 active · expert 704 · + dense 2112layer 8: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 8: MoE: 128 experts, 8 active · expert 704 · + dense 2112layer 9: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 9: MoE: 128 experts, 8 active · expert 704 · + dense 2112layer 10: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 10: MoE: 128 experts, 8 active · expert 704 · + dense 2112layer 11: GQA: 16 query / 2 KV heads · head 512 · K = Vlayer 11: MoE: 128 experts, 8 active · expert 704 · + dense 2112layer 12: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 12: MoE: 128 experts, 8 active · expert 704 · + dense 2112layer 13: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 13: MoE: 128 experts, 8 active · expert 704 · + dense 2112layer 14: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 14: MoE: 128 experts, 8 active · expert 704 · + dense 2112layer 15: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 15: MoE: 128 experts, 8 active · expert 704 · + dense 2112layer 16: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 16: MoE: 128 experts, 8 active · expert 704 · + dense 2112layer 17: GQA: 16 query / 2 KV heads · head 512 · K = Vlayer 17: MoE: 128 experts, 8 active · expert 704 · + dense 2112layer 18: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 18: MoE: 128 experts, 8 active · expert 704 · + dense 2112layer 19: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 19: MoE: 128 experts, 8 active · expert 704 · + dense 2112layer 20: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 20: MoE: 128 experts, 8 active · expert 704 · + dense 2112layer 21: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 21: MoE: 128 experts, 8 active · expert 704 · + dense 2112layer 22: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 22: MoE: 128 experts, 8 active · expert 704 · + dense 2112layer 23: GQA: 16 query / 2 KV heads · head 512 · K = Vlayer 23: MoE: 128 experts, 8 active · expert 704 · + dense 2112layer 24: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 24: MoE: 128 experts, 8 active · expert 704 · + dense 2112layer 25: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 25: MoE: 128 experts, 8 active · expert 704 · + dense 2112layer 26: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 26: MoE: 128 experts, 8 active · expert 704 · + dense 2112layer 27: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 27: MoE: 128 experts, 8 active · expert 704 · + dense 2112layer 28: GQA: 16 query / 8 KV heads · head 256 · window 1,024layer 28: MoE: 128 experts, 8 active · expert 704 · + dense 2112layer 29: GQA: 16 query / 2 KV heads · head 512 · K = Vlayer 29: MoE: 128 experts, 8 active · expert 704 · + dense 211201529× 25normGQA: 16 query / 8 KV heads · head 256 · window 1,024norm+normMoE: 128 experts, 8 active · expert 704 · + dense 2112norm+× 5normGQA: 16 query / 2 KV heads · head 512 · K = Vnorm+normMoE: 128 experts, 8 active · expert 704 · + dense 2112norm+sliding windowfull attentionMoE FFN

Modelled costs

From the cost model, batch size 1. Totals the lab states are in the table above; differences come from rounding, from what a lab counts, or from parts the model does not describe (listed on the about page).

Parameters (modelled)25.2B
Active per token (modelled)3.82B
Without embeddings and output head24.5B total, 3.08B active
Published weights (Hugging Face count)25.8B
KV cache per token, BF16 (layers that grow with context)10 KiB
KV cache + state at 256K tokens, BF162.7 GiB
Decode FLOPs per token at 4K context8.73 GFLOP
Prefill FLOPs for a 4K prompt28.1 TFLOP

KV cache against context

Gemma 4 26B-A4B: KV cache bytes against context length101001,00010,000100,000980 KiB9.5 MiB95 MiB950 MiBcontext (tokens)KV cache + state (BF16)Gemma 4 26B-A4B

Compare with other models →

Every architecture field

FieldValueSource
d_model2,816config.jsonconfig.jsontext_config.hidden_size
vocab262,144config.jsonconfig.jsontext_config.vocab_size
tied_embeddingstruecodemodelling codetransformers 5.18.0 gemma4: tie_word_embeddings default true
mixers.full.typeattncodemodelling codeattention
mixers.full.heads16config.jsonconfig.jsontext_config.num_attention_heads
mixers.full.kv_heads2config.jsonconfig.jsontext_config.num_global_key_value_heads
mixers.full.head_dim512config.jsonconfig.jsontext_config.global_head_dim
mixers.full.qk_normtruecodemodelling codetransformers 5.18.0 gemma4: q_norm and k_norm
mixers.full.k_eq_vtrueconfig.jsonconfig.jsontext_config.attention_k_eq_v
mixers.sliding.typeattncodemodelling codeattention block
mixers.sliding.heads16config.jsonconfig.jsontext_config.num_attention_heads
mixers.sliding.kv_heads8config.jsonconfig.jsontext_config.num_key_value_heads
mixers.sliding.head_dim256config.jsonconfig.jsontext_config.head_dim
mixers.sliding.window1,024config.jsonconfig.jsontext_config.sliding_window
mixers.sliding.qk_normtruecodemodelling codetransformers 5.18.0 gemma4: q_norm and k_norm
ffns.dense.typedensecodemodelling codeMLP block
ffns.dense.d_ff2,112config.jsonconfig.jsontext_config.intermediate_size
ffns.dense.gatedtruecodemodelling codeMLP: gated (SwiGLU/GeGLU)
ffns.moe.typemoecodemodelling codeMoE block
ffns.moe.experts128config.jsonconfig.jsontext_config.num_experts
ffns.moe.active8config.jsonconfig.jsontext_config.top_k_experts
ffns.moe.d_expert704config.jsonconfig.jsontext_config.moe_intermediate_size
ffns.moe.gatedtruecodemodelling codeexperts are gated MLPs
ffns.moe.dense_parallel_d_ff2,112config.jsonconfig.jsontext_config.intermediate_size
layout5× sliding/moe · 1× full/moe · 5× sliding/moe · 1× full/moe · 5× sliding/moe · 1× full/moe · 5× sliding/moe · 1× full/moe · 5× sliding/moe · 1× full/moeconfig.jsonconfig.jsontext_config.layer_types, num_kv_shared_layers
norms_per_layer4codemodelling codetransformers 5.18.0 gemma4: sandwich norms (pre and post around attention and MLP)

Sources

Listed in the LLM Architecture Gallery checklist as “Gemma 4 (26B-A4B)” (name only; see about).