llm-architectures-explained

/models

Gemma 4 E4B

Google · Gemma 4 · open weights · multimodal (text stack modelled)

Facts and where they come from

Released2026-03config.jsonconfig.jsonHugging Face repository creation date (api.createdAt)
Licenceapache-2.0config.jsonconfig.jsonREADME metadata: license
Total parameters8Blabmodel cardREADME: 4.5B effective (8B with embeddings)
Active parametersnot disclosednot disclosed
Context length128K tokenslabmodel cardREADME: 128K tokens
Norm placementsandwichcodemodelling codetransformers 5.18.0 gemma4: pre and post norms around attention and MLP
Norm typeRMSNormcodemodelling codetransformers 5.18.0 gemma4: pre and post norms around attention and MLP
QK-normyescodemodelling codetransformers 5.18.0 gemma4: q_norm and k_norm
Positional encodingRoPE on 25% of each headconfig.jsonconfig.jsontext_config.rope_parameters.full_attention.partial_rotary_factor (global layers)
Parallel attention and MLPnocodemodelling codetransformers 5.18.0 gemma4: pre and post norms around attention and MLP

Architecture, drawn from the data

35× GQA 8q/2kv, window 512 + 7× GQA 8q/2kv. Each column is one layer: its token mixer above, its feed-forward block below. Paler columns reuse another layer’s keys and values.

Gemma 4 E4B: layer stack and blockslayers (42)mixer / FFNlayer 0: GQA: 8 query / 2 KV heads · head 256 · window 512layer 0: Gated MLP: 10240layer 1: GQA: 8 query / 2 KV heads · head 256 · window 512layer 1: Gated MLP: 10240layer 2: GQA: 8 query / 2 KV heads · head 256 · window 512layer 2: Gated MLP: 10240layer 3: GQA: 8 query / 2 KV heads · head 256 · window 512layer 3: Gated MLP: 10240layer 4: GQA: 8 query / 2 KV heads · head 256 · window 512layer 4: Gated MLP: 10240layer 5: GQA: 8 query / 2 KV heads · head 512layer 5: Gated MLP: 10240layer 6: GQA: 8 query / 2 KV heads · head 256 · window 512layer 6: Gated MLP: 10240layer 7: GQA: 8 query / 2 KV heads · head 256 · window 512layer 7: Gated MLP: 10240layer 8: GQA: 8 query / 2 KV heads · head 256 · window 512layer 8: Gated MLP: 10240layer 9: GQA: 8 query / 2 KV heads · head 256 · window 512layer 9: Gated MLP: 10240layer 10: GQA: 8 query / 2 KV heads · head 256 · window 512layer 10: Gated MLP: 10240layer 11: GQA: 8 query / 2 KV heads · head 512layer 11: Gated MLP: 10240layer 12: GQA: 8 query / 2 KV heads · head 256 · window 512layer 12: Gated MLP: 10240layer 13: GQA: 8 query / 2 KV heads · head 256 · window 512layer 13: Gated MLP: 10240layer 14: GQA: 8 query / 2 KV heads · head 256 · window 512layer 14: Gated MLP: 10240layer 15: GQA: 8 query / 2 KV heads · head 256 · window 512layer 15: Gated MLP: 10240layer 16: GQA: 8 query / 2 KV heads · head 256 · window 512layer 16: Gated MLP: 10240layer 17: GQA: 8 query / 2 KV heads · head 512layer 17: Gated MLP: 10240layer 18: GQA: 8 query / 2 KV heads · head 256 · window 512layer 18: Gated MLP: 10240layer 19: GQA: 8 query / 2 KV heads · head 256 · window 512layer 19: Gated MLP: 10240layer 20: GQA: 8 query / 2 KV heads · head 256 · window 512layer 20: Gated MLP: 10240layer 21: GQA: 8 query / 2 KV heads · head 256 · window 512layer 21: Gated MLP: 10240layer 22: GQA: 8 query / 2 KV heads · head 256 · window 512layer 22: Gated MLP: 10240layer 23: GQA: 8 query / 2 KV heads · head 512layer 23: Gated MLP: 10240layer 24: GQA: 8 query / 2 KV heads · head 256 · window 512 (reuses another layer's KV)layer 24: Gated MLP: 10240layer 25: GQA: 8 query / 2 KV heads · head 256 · window 512 (reuses another layer's KV)layer 25: Gated MLP: 10240layer 26: GQA: 8 query / 2 KV heads · head 256 · window 512 (reuses another layer's KV)layer 26: Gated MLP: 10240layer 27: GQA: 8 query / 2 KV heads · head 256 · window 512 (reuses another layer's KV)layer 27: Gated MLP: 10240layer 28: GQA: 8 query / 2 KV heads · head 256 · window 512 (reuses another layer's KV)layer 28: Gated MLP: 10240layer 29: GQA: 8 query / 2 KV heads · head 512 (reuses another layer's KV)layer 29: Gated MLP: 10240layer 30: GQA: 8 query / 2 KV heads · head 256 · window 512 (reuses another layer's KV)layer 30: Gated MLP: 10240layer 31: GQA: 8 query / 2 KV heads · head 256 · window 512 (reuses another layer's KV)layer 31: Gated MLP: 10240layer 32: GQA: 8 query / 2 KV heads · head 256 · window 512 (reuses another layer's KV)layer 32: Gated MLP: 10240layer 33: GQA: 8 query / 2 KV heads · head 256 · window 512 (reuses another layer's KV)layer 33: Gated MLP: 10240layer 34: GQA: 8 query / 2 KV heads · head 256 · window 512 (reuses another layer's KV)layer 34: Gated MLP: 10240layer 35: GQA: 8 query / 2 KV heads · head 512 (reuses another layer's KV)layer 35: Gated MLP: 10240layer 36: GQA: 8 query / 2 KV heads · head 256 · window 512 (reuses another layer's KV)layer 36: Gated MLP: 10240layer 37: GQA: 8 query / 2 KV heads · head 256 · window 512 (reuses another layer's KV)layer 37: Gated MLP: 10240layer 38: GQA: 8 query / 2 KV heads · head 256 · window 512 (reuses another layer's KV)layer 38: Gated MLP: 10240layer 39: GQA: 8 query / 2 KV heads · head 256 · window 512 (reuses another layer's KV)layer 39: Gated MLP: 10240layer 40: GQA: 8 query / 2 KV heads · head 256 · window 512 (reuses another layer's KV)layer 40: Gated MLP: 10240layer 41: GQA: 8 query / 2 KV heads · head 512 (reuses another layer's KV)layer 41: Gated MLP: 1024002141× 35normGQA: 8 query / 2 KV heads · head 256 · window 512norm+normGated MLP: 10240norm+× 7normGQA: 8 query / 2 KV heads · head 512norm+normGated MLP: 10240norm+sliding windowfull attentiondense FFN

Modelled costs

From the cost model, batch size 1. Totals the lab states are in the table above; differences come from rounding, from what a lab counts, or from parts the model does not describe (listed on the about page).

Parameters (modelled)7.38B
Active per token (modelled)7.38B
Without embeddings and output head3.89B total, 3.89B active
Published weights (Hugging Face count)8B
KV cache per token, BF16 (layers that grow with context)16 KiB
KV cache + state at 128K tokens, BF162.02 GiB
Decode FLOPs per token at 4K context9.74 GFLOP
Prefill FLOPs for a 4K prompt33.4 TFLOP

KV cache against context

Gemma 4 E4B: KV cache bytes against context length101001,00010,000100,000980 KiB9.5 MiB95 MiB950 MiBcontext (tokens)KV cache + state (BF16)Gemma 4 E4B

Compare with other models →

Every architecture field

FieldValueSource
d_model2,560config.jsonconfig.jsontext_config.hidden_size
vocab262,144config.jsonconfig.jsontext_config.vocab_size
tied_embeddingstruecodemodelling codetransformers 5.18.0 gemma4: tie_word_embeddings default true
mixers.full.typeattncodemodelling codeattention
mixers.full.heads8config.jsonconfig.jsontext_config.num_attention_heads
mixers.full.kv_heads2config.jsonconfig.jsontext_config.num_key_value_heads
mixers.full.head_dim512config.jsonconfig.jsontext_config.global_head_dim
mixers.full.qk_normtruecodemodelling codetransformers 5.18.0 gemma4: q_norm and k_norm
mixers.sliding.typeattncodemodelling codeattention block
mixers.sliding.heads8config.jsonconfig.jsontext_config.num_attention_heads
mixers.sliding.kv_heads2config.jsonconfig.jsontext_config.num_key_value_heads
mixers.sliding.head_dim256config.jsonconfig.jsontext_config.head_dim
mixers.sliding.window512config.jsonconfig.jsontext_config.sliding_window
mixers.sliding.qk_normtruecodemodelling codetransformers 5.18.0 gemma4: q_norm and k_norm
ffns.dense.typedensecodemodelling codeMLP block
ffns.dense.d_ff10,240config.jsonconfig.jsontext_config.intermediate_size
ffns.dense.gatedtruecodemodelling codeMLP: gated (SwiGLU/GeGLU)
layout5× sliding/dense · 1× full/dense · 5× sliding/dense · 1× full/dense · 5× sliding/dense · 1× full/dense · 5× sliding/dense · 1× full/dense · 5× sliding/dense (kv_shared=true) · 1× full/dense (kv_shared=true) · 5× sliding/dense (kv_shared=true) · 1× full/dense (kv_shared=true) · 5× sliding/dense (kv_shared=true) · 1× full/dense (kv_shared=true)config.jsonconfig.jsontext_config.layer_types, num_kv_shared_layers
norms_per_layer4codemodelling codetransformers 5.18.0 gemma4: sandwich norms (pre and post around attention and MLP)
extra_embedding_params2,818,572,288codemodelling codetransformers 5.18.0 gemma4: per-layer embeddings, vocab_size_per_layer_input x num_hidden_layers x hidden_size_per_layer_input

Sources

Listed in the LLM Architecture Gallery checklist as “Gemma 4 (E4B)” (name only; see about).