Gemma 4 31B
Google · Gemma 4 · open weights · multimodal (text stack modelled)
- GQA
- Sliding window
- RoPE
- Partial RoPE
- Sandwich norm
- QK-norm
Facts and where they come from
| Released | 2026-03 | config.jsonconfig.jsonHugging Face repository creation date (api.createdAt) |
|---|---|---|
| Licence | apache-2.0 | config.jsonconfig.jsonREADME metadata: license |
| Total parameters | 30.7B | labmodel cardREADME: Total Parameters 30.7B |
| Active parameters | not disclosed | not disclosed |
| Context length | 256K tokens | labmodel cardREADME: 256K tokens |
| Norm placement | sandwich | codemodelling codetransformers 5.18.0 gemma4: pre and post norms around attention and MLP |
| Norm type | RMSNorm | codemodelling codetransformers 5.18.0 gemma4: pre and post norms around attention and MLP |
| QK-norm | yes | codemodelling codetransformers 5.18.0 gemma4: q_norm and k_norm |
| Positional encoding | RoPE on 25% of each head | config.jsonconfig.jsontext_config.rope_parameters.full_attention.partial_rotary_factor (global layers) |
| Parallel attention and MLP | no | codemodelling codetransformers 5.18.0 gemma4: pre and post norms around attention and MLP |
Architecture, drawn from the data
50× GQA 32q/16kv, window 1024 + 10× GQA 32q/4kv. Each column is one layer: its token mixer above, its feed-forward block below. Paler columns reuse another layer’s keys and values.
Modelled costs
From the cost model, batch size 1. Totals the lab states are in the table above; differences come from rounding, from what a lab counts, or from parts the model does not describe (listed on the about page).
| Parameters (modelled) | 30.7B |
|---|---|
| Active per token (modelled) | 30.7B |
| Without embeddings and output head | 29.3B total, 29.3B active |
| Published weights (Hugging Face count) | 31.3B |
| KV cache per token, BF16 (layers that grow with context) | 40 KiB |
| KV cache + state at 256K tokens, BF16 | 10.8 GiB |
| Decode FLOPs per token at 4K context | 65.8 GFLOP |
| Prefill FLOPs for a 4K prompt | 251 TFLOP |
KV cache against context
Every architecture field
| Field | Value | Source |
|---|---|---|
| d_model | 5,376 | config.jsonconfig.jsontext_config.hidden_size |
| vocab | 262,144 | config.jsonconfig.jsontext_config.vocab_size |
| tied_embeddings | true | codemodelling codetransformers 5.18.0 gemma4: tie_word_embeddings default true |
| mixers.full.type | attn | codemodelling codeattention |
| mixers.full.heads | 32 | config.jsonconfig.jsontext_config.num_attention_heads |
| mixers.full.kv_heads | 4 | config.jsonconfig.jsontext_config.num_global_key_value_heads |
| mixers.full.head_dim | 512 | config.jsonconfig.jsontext_config.global_head_dim |
| mixers.full.qk_norm | true | codemodelling codetransformers 5.18.0 gemma4: q_norm and k_norm |
| mixers.full.k_eq_v | true | config.jsonconfig.jsontext_config.attention_k_eq_v |
| mixers.sliding.type | attn | codemodelling codeattention block |
| mixers.sliding.heads | 32 | config.jsonconfig.jsontext_config.num_attention_heads |
| mixers.sliding.kv_heads | 16 | config.jsonconfig.jsontext_config.num_key_value_heads |
| mixers.sliding.head_dim | 256 | config.jsonconfig.jsontext_config.head_dim |
| mixers.sliding.window | 1,024 | config.jsonconfig.jsontext_config.sliding_window |
| mixers.sliding.qk_norm | true | codemodelling codetransformers 5.18.0 gemma4: q_norm and k_norm |
| ffns.dense.type | dense | codemodelling codeMLP block |
| ffns.dense.d_ff | 21,504 | config.jsonconfig.jsontext_config.intermediate_size |
| ffns.dense.gated | true | codemodelling codeMLP: gated (SwiGLU/GeGLU) |
| layout | 5× sliding/dense · 1× full/dense · 5× sliding/dense · 1× full/dense · 5× sliding/dense · 1× full/dense · 5× sliding/dense · 1× full/dense · 5× sliding/dense · 1× full/dense · 5× sliding/dense · 1× full/dense · 5× sliding/dense · 1× full/dense · 5× sliding/dense · 1× full/dense · 5× sliding/dense · 1× full/dense · 5× sliding/dense · 1× full/dense | config.jsonconfig.jsontext_config.layer_types, num_kv_shared_layers |
| norms_per_layer | 4 | codemodelling codetransformers 5.18.0 gemma4: sandwich norms (pre and post around attention and MLP) |
Sources
- config.json @ 842da37
- model card
- modelling code · transformers 5.18.0 modelling code, or the model repository's own modelling file at the pinned revision
Listed in the LLM Architecture Gallery checklist as “Gemma 4 (31B)” (name only; see about).