Gemma 3 27B
Google · Gemma 3 · open weights · multimodal (text stack modelled)
- GQA
- Sliding window
- RoPE
- Sandwich norm
- QK-norm
Facts and where they come from
| Released | 2025-03 | config.jsonconfig.jsonHugging Face repository creation date (api.createdAt) |
|---|---|---|
| Licence | gemma | config.jsonconfig.jsonREADME metadata: license |
| Total parameters | 27B | labmodel cardmodel name gemma-3-27b |
| Active parameters | not disclosed | not disclosed |
| Context length | 128K tokens | labgoogle-deepmind/gemma _gemma.py (pinned)model card (README): 128K context window |
| Norm placement | sandwich | codemodelling codegoogle-deepmind/gemma _gemma.py: use_post_attn_norm and use_post_ffw_norm |
| Norm type | RMSNorm | codemodelling codegoogle-deepmind/gemma _gemma.py: use_post_attn_norm and use_post_ffw_norm |
| QK-norm | yes | config.jsonconfig.jsonuse_qk_norm |
| Positional encoding | RoPE | codemodelling coderotary on the full head (default) |
| Parallel attention and MLP | no | codemodelling codegoogle-deepmind/gemma _gemma.py: use_post_attn_norm and use_post_ffw_norm |
Architecture, drawn from the data
52× GQA 32q/16kv, window 1024 + 10× GQA 32q/16kv. Each column is one layer: its token mixer above, its feed-forward block below. Paler columns reuse another layer’s keys and values.
Modelled costs
From the cost model, batch size 1. Totals the lab states are in the table above; differences come from rounding, from what a lab counts, or from parts the model does not describe (listed on the about page).
| Parameters (modelled) | 27B |
|---|---|
| Active per token (modelled) | 27B |
| Without embeddings and output head | 25.6B total, 25.6B active |
| Published weights (Hugging Face count) | 27.4B |
| KV cache per token, BF16 (layers that grow with context) | 80 KiB |
| KV cache + state at 128K tokens, BF16 | 10.4 GiB |
| Decode FLOPs per token at 4K context | 55.6 GFLOP |
| Prefill FLOPs for a 4K prompt | 214 TFLOP |
KV cache against context
Every architecture field
| Field | Value | Source |
|---|---|---|
| d_model | 5,376 | labgoogle-deepmind/gemma _gemma.py (pinned)_gemma.py Gemma3_27B: embed_dim |
| vocab | 262,144 | labgoogle-deepmind/gemma _gemma.py (pinned)_gemma.py Gemma3_27B: num_embed |
| tied_embeddings | true | labgoogle-deepmind/gemma _gemma.py (pinned)_gemma.py Gemma3_27B: embedder shared with the output layer |
| mixers.full.type | attn | codemodelling codeattention block |
| mixers.full.heads | 32 | labgoogle-deepmind/gemma _gemma.py (pinned)_gemma.py Gemma3_27B: num_heads |
| mixers.full.kv_heads | 16 | labgoogle-deepmind/gemma _gemma.py (pinned)_gemma.py Gemma3_27B: num_kv_heads |
| mixers.full.head_dim | 128 | labgoogle-deepmind/gemma _gemma.py (pinned)_gemma.py Gemma3_27B: head_dim |
| mixers.full.qk_norm | true | labgoogle-deepmind/gemma _gemma.py (pinned)_gemma.py Gemma3_27B: use_qk_norm |
| mixers.sliding.type | attn | codemodelling codeattention block |
| mixers.sliding.heads | 32 | labgoogle-deepmind/gemma _gemma.py (pinned)_gemma.py Gemma3_27B: num_heads |
| mixers.sliding.kv_heads | 16 | labgoogle-deepmind/gemma _gemma.py (pinned)_gemma.py Gemma3_27B: num_kv_heads |
| mixers.sliding.head_dim | 128 | labgoogle-deepmind/gemma _gemma.py (pinned)_gemma.py Gemma3_27B: head_dim |
| mixers.sliding.window | 1,024 | labgoogle-deepmind/gemma _gemma.py (pinned)_gemma.py Gemma3_27B: sliding_window_size |
| mixers.sliding.qk_norm | true | labgoogle-deepmind/gemma _gemma.py (pinned)_gemma.py Gemma3_27B: use_qk_norm |
| ffns.dense.type | dense | codemodelling codeMLP block |
| ffns.dense.d_ff | 21,504 | labgoogle-deepmind/gemma _gemma.py (pinned)_gemma.py Gemma3_27B: hidden_dim |
| ffns.dense.gated | true | codemodelling codeMLP: gated (SwiGLU/GeGLU) |
| layout | 5× sliding/dense · 1× full/dense · 5× sliding/dense · 1× full/dense · 5× sliding/dense · 1× full/dense · 5× sliding/dense · 1× full/dense · 5× sliding/dense · 1× full/dense · 5× sliding/dense · 1× full/dense · 5× sliding/dense · 1× full/dense · 5× sliding/dense · 1× full/dense · 5× sliding/dense · 1× full/dense · 5× sliding/dense · 1× full/dense · 2× sliding/dense | labgoogle-deepmind/gemma _gemma.py (pinned)_gemma.py Gemma3_27B: attention_types (local sliding / global pattern) |
| norms_per_layer | 4 | codemodelling codeGemma 2/3: sandwich norms (pre and post around attention and MLP) |
Sources
- config.json @ 005ad34
- model card
- arXiv 2503.19786
- google-deepmind/gemma _gemma.py (pinned) · google-deepmind/gemma _gemma.py (pinned)
- modelling code · transformers 5.18.0 modelling code, or the model repository's own modelling file at the pinned revision
Listed in the LLM Architecture Gallery checklist as “Gemma 3 (27B)” (name only; see about).