llm-architectures-explained

/models

Gemma 2 27B

Google · Gemma 2 · open weights

Facts and where they come from

Released2024-06config.jsonconfig.jsonHugging Face repository creation date (api.createdAt)
Licencegemmaconfig.jsonconfig.jsonREADME metadata: license
Total parameters27Blabmodel cardmodel name gemma-2-27b
Active parametersnot disclosednot disclosed
Context length8K tokenslabgoogle-deepmind/gemma _gemma.py (pinned)Gemma 2 report (arXiv 2408.00118): context length 8192
Norm placementsandwichcodemodelling codegoogle-deepmind/gemma _gemma.py: use_post_attn_norm and use_post_ffw_norm
Norm typeRMSNormcodemodelling codegoogle-deepmind/gemma _gemma.py: use_post_attn_norm and use_post_ffw_norm
QK-normnoconfig.jsonconfig.jsonuse_qk_norm
Positional encodingRoPEcodemodelling coderotary on the full head (default)
Parallel attention and MLPnocodemodelling codegoogle-deepmind/gemma _gemma.py: use_post_attn_norm and use_post_ffw_norm

Architecture, drawn from the data

23× GQA 32q/16kv, window 4096 + 23× GQA 32q/16kv. Each column is one layer: its token mixer above, its feed-forward block below. Paler columns reuse another layer’s keys and values.

Gemma 2 27B: layer stack and blockslayers (46)mixer / FFNlayer 0: GQA: 32 query / 16 KV heads · head 128 · window 4,096layer 0: Gated MLP: 36864layer 1: GQA: 32 query / 16 KV heads · head 128layer 1: Gated MLP: 36864layer 2: GQA: 32 query / 16 KV heads · head 128 · window 4,096layer 2: Gated MLP: 36864layer 3: GQA: 32 query / 16 KV heads · head 128layer 3: Gated MLP: 36864layer 4: GQA: 32 query / 16 KV heads · head 128 · window 4,096layer 4: Gated MLP: 36864layer 5: GQA: 32 query / 16 KV heads · head 128layer 5: Gated MLP: 36864layer 6: GQA: 32 query / 16 KV heads · head 128 · window 4,096layer 6: Gated MLP: 36864layer 7: GQA: 32 query / 16 KV heads · head 128layer 7: Gated MLP: 36864layer 8: GQA: 32 query / 16 KV heads · head 128 · window 4,096layer 8: Gated MLP: 36864layer 9: GQA: 32 query / 16 KV heads · head 128layer 9: Gated MLP: 36864layer 10: GQA: 32 query / 16 KV heads · head 128 · window 4,096layer 10: Gated MLP: 36864layer 11: GQA: 32 query / 16 KV heads · head 128layer 11: Gated MLP: 36864layer 12: GQA: 32 query / 16 KV heads · head 128 · window 4,096layer 12: Gated MLP: 36864layer 13: GQA: 32 query / 16 KV heads · head 128layer 13: Gated MLP: 36864layer 14: GQA: 32 query / 16 KV heads · head 128 · window 4,096layer 14: Gated MLP: 36864layer 15: GQA: 32 query / 16 KV heads · head 128layer 15: Gated MLP: 36864layer 16: GQA: 32 query / 16 KV heads · head 128 · window 4,096layer 16: Gated MLP: 36864layer 17: GQA: 32 query / 16 KV heads · head 128layer 17: Gated MLP: 36864layer 18: GQA: 32 query / 16 KV heads · head 128 · window 4,096layer 18: Gated MLP: 36864layer 19: GQA: 32 query / 16 KV heads · head 128layer 19: Gated MLP: 36864layer 20: GQA: 32 query / 16 KV heads · head 128 · window 4,096layer 20: Gated MLP: 36864layer 21: GQA: 32 query / 16 KV heads · head 128layer 21: Gated MLP: 36864layer 22: GQA: 32 query / 16 KV heads · head 128 · window 4,096layer 22: Gated MLP: 36864layer 23: GQA: 32 query / 16 KV heads · head 128layer 23: Gated MLP: 36864layer 24: GQA: 32 query / 16 KV heads · head 128 · window 4,096layer 24: Gated MLP: 36864layer 25: GQA: 32 query / 16 KV heads · head 128layer 25: Gated MLP: 36864layer 26: GQA: 32 query / 16 KV heads · head 128 · window 4,096layer 26: Gated MLP: 36864layer 27: GQA: 32 query / 16 KV heads · head 128layer 27: Gated MLP: 36864layer 28: GQA: 32 query / 16 KV heads · head 128 · window 4,096layer 28: Gated MLP: 36864layer 29: GQA: 32 query / 16 KV heads · head 128layer 29: Gated MLP: 36864layer 30: GQA: 32 query / 16 KV heads · head 128 · window 4,096layer 30: Gated MLP: 36864layer 31: GQA: 32 query / 16 KV heads · head 128layer 31: Gated MLP: 36864layer 32: GQA: 32 query / 16 KV heads · head 128 · window 4,096layer 32: Gated MLP: 36864layer 33: GQA: 32 query / 16 KV heads · head 128layer 33: Gated MLP: 36864layer 34: GQA: 32 query / 16 KV heads · head 128 · window 4,096layer 34: Gated MLP: 36864layer 35: GQA: 32 query / 16 KV heads · head 128layer 35: Gated MLP: 36864layer 36: GQA: 32 query / 16 KV heads · head 128 · window 4,096layer 36: Gated MLP: 36864layer 37: GQA: 32 query / 16 KV heads · head 128layer 37: Gated MLP: 36864layer 38: GQA: 32 query / 16 KV heads · head 128 · window 4,096layer 38: Gated MLP: 36864layer 39: GQA: 32 query / 16 KV heads · head 128layer 39: Gated MLP: 36864layer 40: GQA: 32 query / 16 KV heads · head 128 · window 4,096layer 40: Gated MLP: 36864layer 41: GQA: 32 query / 16 KV heads · head 128layer 41: Gated MLP: 36864layer 42: GQA: 32 query / 16 KV heads · head 128 · window 4,096layer 42: Gated MLP: 36864layer 43: GQA: 32 query / 16 KV heads · head 128layer 43: Gated MLP: 36864layer 44: GQA: 32 query / 16 KV heads · head 128 · window 4,096layer 44: Gated MLP: 36864layer 45: GQA: 32 query / 16 KV heads · head 128layer 45: Gated MLP: 3686402345× 23normGQA: 32 query / 16 KV heads · head 128 · window 4,096norm+normGated MLP: 36864norm+× 23normGQA: 32 query / 16 KV heads · head 128norm+normGated MLP: 36864norm+sliding windowfull attentiondense FFN

Modelled costs

From the cost model, batch size 1. Totals the lab states are in the table above; differences come from rounding, from what a lab counts, or from parts the model does not describe (listed on the about page).

Parameters (modelled)27.2B
Active per token (modelled)27.2B
Without embeddings and output head26B total, 26B active
Published weights (Hugging Face count)27.2B
KV cache per token, BF16 (layers that grow with context)184 KiB
KV cache + state at 8K tokens, BF162.16 GiB
Decode FLOPs per token at 4K context57.5 GFLOP
Prefill FLOPs for a 4K prompt220 TFLOP

KV cache against context

Gemma 2 27B: KV cache bytes against context length101001,000980 KiB9.5 MiB95 MiB950 MiBcontext (tokens)KV cache + state (BF16)Gemma 2 27B

Compare with other models →

Every architecture field

FieldValueSource
d_model4,608labgoogle-deepmind/gemma _gemma.py (pinned)_gemma.py Gemma2_27B: embed_dim
vocab256,128labgoogle-deepmind/gemma _gemma.py (pinned)_gemma.py Gemma2_27B: num_embed
tied_embeddingstruelabgoogle-deepmind/gemma _gemma.py (pinned)_gemma.py Gemma2_27B: embedder shared with the output layer
mixers.full.typeattncodemodelling codeattention block
mixers.full.heads32labgoogle-deepmind/gemma _gemma.py (pinned)_gemma.py Gemma2_27B: num_heads
mixers.full.kv_heads16labgoogle-deepmind/gemma _gemma.py (pinned)_gemma.py Gemma2_27B: num_kv_heads
mixers.full.head_dim128labgoogle-deepmind/gemma _gemma.py (pinned)_gemma.py Gemma2_27B: head_dim
mixers.sliding.typeattncodemodelling codeattention block
mixers.sliding.heads32labgoogle-deepmind/gemma _gemma.py (pinned)_gemma.py Gemma2_27B: num_heads
mixers.sliding.kv_heads16labgoogle-deepmind/gemma _gemma.py (pinned)_gemma.py Gemma2_27B: num_kv_heads
mixers.sliding.head_dim128labgoogle-deepmind/gemma _gemma.py (pinned)_gemma.py Gemma2_27B: head_dim
mixers.sliding.window4,096labgoogle-deepmind/gemma _gemma.py (pinned)_gemma.py Gemma2_27B: sliding_window_size
ffns.dense.typedensecodemodelling codeMLP block
ffns.dense.d_ff36,864labgoogle-deepmind/gemma _gemma.py (pinned)_gemma.py Gemma2_27B: hidden_dim
ffns.dense.gatedtruecodemodelling codeMLP: gated (SwiGLU/GeGLU)
layout1× sliding/dense · 1× full/dense · 1× sliding/dense · 1× full/dense · 1× sliding/dense · 1× full/dense · 1× sliding/dense · 1× full/dense · 1× sliding/dense · 1× full/dense · 1× sliding/dense · 1× full/dense · 1× sliding/dense · 1× full/dense · 1× sliding/dense · 1× full/dense · 1× sliding/dense · 1× full/dense · 1× sliding/dense · 1× full/dense · 1× sliding/dense · 1× full/dense · 1× sliding/dense · 1× full/dense · 1× sliding/dense · 1× full/dense · 1× sliding/dense · 1× full/dense · 1× sliding/dense · 1× full/dense · 1× sliding/dense · 1× full/dense · 1× sliding/dense · 1× full/dense · 1× sliding/dense · 1× full/dense · 1× sliding/dense · 1× full/dense · 1× sliding/dense · 1× full/dense · 1× sliding/dense · 1× full/dense · 1× sliding/dense · 1× full/dense · 1× sliding/dense · 1× full/denselabgoogle-deepmind/gemma _gemma.py (pinned)_gemma.py Gemma2_27B: attention_types (local sliding / global pattern)
norms_per_layer4codemodelling codeGemma 2/3: sandwich norms (pre and post around attention and MLP)

Sources