llm-architectures-explained

/models

Grok-1

xAI · Grok · open weights

Facts and where they come from

Released2024-03codemodelling codeGitHub release of the open weights, March 2024
Licenceapache-2.0config.jsonconfig.jsonREADME metadata: license
Total parameters314Blabmodel cardREADME: Grok-1 ... 314B parameters
Active parametersnot disclosednot disclosed
Context length8K tokenslabxai-org/grok-1 run.py (pinned)run.py: sequence_len
Norm placementnot disclosednot disclosednot checked in the modelling code
Norm typenot disclosednot disclosednot checked in the modelling code
QK-normnocodemodelling codeno q/k normalisation in the attention block
Positional encodingRoPEcodemodelling coderotary on the full head (default)

Architecture, drawn from the data

GQA 48q/8kv. Each column is one layer: its token mixer above, its feed-forward block below. Paler columns reuse another layer’s keys and values.

Grok-1: layer stack and blockslayers (64)mixer / FFNlayer 0: GQA: 48 query / 8 KV heads · head 128layer 0: MoE: 8 experts, 2 active · expert 32768layer 1: GQA: 48 query / 8 KV heads · head 128layer 1: MoE: 8 experts, 2 active · expert 32768layer 2: GQA: 48 query / 8 KV heads · head 128layer 2: MoE: 8 experts, 2 active · expert 32768layer 3: GQA: 48 query / 8 KV heads · head 128layer 3: MoE: 8 experts, 2 active · expert 32768layer 4: GQA: 48 query / 8 KV heads · head 128layer 4: MoE: 8 experts, 2 active · expert 32768layer 5: GQA: 48 query / 8 KV heads · head 128layer 5: MoE: 8 experts, 2 active · expert 32768layer 6: GQA: 48 query / 8 KV heads · head 128layer 6: MoE: 8 experts, 2 active · expert 32768layer 7: GQA: 48 query / 8 KV heads · head 128layer 7: MoE: 8 experts, 2 active · expert 32768layer 8: GQA: 48 query / 8 KV heads · head 128layer 8: MoE: 8 experts, 2 active · expert 32768layer 9: GQA: 48 query / 8 KV heads · head 128layer 9: MoE: 8 experts, 2 active · expert 32768layer 10: GQA: 48 query / 8 KV heads · head 128layer 10: MoE: 8 experts, 2 active · expert 32768layer 11: GQA: 48 query / 8 KV heads · head 128layer 11: MoE: 8 experts, 2 active · expert 32768layer 12: GQA: 48 query / 8 KV heads · head 128layer 12: MoE: 8 experts, 2 active · expert 32768layer 13: GQA: 48 query / 8 KV heads · head 128layer 13: MoE: 8 experts, 2 active · expert 32768layer 14: GQA: 48 query / 8 KV heads · head 128layer 14: MoE: 8 experts, 2 active · expert 32768layer 15: GQA: 48 query / 8 KV heads · head 128layer 15: MoE: 8 experts, 2 active · expert 32768layer 16: GQA: 48 query / 8 KV heads · head 128layer 16: MoE: 8 experts, 2 active · expert 32768layer 17: GQA: 48 query / 8 KV heads · head 128layer 17: MoE: 8 experts, 2 active · expert 32768layer 18: GQA: 48 query / 8 KV heads · head 128layer 18: MoE: 8 experts, 2 active · expert 32768layer 19: GQA: 48 query / 8 KV heads · head 128layer 19: MoE: 8 experts, 2 active · expert 32768layer 20: GQA: 48 query / 8 KV heads · head 128layer 20: MoE: 8 experts, 2 active · expert 32768layer 21: GQA: 48 query / 8 KV heads · head 128layer 21: MoE: 8 experts, 2 active · expert 32768layer 22: GQA: 48 query / 8 KV heads · head 128layer 22: MoE: 8 experts, 2 active · expert 32768layer 23: GQA: 48 query / 8 KV heads · head 128layer 23: MoE: 8 experts, 2 active · expert 32768layer 24: GQA: 48 query / 8 KV heads · head 128layer 24: MoE: 8 experts, 2 active · expert 32768layer 25: GQA: 48 query / 8 KV heads · head 128layer 25: MoE: 8 experts, 2 active · expert 32768layer 26: GQA: 48 query / 8 KV heads · head 128layer 26: MoE: 8 experts, 2 active · expert 32768layer 27: GQA: 48 query / 8 KV heads · head 128layer 27: MoE: 8 experts, 2 active · expert 32768layer 28: GQA: 48 query / 8 KV heads · head 128layer 28: MoE: 8 experts, 2 active · expert 32768layer 29: GQA: 48 query / 8 KV heads · head 128layer 29: MoE: 8 experts, 2 active · expert 32768layer 30: GQA: 48 query / 8 KV heads · head 128layer 30: MoE: 8 experts, 2 active · expert 32768layer 31: GQA: 48 query / 8 KV heads · head 128layer 31: MoE: 8 experts, 2 active · expert 32768layer 32: GQA: 48 query / 8 KV heads · head 128layer 32: MoE: 8 experts, 2 active · expert 32768layer 33: GQA: 48 query / 8 KV heads · head 128layer 33: MoE: 8 experts, 2 active · expert 32768layer 34: GQA: 48 query / 8 KV heads · head 128layer 34: MoE: 8 experts, 2 active · expert 32768layer 35: GQA: 48 query / 8 KV heads · head 128layer 35: MoE: 8 experts, 2 active · expert 32768layer 36: GQA: 48 query / 8 KV heads · head 128layer 36: MoE: 8 experts, 2 active · expert 32768layer 37: GQA: 48 query / 8 KV heads · head 128layer 37: MoE: 8 experts, 2 active · expert 32768layer 38: GQA: 48 query / 8 KV heads · head 128layer 38: MoE: 8 experts, 2 active · expert 32768layer 39: GQA: 48 query / 8 KV heads · head 128layer 39: MoE: 8 experts, 2 active · expert 32768layer 40: GQA: 48 query / 8 KV heads · head 128layer 40: MoE: 8 experts, 2 active · expert 32768layer 41: GQA: 48 query / 8 KV heads · head 128layer 41: MoE: 8 experts, 2 active · expert 32768layer 42: GQA: 48 query / 8 KV heads · head 128layer 42: MoE: 8 experts, 2 active · expert 32768layer 43: GQA: 48 query / 8 KV heads · head 128layer 43: MoE: 8 experts, 2 active · expert 32768layer 44: GQA: 48 query / 8 KV heads · head 128layer 44: MoE: 8 experts, 2 active · expert 32768layer 45: GQA: 48 query / 8 KV heads · head 128layer 45: MoE: 8 experts, 2 active · expert 32768layer 46: GQA: 48 query / 8 KV heads · head 128layer 46: MoE: 8 experts, 2 active · expert 32768layer 47: GQA: 48 query / 8 KV heads · head 128layer 47: MoE: 8 experts, 2 active · expert 32768layer 48: GQA: 48 query / 8 KV heads · head 128layer 48: MoE: 8 experts, 2 active · expert 32768layer 49: GQA: 48 query / 8 KV heads · head 128layer 49: MoE: 8 experts, 2 active · expert 32768layer 50: GQA: 48 query / 8 KV heads · head 128layer 50: MoE: 8 experts, 2 active · expert 32768layer 51: GQA: 48 query / 8 KV heads · head 128layer 51: MoE: 8 experts, 2 active · expert 32768layer 52: GQA: 48 query / 8 KV heads · head 128layer 52: MoE: 8 experts, 2 active · expert 32768layer 53: GQA: 48 query / 8 KV heads · head 128layer 53: MoE: 8 experts, 2 active · expert 32768layer 54: GQA: 48 query / 8 KV heads · head 128layer 54: MoE: 8 experts, 2 active · expert 32768layer 55: GQA: 48 query / 8 KV heads · head 128layer 55: MoE: 8 experts, 2 active · expert 32768layer 56: GQA: 48 query / 8 KV heads · head 128layer 56: MoE: 8 experts, 2 active · expert 32768layer 57: GQA: 48 query / 8 KV heads · head 128layer 57: MoE: 8 experts, 2 active · expert 32768layer 58: GQA: 48 query / 8 KV heads · head 128layer 58: MoE: 8 experts, 2 active · expert 32768layer 59: GQA: 48 query / 8 KV heads · head 128layer 59: MoE: 8 experts, 2 active · expert 32768layer 60: GQA: 48 query / 8 KV heads · head 128layer 60: MoE: 8 experts, 2 active · expert 32768layer 61: GQA: 48 query / 8 KV heads · head 128layer 61: MoE: 8 experts, 2 active · expert 32768layer 62: GQA: 48 query / 8 KV heads · head 128layer 62: MoE: 8 experts, 2 active · expert 32768layer 63: GQA: 48 query / 8 KV heads · head 128layer 63: MoE: 8 experts, 2 active · expert 3276803263× 64GQA: 48 query / 8 KV heads · head 128+MoE: 8 experts, 2 active · expert 32768+full attentionMoE FFN

Modelled costs

From the cost model, batch size 1. Totals the lab states are in the table above; differences come from rounding, from what a lab counts, or from parts the model does not describe (listed on the about page).

Parameters (modelled)316B
Active per token (modelled)83.8B
Without embeddings and output head315B total, 83B active
KV cache per token, BF16 (layers that grow with context)256 KiB
KV cache + state at 8K tokens, BF162 GiB
Decode FLOPs per token at 4K context174 GFLOP
Prefill FLOPs for a 4K prompt693 TFLOP

KV cache against context

Grok-1: KV cache bytes against context length101001,000980 KiB9.5 MiB95 MiB950 MiBcontext (tokens)KV cache + state (BF16)Grok-1

Compare with other models →

Every architecture field

FieldValueSource
d_model6,144labxai-org/grok-1 run.py (pinned)run.py: emb_size = 48 * 128
vocab131,072labxai-org/grok-1 run.py (pinned)run.py: vocab_size = 128 * 1024
tied_embeddingstruelabxai-org/grok-1 run.py (pinned)data/transcribed/grok-1.json: tie_word_embeddings
mixers.full.typeattncodemodelling codeattention block
mixers.full.heads48labxai-org/grok-1 run.py (pinned)run.py: num_q_heads
mixers.full.kv_heads8labxai-org/grok-1 run.py (pinned)run.py: num_kv_heads
mixers.full.head_dim128labxai-org/grok-1 run.py (pinned)run.py: key_size
ffns.moe.typemoecodemodelling codeMoE block
ffns.moe.experts8labxai-org/grok-1 run.py (pinned)run.py: num_experts
ffns.moe.active2labxai-org/grok-1 run.py (pinned)run.py: num_selected_experts
ffns.moe.d_expert32,768labxai-org/grok-1 run.py (pinned)model.py ffn_size(emb_size, widening_factor=8): int(8 * 6144) * 2 // 3, rounded to a multiple of 8
ffns.moe.gatedtruecodemodelling codeexperts are gated MLPs
layout64× full/moelabxai-org/grok-1 run.py (pinned)run.py: num_layers

Sources