Llama 4 Maverick
Meta · Llama 4 · open weights · multimodal (text stack modelled)
- GQA
- Chunked attention
- RoPE
- NoPE layers
- Pre-norm
- MoE
- Shared expert
Facts and where they come from
| Released | 2025-04 | config.jsonconfig.jsonHugging Face repository creation date (api.createdAt) |
|---|---|---|
| Licence | other | config.jsonconfig.jsonREADME metadata: license |
| Total parameters | 400B | labmodel cardREADME table: 400B (Total) |
| Active parameters | 17B | labmodel cardREADME table: 17B (Activated) |
| Context length | 1,000,000 tokens | labmodel cardREADME table: context length 1M |
| Experts | 128 | labmodel cardREADME: Llama 4 Maverick (17Bx128E) |
| Norm placement | pre | codemodelling codetransformers 5.18.0 / repo modelling code for llama4_text: input_layernorm before attention, post_attention_layernorm before the MLP |
| Norm type | RMSNorm | codemodelling codetransformers 5.18.0 / repo modelling code for llama4_text: input_layernorm before attention, post_attention_layernorm before the MLP |
| QK-norm | no | config.jsonconfig.jsontext_config.use_qk_norm |
| Positional encoding | RoPE; every 4th layer has no RoPE (global attention); the others use RoPE within their attention chunk | codemodelling codetransformers 5.18.0 llama4: no_rope_layer_interval default 4; RoPE layers are chunked_attention |
| Parallel attention and MLP | no | codemodelling codetransformers 5.18.0 / repo modelling code for llama4_text: input_layernorm before attention, post_attention_layernorm before the MLP |
Architecture, drawn from the data
36× GQA 40q/8kv, chunks of 8192 + 12× GQA 40q/8kv. Each column is one layer: its token mixer above, its feed-forward block below. Paler columns reuse another layer’s keys and values.
Modelled costs
From the cost model, batch size 1. Totals the lab states are in the table above; differences come from rounding, from what a lab counts, or from parts the model does not describe (listed on the about page).
| Parameters (modelled) | 401B |
|---|---|
| Active per token (modelled) | 17.2B |
| Without embeddings and output head | 399B total, 15.1B active |
| Published weights (Hugging Face count) | 402B |
| KV cache per token, BF16 (layers that grow with context) | 48 KiB |
| KV cache + state at 1,000,000 tokens, BF16 | 46.9 GiB |
| Decode FLOPs per token at 4K context | 36.3 GFLOP |
| Prefill FLOPs for a 4K prompt | 132 TFLOP |
KV cache against context
Every architecture field
| Field | Value | Source |
|---|---|---|
| d_model | 5,120 | config.jsonconfig.jsontext_config.hidden_size |
| vocab | 202,048 | config.jsonconfig.jsontext_config.vocab_size |
| tied_embeddings | false | codemodelling codetransformers 5.18.0: PretrainedConfig.tie_word_embeddings defaultnot in config |
| mixers.full.type | attn | codemodelling codeattention block |
| mixers.full.heads | 40 | config.jsonconfig.jsontext_config.num_attention_heads |
| mixers.full.kv_heads | 8 | config.jsonconfig.jsontext_config.num_key_value_heads |
| mixers.full.head_dim | 128 | config.jsonconfig.jsontext_config.head_dim |
| mixers.chunked.type | attn | codemodelling codeattention block |
| mixers.chunked.heads | 40 | config.jsonconfig.jsontext_config.num_attention_heads |
| mixers.chunked.kv_heads | 8 | config.jsonconfig.jsontext_config.num_key_value_heads |
| mixers.chunked.head_dim | 128 | config.jsonconfig.jsontext_config.head_dim |
| mixers.chunked.chunk | 8,192 | config.jsonconfig.jsontext_config.attention_chunk_size |
| ffns.dense.type | dense | codemodelling codeMLP block |
| ffns.dense.d_ff | 16,384 | config.jsonconfig.jsontext_config.intermediate_size_mlp |
| ffns.dense.gated | true | codemodelling codeMLP: gated (SwiGLU/GeGLU) |
| ffns.moe.type | moe | codemodelling codeMoE block |
| ffns.moe.experts | 128 | config.jsonconfig.jsontext_config.num_local_experts |
| ffns.moe.active | 1 | config.jsonconfig.jsontext_config.num_experts_per_tok |
| ffns.moe.d_expert | 8,192 | config.jsonconfig.jsontext_config.intermediate_size |
| ffns.moe.gated | true | codemodelling codeexperts are gated MLPs |
| ffns.moe.shared | 1 | codemodelling codetransformers 5.18.0 llama4: Llama4TextMoe.shared_expert, one Llama4TextMLP of intermediate_size |
| ffns.moe.d_shared | 8,192 | config.jsonconfig.jsontext_config.intermediate_size |
| layout | 1× chunked/dense · 1× chunked/moe · 1× chunked/dense · 1× full/moe · 1× chunked/dense · 1× chunked/moe · 1× chunked/dense · 1× full/moe · 1× chunked/dense · 1× chunked/moe · 1× chunked/dense · 1× full/moe · 1× chunked/dense · 1× chunked/moe · 1× chunked/dense · 1× full/moe · 1× chunked/dense · 1× chunked/moe · 1× chunked/dense · 1× full/moe · 1× chunked/dense · 1× chunked/moe · 1× chunked/dense · 1× full/moe · 1× chunked/dense · 1× chunked/moe · 1× chunked/dense · 1× full/moe · 1× chunked/dense · 1× chunked/moe · 1× chunked/dense · 1× full/moe · 1× chunked/dense · 1× chunked/moe · 1× chunked/dense · 1× full/moe · 1× chunked/dense · 1× chunked/moe · 1× chunked/dense · 1× full/moe · 1× chunked/dense · 1× chunked/moe · 1× chunked/dense · 1× full/moe · 1× chunked/dense · 1× chunked/moe · 1× chunked/dense · 1× full/moe | codemodelling codetransformers 5.18.0 llama4: no_rope_layers default (layer i uses RoPE and chunked_attention unless (i + 1) % no_rope_layer_interval == 0; interval 4); moe_layers default range(interleave_moe_layer_step - 1, num_hidden_layers, interleave_moe_layer_step) with text_config.interleave_moe_layer_step = 2 |
Sources
- config.json @ 73d1471
- model card
- announcement
- modelling code · transformers 5.18.0 modelling code, or the model repository's own modelling file at the pinned revision
Listed in the LLM Architecture Gallery checklist as “Llama 4 Maverick (400B)” (name only; see about).