llm-architectures-explained

/models

Mistral Small 4

Mistral AI · Mistral Small · open weights · multimodal (text stack modelled)

Facts and where they come from

Released2026-01config.jsonconfig.jsonHugging Face repository creation date (api.createdAt)
Licenceapache-2.0config.jsonconfig.jsonREADME metadata: license
Total parameters119Blabmodel cardREADME: 119B parameters, with 6.5B activated per token
Active parameters6.5Blabmodel cardREADME: 6.5B activated
Context length256K tokenslabmodel cardREADME: 256k context length
Norm placementprecodemodelling codetransformers 5.18.0 / repo modelling code for mistral4: input_layernorm before attention, post_attention_layernorm before the MLP
Norm typeRMSNormcodemodelling codetransformers 5.18.0 / repo modelling code for mistral4: input_layernorm before attention, post_attention_layernorm before the MLP
QK-normnocodemodelling codeno q/k normalisation in the attention block (MLA normalises its latent vectors, which is not QK-norm)
Positional encodingRoPE on 50% of each headconfig.jsonconfig.jsonqk_rope_head_dim / (qk_nope_head_dim + qk_rope_head_dim): decoupled RoPE
Parallel attention and MLPnocodemodelling codetransformers 5.18.0 / repo modelling code for mistral4: input_layernorm before attention, post_attention_layernorm before the MLP

Architecture, drawn from the data

MLA 32h, latent 256. Each column is one layer: its token mixer above, its feed-forward block below. Paler columns reuse another layer’s keys and values.

Mistral Small 4: layer stack and blockslayers (36)mixer / FFNlayer 0: MLA: 32 heads · KV latent 256 + RoPE 64 · Q latent 1024layer 0: MoE: 128 experts, 4 active · expert 2048 · 1 sharedlayer 1: MLA: 32 heads · KV latent 256 + RoPE 64 · Q latent 1024layer 1: MoE: 128 experts, 4 active · expert 2048 · 1 sharedlayer 2: MLA: 32 heads · KV latent 256 + RoPE 64 · Q latent 1024layer 2: MoE: 128 experts, 4 active · expert 2048 · 1 sharedlayer 3: MLA: 32 heads · KV latent 256 + RoPE 64 · Q latent 1024layer 3: MoE: 128 experts, 4 active · expert 2048 · 1 sharedlayer 4: MLA: 32 heads · KV latent 256 + RoPE 64 · Q latent 1024layer 4: MoE: 128 experts, 4 active · expert 2048 · 1 sharedlayer 5: MLA: 32 heads · KV latent 256 + RoPE 64 · Q latent 1024layer 5: MoE: 128 experts, 4 active · expert 2048 · 1 sharedlayer 6: MLA: 32 heads · KV latent 256 + RoPE 64 · Q latent 1024layer 6: MoE: 128 experts, 4 active · expert 2048 · 1 sharedlayer 7: MLA: 32 heads · KV latent 256 + RoPE 64 · Q latent 1024layer 7: MoE: 128 experts, 4 active · expert 2048 · 1 sharedlayer 8: MLA: 32 heads · KV latent 256 + RoPE 64 · Q latent 1024layer 8: MoE: 128 experts, 4 active · expert 2048 · 1 sharedlayer 9: MLA: 32 heads · KV latent 256 + RoPE 64 · Q latent 1024layer 9: MoE: 128 experts, 4 active · expert 2048 · 1 sharedlayer 10: MLA: 32 heads · KV latent 256 + RoPE 64 · Q latent 1024layer 10: MoE: 128 experts, 4 active · expert 2048 · 1 sharedlayer 11: MLA: 32 heads · KV latent 256 + RoPE 64 · Q latent 1024layer 11: MoE: 128 experts, 4 active · expert 2048 · 1 sharedlayer 12: MLA: 32 heads · KV latent 256 + RoPE 64 · Q latent 1024layer 12: MoE: 128 experts, 4 active · expert 2048 · 1 sharedlayer 13: MLA: 32 heads · KV latent 256 + RoPE 64 · Q latent 1024layer 13: MoE: 128 experts, 4 active · expert 2048 · 1 sharedlayer 14: MLA: 32 heads · KV latent 256 + RoPE 64 · Q latent 1024layer 14: MoE: 128 experts, 4 active · expert 2048 · 1 sharedlayer 15: MLA: 32 heads · KV latent 256 + RoPE 64 · Q latent 1024layer 15: MoE: 128 experts, 4 active · expert 2048 · 1 sharedlayer 16: MLA: 32 heads · KV latent 256 + RoPE 64 · Q latent 1024layer 16: MoE: 128 experts, 4 active · expert 2048 · 1 sharedlayer 17: MLA: 32 heads · KV latent 256 + RoPE 64 · Q latent 1024layer 17: MoE: 128 experts, 4 active · expert 2048 · 1 sharedlayer 18: MLA: 32 heads · KV latent 256 + RoPE 64 · Q latent 1024layer 18: MoE: 128 experts, 4 active · expert 2048 · 1 sharedlayer 19: MLA: 32 heads · KV latent 256 + RoPE 64 · Q latent 1024layer 19: MoE: 128 experts, 4 active · expert 2048 · 1 sharedlayer 20: MLA: 32 heads · KV latent 256 + RoPE 64 · Q latent 1024layer 20: MoE: 128 experts, 4 active · expert 2048 · 1 sharedlayer 21: MLA: 32 heads · KV latent 256 + RoPE 64 · Q latent 1024layer 21: MoE: 128 experts, 4 active · expert 2048 · 1 sharedlayer 22: MLA: 32 heads · KV latent 256 + RoPE 64 · Q latent 1024layer 22: MoE: 128 experts, 4 active · expert 2048 · 1 sharedlayer 23: MLA: 32 heads · KV latent 256 + RoPE 64 · Q latent 1024layer 23: MoE: 128 experts, 4 active · expert 2048 · 1 sharedlayer 24: MLA: 32 heads · KV latent 256 + RoPE 64 · Q latent 1024layer 24: MoE: 128 experts, 4 active · expert 2048 · 1 sharedlayer 25: MLA: 32 heads · KV latent 256 + RoPE 64 · Q latent 1024layer 25: MoE: 128 experts, 4 active · expert 2048 · 1 sharedlayer 26: MLA: 32 heads · KV latent 256 + RoPE 64 · Q latent 1024layer 26: MoE: 128 experts, 4 active · expert 2048 · 1 sharedlayer 27: MLA: 32 heads · KV latent 256 + RoPE 64 · Q latent 1024layer 27: MoE: 128 experts, 4 active · expert 2048 · 1 sharedlayer 28: MLA: 32 heads · KV latent 256 + RoPE 64 · Q latent 1024layer 28: MoE: 128 experts, 4 active · expert 2048 · 1 sharedlayer 29: MLA: 32 heads · KV latent 256 + RoPE 64 · Q latent 1024layer 29: MoE: 128 experts, 4 active · expert 2048 · 1 sharedlayer 30: MLA: 32 heads · KV latent 256 + RoPE 64 · Q latent 1024layer 30: MoE: 128 experts, 4 active · expert 2048 · 1 sharedlayer 31: MLA: 32 heads · KV latent 256 + RoPE 64 · Q latent 1024layer 31: MoE: 128 experts, 4 active · expert 2048 · 1 sharedlayer 32: MLA: 32 heads · KV latent 256 + RoPE 64 · Q latent 1024layer 32: MoE: 128 experts, 4 active · expert 2048 · 1 sharedlayer 33: MLA: 32 heads · KV latent 256 + RoPE 64 · Q latent 1024layer 33: MoE: 128 experts, 4 active · expert 2048 · 1 sharedlayer 34: MLA: 32 heads · KV latent 256 + RoPE 64 · Q latent 1024layer 34: MoE: 128 experts, 4 active · expert 2048 · 1 sharedlayer 35: MLA: 32 heads · KV latent 256 + RoPE 64 · Q latent 1024layer 35: MoE: 128 experts, 4 active · expert 2048 · 1 shared01835× 36normMLA: 32 heads · KV latent 256 + RoPE 64 · Q latent 1024+normMoE: 128 experts, 4 active · expert 2048 · 1 shared+MLAMoE FFN

Modelled costs

From the cost model, batch size 1. Totals the lab states are in the table above; differences come from rounding, from what a lab counts, or from parts the model does not describe (listed on the about page).

Parameters (modelled)119B
Active per token (modelled)6.63B
Without embeddings and output head118B total, 5.56B active
Published weights (Hugging Face count)119B
KV cache per token, BF16 (layers that grow with context)22.5 KiB
KV cache + state at 256K tokens, BF165.63 GiB
Decode FLOPs per token at 4K context14.6 GFLOP
Prefill FLOPs for a 4K prompt50.5 TFLOP

KV cache against context

Mistral Small 4: KV cache bytes against context length101001,00010,000100,00098 KiB980 KiB9.5 MiB95 MiB950 MiBcontext (tokens)KV cache + state (BF16)Mistral Small 4

Compare with other models →

Every architecture field

FieldValueSource
d_model4,096config.jsonconfig.jsontext_config.hidden_size
vocab131,072config.jsonconfig.jsontext_config.vocab_size
tied_embeddingsfalseconfig.jsonconfig.jsontext_config.tie_word_embeddings
mixers.mla.typemlacodemodelling codemulti-head latent attention
mixers.mla.heads32config.jsonconfig.jsontext_config.num_attention_heads
mixers.mla.q_lora_rank1,024config.jsonconfig.jsontext_config.q_lora_rank
mixers.mla.kv_lora_rank256config.jsonconfig.jsontext_config.kv_lora_rank
mixers.mla.qk_nope64config.jsonconfig.jsontext_config.qk_nope_head_dim
mixers.mla.qk_rope64config.jsonconfig.jsontext_config.qk_rope_head_dim
mixers.mla.v_head_dim128config.jsonconfig.jsontext_config.v_head_dim
ffns.dense.typedensecodemodelling codeMLP block
ffns.dense.d_ff12,288config.jsonconfig.jsontext_config.intermediate_size
ffns.dense.gatedtruecodemodelling codeMLP: gated (SwiGLU/GeGLU)
ffns.moe.typemoecodemodelling codeMoE block
ffns.moe.experts128config.jsonconfig.jsontext_config.n_routed_experts
ffns.moe.active4config.jsonconfig.jsontext_config.num_experts_per_tok
ffns.moe.d_expert2,048config.jsonconfig.jsontext_config.moe_intermediate_size
ffns.moe.gatedtruecodemodelling codeexperts are gated MLPs
ffns.moe.shared1config.jsonconfig.jsontext_config.n_shared_experts
ffns.moe.d_shared2,048config.jsonconfig.jsontext_config.moe_intermediate_size
layout36× mla/moeconfig.jsonconfig.jsontext_config.num_hidden_layers, first_k_dense_replace

Sources

Listed in the LLM Architecture Gallery checklist as “Mistral Small 4 (119B)” (name only; see about).