Mistral Large 3
Mistral AI · Mistral Large · open weights · multimodal (text stack modelled)
- MLA
- RoPE
- Partial RoPE
- MoE
- Shared expert
- Dense first layers
Facts and where they come from
| Released | 2025-11 | config.jsonconfig.jsonHugging Face repository creation date (api.createdAt) |
|---|---|---|
| Licence | apache-2.0 | config.jsonconfig.jsonREADME metadata: license |
| Total parameters | 673B | labmodel cardREADME: a granular MoE language model with 673B params and 39B active |
| Active parameters | 39B | labmodel cardREADME: 39B active (41B with the vision encoder) |
| Context length | 288K tokens | config.jsonconfig.jsonmax_position_embeddings |
| Norm placement | not disclosed | not disclosednot checked in the modelling code |
| Norm type | not disclosed | not disclosednot checked in the modelling code |
| QK-norm | no | codemodelling codeno q/k normalisation in the attention block (MLA normalises its latent vectors, which is not QK-norm) |
| Positional encoding | RoPE on 33.3% of each head | config.jsonconfig.jsonqk_rope_head_dim / (qk_nope_head_dim + qk_rope_head_dim): decoupled RoPE |
Architecture, drawn from the data
MLA 128h, latent 512. Each column is one layer: its token mixer above, its feed-forward block below. Paler columns reuse another layer’s keys and values.
Modelled costs
From the cost model, batch size 1. Totals the lab states are in the table above; differences come from rounding, from what a lab counts, or from parts the model does not describe (listed on the about page).
| Parameters (modelled) | 673B |
|---|---|
| Active per token (modelled) | 39.9B |
| Without embeddings and output head | 672B total, 38.1B active |
| KV cache per token, BF16 (layers that grow with context) | 68.6 KiB |
| KV cache + state at 288K tokens, BF16 | 19.3 GiB |
| Decode FLOPs per token at 4K context | 98.5 GFLOP |
| Prefill FLOPs for a 4K prompt | 354 TFLOP |
KV cache against context
Every architecture field
| Field | Value | Source |
|---|---|---|
| d_model | 7,168 | config.jsonconfig.jsondim |
| vocab | 131,072 | config.jsonconfig.jsonvocab_size |
| tied_embeddings | false | config.jsonconfig.jsontied_embeddings |
| mixers.mla.type | mla | codemodelling codemulti-head latent attention |
| mixers.mla.heads | 128 | config.jsonconfig.jsonn_heads |
| mixers.mla.q_lora_rank | 1,536 | config.jsonconfig.jsonq_lora_rank |
| mixers.mla.kv_lora_rank | 512 | config.jsonconfig.jsonkv_lora_rank |
| mixers.mla.qk_nope | 128 | config.jsonconfig.jsonqk_nope_head_dim |
| mixers.mla.qk_rope | 64 | config.jsonconfig.jsonqk_rope_head_dim |
| mixers.mla.v_head_dim | 128 | config.jsonconfig.jsonv_head_dim |
| ffns.dense.type | dense | codemodelling codeMLP |
| ffns.dense.d_ff | 16,384 | config.jsonconfig.jsonhidden_dim |
| ffns.dense.gated | true | codemodelling codeSwiGLU |
| ffns.moe.type | moe | codemodelling codeMoE |
| ffns.moe.experts | 128 | config.jsonconfig.jsonmoe.num_experts |
| ffns.moe.active | 4 | config.jsonconfig.jsonmoe.num_experts_per_tok |
| ffns.moe.d_expert | 4,096 | config.jsonconfig.jsonmoe.expert_hidden_dim |
| ffns.moe.shared | 1 | config.jsonconfig.jsonmoe.num_shared_experts |
| ffns.moe.d_shared | 4,096 | config.jsonconfig.jsonmoe.expert_hidden_dim |
| ffns.moe.gated | true | codemodelling codeSwiGLU experts |
| layout | 3× mla/dense · 58× mla/moe | config.jsonconfig.jsonn_layers, moe.first_k_dense_replace |
Sources
- config.json @ 383ffea
- model card
- modelling code · transformers 5.18.0 modelling code, or the model repository's own modelling file at the pinned revision
Listed in the LLM Architecture Gallery checklist as “Mistral Large 3 (673B)” (name only; see about).