MiniMax-M3
MiniMax · MiniMax M3 · open weights · multimodal (text stack modelled)
- GQA
- Sparse attention
- RoPE
- Partial RoPE
- QK-norm
- MoE
- Shared expert
- Dense first layers
- MTP
Facts and where they come from
| Released | 2026-06 | config.jsonconfig.jsonHugging Face repository creation date (api.createdAt) |
|---|---|---|
| Licence | other | config.jsonconfig.jsonREADME metadata: license |
| Total parameters | 428B | labmodel cardREADME: ~428B parameters and ~23B activated |
| Active parameters | 23B | labmodel cardREADME: ~23B activated |
| Context length | 1M tokens | labmodel cardREADME: 1M context |
| Norm placement | not disclosed | not disclosednot checked in the modelling code |
| Norm type | not disclosed | not disclosednot checked in the modelling code |
| QK-norm | yes | config.jsonconfig.jsontext_config.use_qk_norm |
| Positional encoding | RoPE on 50% of each head | config.jsonconfig.jsonrotary_dim / head_dim |
Architecture, drawn from the data
3× GQA 64q/4kv + 57× GQA 64q/4kv, top-2176. Each column is one layer: its token mixer above, its feed-forward block below. Paler columns reuse another layer’s keys and values.
Modelled costs
From the cost model, batch size 1. Totals the lab states are in the table above; differences come from rounding, from what a lab counts, or from parts the model does not describe (listed on the about page).
| Parameters (modelled) | 426B |
|---|---|
| Active per token (modelled) | 26B |
| Without embeddings and output head | 424B total, 23.5B active |
| Multi-token-prediction layers (extra) | 52.4B |
| Published weights (Hugging Face count) | 427B |
| KV cache per token, BF16 (layers that grow with context) | 134 KiB |
| KV cache + state at 1M tokens, BF16 | 134 GiB |
| Decode FLOPs per token at 4K context | 54.2 GFLOP |
| Prefill FLOPs for a 4K prompt | 206 TFLOP |
KV cache against context
Every architecture field
| Field | Value | Source |
|---|---|---|
| d_model | 6,144 | config.jsonconfig.jsontext_config.hidden_size |
| vocab | 200,064 | config.jsonconfig.jsontext_config.vocab_size |
| tied_embeddings | false | config.jsonconfig.jsontext_config.tie_word_embeddings |
| mixers.full.type | attn | codemodelling codeattention block |
| mixers.full.heads | 64 | config.jsonconfig.jsontext_config.num_attention_heads |
| mixers.full.kv_heads | 4 | config.jsonconfig.jsontext_config.num_key_value_heads |
| mixers.full.head_dim | 128 | config.jsonconfig.jsontext_config.head_dim |
| mixers.full.qk_norm | true | config.jsonconfig.jsontext_config.use_qk_norm |
| mixers.sparse.type | attn | codemodelling codeattention block |
| mixers.sparse.heads | 64 | config.jsonconfig.jsontext_config.num_attention_heads |
| mixers.sparse.kv_heads | 4 | config.jsonconfig.jsontext_config.num_key_value_heads |
| mixers.sparse.head_dim | 128 | config.jsonconfig.jsontext_config.head_dim |
| mixers.sparse.qk_norm | true | config.jsonconfig.jsontext_config.use_qk_norm |
| mixers.sparse.indexer.heads | 4 | config.jsonconfig.jsontext_config.sparse_attention_config.sparse_num_index_heads |
| mixers.sparse.indexer.head_dim | 128 | config.jsonconfig.jsontext_config.sparse_attention_config.sparse_index_dim |
| mixers.sparse.indexer.topk | 2,176 | codemodelling code(sparse_topk_blocks + sparse_local_block) x sparse_block_size tokens read per query (transformers 5.18.0 minimax_m3_vl lightning indexer) |
| ffns.dense.type | dense | codemodelling codeMLP block |
| ffns.dense.d_ff | 12,288 | config.jsonconfig.jsontext_config.dense_intermediate_size |
| ffns.dense.gated | true | codemodelling codeMLP: gated (SwiGLU/GeGLU) |
| ffns.moe.type | moe | codemodelling codeMoE block |
| ffns.moe.experts | 128 | config.jsonconfig.jsontext_config.num_local_experts |
| ffns.moe.active | 4 | config.jsonconfig.jsontext_config.num_experts_per_tok |
| ffns.moe.d_expert | 3,072 | config.jsonconfig.jsontext_config.intermediate_size |
| ffns.moe.gated | true | codemodelling codeexperts are gated MLPs |
| ffns.moe.shared | 1 | config.jsonconfig.jsontext_config.n_shared_experts |
| ffns.moe.d_shared | 3,072 | config.jsonconfig.jsontext_config.shared_intermediate_size |
| layout | 3× full/dense · 57× sparse/moe | config.jsonconfig.jsontext_config.moe_layer_freq, sparse_attention_config.sparse_attention_freq |
| mtp_layers | 7 | config.jsonconfig.jsontext_config.num_mtp_modules |
Sources
- config.json @ f0e1c1e
- model card
- arXiv 2606.13392
- modelling code · transformers 5.18.0 modelling code, or the model repository's own modelling file at the pinned revision
Listed in the LLM Architecture Gallery checklist as “MiniMax M3 (428B)” (name only; see about).