llm-architectures-explained

/models

MiniMax-Text-01

MiniMax · MiniMax-01 · open weights

Facts and where they come from

Released2025-01config.jsonconfig.jsonHugging Face repository creation date (api.createdAt)
Licencenot disclosednot disclosed
Total parameters456Blabmodel cardREADME: 456 billion total parameters, of which 45.9 billion are activated
Active parameters45.9Blabmodel cardREADME: 45.9 billion activated
Context length10000K tokensconfig.jsonconfig.jsonmax_position_embeddings
Norm placementprecodemodelling codetransformers 5.18.0 / repo modelling code for minimax_text_01: input_layernorm before attention, post_attention_layernorm before the MLP
Norm typeRMSNormcodemodelling codetransformers 5.18.0 / repo modelling code for minimax_text_01: input_layernorm before attention, post_attention_layernorm before the MLP
QK-normnocodemodelling codeno q/k normalisation in the attention block
Positional encodingRoPE on 50% of each headconfig.jsonconfig.jsonrotary_dim / head_dim
Parallel attention and MLPnocodemodelling codetransformers 5.18.0 / repo modelling code for minimax_text_01: input_layernorm before attention, post_attention_layernorm before the MLP

Architecture, drawn from the data

70× Linear attention + 10× GQA 64q/8kv. Each column is one layer: its token mixer above, its feed-forward block below. Paler columns reuse another layer’s keys and values.

MiniMax-Text-01: layer stack and blockslayers (80)mixer / FFNlayer 0: Linear attention: 64 heads × 128layer 0: MoE: 32 experts, 2 active · expert 9216layer 1: Linear attention: 64 heads × 128layer 1: MoE: 32 experts, 2 active · expert 9216layer 2: Linear attention: 64 heads × 128layer 2: MoE: 32 experts, 2 active · expert 9216layer 3: Linear attention: 64 heads × 128layer 3: MoE: 32 experts, 2 active · expert 9216layer 4: Linear attention: 64 heads × 128layer 4: MoE: 32 experts, 2 active · expert 9216layer 5: Linear attention: 64 heads × 128layer 5: MoE: 32 experts, 2 active · expert 9216layer 6: Linear attention: 64 heads × 128layer 6: MoE: 32 experts, 2 active · expert 9216layer 7: GQA: 64 query / 8 KV heads · head 128layer 7: MoE: 32 experts, 2 active · expert 9216layer 8: Linear attention: 64 heads × 128layer 8: MoE: 32 experts, 2 active · expert 9216layer 9: Linear attention: 64 heads × 128layer 9: MoE: 32 experts, 2 active · expert 9216layer 10: Linear attention: 64 heads × 128layer 10: MoE: 32 experts, 2 active · expert 9216layer 11: Linear attention: 64 heads × 128layer 11: MoE: 32 experts, 2 active · expert 9216layer 12: Linear attention: 64 heads × 128layer 12: MoE: 32 experts, 2 active · expert 9216layer 13: Linear attention: 64 heads × 128layer 13: MoE: 32 experts, 2 active · expert 9216layer 14: Linear attention: 64 heads × 128layer 14: MoE: 32 experts, 2 active · expert 9216layer 15: GQA: 64 query / 8 KV heads · head 128layer 15: MoE: 32 experts, 2 active · expert 9216layer 16: Linear attention: 64 heads × 128layer 16: MoE: 32 experts, 2 active · expert 9216layer 17: Linear attention: 64 heads × 128layer 17: MoE: 32 experts, 2 active · expert 9216layer 18: Linear attention: 64 heads × 128layer 18: MoE: 32 experts, 2 active · expert 9216layer 19: Linear attention: 64 heads × 128layer 19: MoE: 32 experts, 2 active · expert 9216layer 20: Linear attention: 64 heads × 128layer 20: MoE: 32 experts, 2 active · expert 9216layer 21: Linear attention: 64 heads × 128layer 21: MoE: 32 experts, 2 active · expert 9216layer 22: Linear attention: 64 heads × 128layer 22: MoE: 32 experts, 2 active · expert 9216layer 23: GQA: 64 query / 8 KV heads · head 128layer 23: MoE: 32 experts, 2 active · expert 9216layer 24: Linear attention: 64 heads × 128layer 24: MoE: 32 experts, 2 active · expert 9216layer 25: Linear attention: 64 heads × 128layer 25: MoE: 32 experts, 2 active · expert 9216layer 26: Linear attention: 64 heads × 128layer 26: MoE: 32 experts, 2 active · expert 9216layer 27: Linear attention: 64 heads × 128layer 27: MoE: 32 experts, 2 active · expert 9216layer 28: Linear attention: 64 heads × 128layer 28: MoE: 32 experts, 2 active · expert 9216layer 29: Linear attention: 64 heads × 128layer 29: MoE: 32 experts, 2 active · expert 9216layer 30: Linear attention: 64 heads × 128layer 30: MoE: 32 experts, 2 active · expert 9216layer 31: GQA: 64 query / 8 KV heads · head 128layer 31: MoE: 32 experts, 2 active · expert 9216layer 32: Linear attention: 64 heads × 128layer 32: MoE: 32 experts, 2 active · expert 9216layer 33: Linear attention: 64 heads × 128layer 33: MoE: 32 experts, 2 active · expert 9216layer 34: Linear attention: 64 heads × 128layer 34: MoE: 32 experts, 2 active · expert 9216layer 35: Linear attention: 64 heads × 128layer 35: MoE: 32 experts, 2 active · expert 9216layer 36: Linear attention: 64 heads × 128layer 36: MoE: 32 experts, 2 active · expert 9216layer 37: Linear attention: 64 heads × 128layer 37: MoE: 32 experts, 2 active · expert 9216layer 38: Linear attention: 64 heads × 128layer 38: MoE: 32 experts, 2 active · expert 9216layer 39: GQA: 64 query / 8 KV heads · head 128layer 39: MoE: 32 experts, 2 active · expert 9216layer 40: Linear attention: 64 heads × 128layer 40: MoE: 32 experts, 2 active · expert 9216layer 41: Linear attention: 64 heads × 128layer 41: MoE: 32 experts, 2 active · expert 9216layer 42: Linear attention: 64 heads × 128layer 42: MoE: 32 experts, 2 active · expert 9216layer 43: Linear attention: 64 heads × 128layer 43: MoE: 32 experts, 2 active · expert 9216layer 44: Linear attention: 64 heads × 128layer 44: MoE: 32 experts, 2 active · expert 9216layer 45: Linear attention: 64 heads × 128layer 45: MoE: 32 experts, 2 active · expert 9216layer 46: Linear attention: 64 heads × 128layer 46: MoE: 32 experts, 2 active · expert 9216layer 47: GQA: 64 query / 8 KV heads · head 128layer 47: MoE: 32 experts, 2 active · expert 9216layer 48: Linear attention: 64 heads × 128layer 48: MoE: 32 experts, 2 active · expert 9216layer 49: Linear attention: 64 heads × 128layer 49: MoE: 32 experts, 2 active · expert 9216layer 50: Linear attention: 64 heads × 128layer 50: MoE: 32 experts, 2 active · expert 9216layer 51: Linear attention: 64 heads × 128layer 51: MoE: 32 experts, 2 active · expert 9216layer 52: Linear attention: 64 heads × 128layer 52: MoE: 32 experts, 2 active · expert 9216layer 53: Linear attention: 64 heads × 128layer 53: MoE: 32 experts, 2 active · expert 9216layer 54: Linear attention: 64 heads × 128layer 54: MoE: 32 experts, 2 active · expert 9216layer 55: GQA: 64 query / 8 KV heads · head 128layer 55: MoE: 32 experts, 2 active · expert 9216layer 56: Linear attention: 64 heads × 128layer 56: MoE: 32 experts, 2 active · expert 9216layer 57: Linear attention: 64 heads × 128layer 57: MoE: 32 experts, 2 active · expert 9216layer 58: Linear attention: 64 heads × 128layer 58: MoE: 32 experts, 2 active · expert 9216layer 59: Linear attention: 64 heads × 128layer 59: MoE: 32 experts, 2 active · expert 9216layer 60: Linear attention: 64 heads × 128layer 60: MoE: 32 experts, 2 active · expert 9216layer 61: Linear attention: 64 heads × 128layer 61: MoE: 32 experts, 2 active · expert 9216layer 62: Linear attention: 64 heads × 128layer 62: MoE: 32 experts, 2 active · expert 9216layer 63: GQA: 64 query / 8 KV heads · head 128layer 63: MoE: 32 experts, 2 active · expert 9216layer 64: Linear attention: 64 heads × 128layer 64: MoE: 32 experts, 2 active · expert 9216layer 65: Linear attention: 64 heads × 128layer 65: MoE: 32 experts, 2 active · expert 9216layer 66: Linear attention: 64 heads × 128layer 66: MoE: 32 experts, 2 active · expert 9216layer 67: Linear attention: 64 heads × 128layer 67: MoE: 32 experts, 2 active · expert 9216layer 68: Linear attention: 64 heads × 128layer 68: MoE: 32 experts, 2 active · expert 9216layer 69: Linear attention: 64 heads × 128layer 69: MoE: 32 experts, 2 active · expert 9216layer 70: Linear attention: 64 heads × 128layer 70: MoE: 32 experts, 2 active · expert 9216layer 71: GQA: 64 query / 8 KV heads · head 128layer 71: MoE: 32 experts, 2 active · expert 9216layer 72: Linear attention: 64 heads × 128layer 72: MoE: 32 experts, 2 active · expert 9216layer 73: Linear attention: 64 heads × 128layer 73: MoE: 32 experts, 2 active · expert 9216layer 74: Linear attention: 64 heads × 128layer 74: MoE: 32 experts, 2 active · expert 9216layer 75: Linear attention: 64 heads × 128layer 75: MoE: 32 experts, 2 active · expert 9216layer 76: Linear attention: 64 heads × 128layer 76: MoE: 32 experts, 2 active · expert 9216layer 77: Linear attention: 64 heads × 128layer 77: MoE: 32 experts, 2 active · expert 9216layer 78: Linear attention: 64 heads × 128layer 78: MoE: 32 experts, 2 active · expert 9216layer 79: GQA: 64 query / 8 KV heads · head 128layer 79: MoE: 32 experts, 2 active · expert 921604079× 70normLinear attention: 64 heads × 128+normMoE: 32 experts, 2 active · expert 9216+× 10normGQA: 64 query / 8 KV heads · head 128+normMoE: 32 experts, 2 active · expert 9216+linear attentionfull attentionMoE FFN

Modelled costs

From the cost model, batch size 1. Totals the lab states are in the table above; differences come from rounding, from what a lab counts, or from parts the model does not describe (listed on the about page).

Parameters (modelled)456B
Active per token (modelled)48.4B
Without embeddings and output head454B total, 45.9B active
Published weights (Hugging Face count)456B
KV cache per token, BF16 (layers that grow with context)40 KiB
KV cache + state at 10000K tokens, BF16391 GiB
Decode FLOPs per token at 4K context96 GFLOP
Prefill FLOPs for a 4K prompt380 TFLOP

KV cache against context

MiniMax-Text-01: KV cache bytes against context length101001,00010,000100,0001,000,000950 MiB9.3 GiBcontext (tokens)KV cache + state (BF16)MiniMax-Text-01

Compare with other models →

Every architecture field

FieldValueSource
d_model6,144config.jsonconfig.jsonhidden_size
vocab200,064config.jsonconfig.jsonvocab_size
tied_embeddingsfalseconfig.jsonconfig.jsontie_word_embeddings
mixers.full.typeattncodemodelling codeattention block
mixers.full.heads64config.jsonconfig.jsonnum_attention_heads
mixers.full.kv_heads8config.jsonconfig.jsonnum_key_value_heads
mixers.full.head_dim128config.jsonconfig.jsonhead_dim
mixers.linear.typelinearcodemodelling codeLightning attention (linear)
mixers.linear.heads64config.jsonconfig.jsonnum_attention_heads
mixers.linear.head_dim128config.jsonconfig.jsonhead_dim
ffns.moe.typemoecodemodelling codeMoE block
ffns.moe.experts32config.jsonconfig.jsonnum_local_experts
ffns.moe.active2config.jsonconfig.jsonnum_experts_per_tok
ffns.moe.d_expert9,216config.jsonconfig.jsonintermediate_size
ffns.moe.gatedtruecodemodelling codeexperts are gated MLPs
layout7× linear/moe · 1× full/moe · 7× linear/moe · 1× full/moe · 7× linear/moe · 1× full/moe · 7× linear/moe · 1× full/moe · 7× linear/moe · 1× full/moe · 7× linear/moe · 1× full/moe · 7× linear/moe · 1× full/moe · 7× linear/moe · 1× full/moe · 7× linear/moe · 1× full/moe · 7× linear/moe · 1× full/moeconfig.jsonconfig.jsonattn_type_list

Sources