llm-architectures-explained

/models

Mixtral 8x7B

Mistral AI · Mixtral · open weights

Facts and where they come from

Released2023-12labmodel cardHF repository created December 2023
Licenceapache-2.0config.jsonconfig.jsonREADME metadata: license
Total parameters47BpaperarXiv 2401.04088abstract: 47B parameters, 13B active
Active parameters13BpaperarXiv 2401.04088abstract: 13B active parameters
Context length32K tokensconfig.jsonconfig.jsonmax_position_embeddings
Norm placementprecodemodelling codetransformers 5.18.0 / repo modelling code for mixtral: input_layernorm before attention, post_attention_layernorm before the MLP
Norm typeRMSNormcodemodelling codetransformers 5.18.0 / repo modelling code for mixtral: input_layernorm before attention, post_attention_layernorm before the MLP
QK-normnocodemodelling codeno q/k normalisation in the attention block
Positional encodingRoPEcodemodelling coderotary on the full head (default)
Parallel attention and MLPnocodemodelling codetransformers 5.18.0 / repo modelling code for mixtral: input_layernorm before attention, post_attention_layernorm before the MLP

Architecture, drawn from the data

GQA 32q/8kv. Each column is one layer: its token mixer above, its feed-forward block below. Paler columns reuse another layer’s keys and values.

Mixtral 8x7B: layer stack and blockslayers (32)mixer / FFNlayer 0: GQA: 32 query / 8 KV heads · head 128layer 0: MoE: 8 experts, 2 active · expert 14336layer 1: GQA: 32 query / 8 KV heads · head 128layer 1: MoE: 8 experts, 2 active · expert 14336layer 2: GQA: 32 query / 8 KV heads · head 128layer 2: MoE: 8 experts, 2 active · expert 14336layer 3: GQA: 32 query / 8 KV heads · head 128layer 3: MoE: 8 experts, 2 active · expert 14336layer 4: GQA: 32 query / 8 KV heads · head 128layer 4: MoE: 8 experts, 2 active · expert 14336layer 5: GQA: 32 query / 8 KV heads · head 128layer 5: MoE: 8 experts, 2 active · expert 14336layer 6: GQA: 32 query / 8 KV heads · head 128layer 6: MoE: 8 experts, 2 active · expert 14336layer 7: GQA: 32 query / 8 KV heads · head 128layer 7: MoE: 8 experts, 2 active · expert 14336layer 8: GQA: 32 query / 8 KV heads · head 128layer 8: MoE: 8 experts, 2 active · expert 14336layer 9: GQA: 32 query / 8 KV heads · head 128layer 9: MoE: 8 experts, 2 active · expert 14336layer 10: GQA: 32 query / 8 KV heads · head 128layer 10: MoE: 8 experts, 2 active · expert 14336layer 11: GQA: 32 query / 8 KV heads · head 128layer 11: MoE: 8 experts, 2 active · expert 14336layer 12: GQA: 32 query / 8 KV heads · head 128layer 12: MoE: 8 experts, 2 active · expert 14336layer 13: GQA: 32 query / 8 KV heads · head 128layer 13: MoE: 8 experts, 2 active · expert 14336layer 14: GQA: 32 query / 8 KV heads · head 128layer 14: MoE: 8 experts, 2 active · expert 14336layer 15: GQA: 32 query / 8 KV heads · head 128layer 15: MoE: 8 experts, 2 active · expert 14336layer 16: GQA: 32 query / 8 KV heads · head 128layer 16: MoE: 8 experts, 2 active · expert 14336layer 17: GQA: 32 query / 8 KV heads · head 128layer 17: MoE: 8 experts, 2 active · expert 14336layer 18: GQA: 32 query / 8 KV heads · head 128layer 18: MoE: 8 experts, 2 active · expert 14336layer 19: GQA: 32 query / 8 KV heads · head 128layer 19: MoE: 8 experts, 2 active · expert 14336layer 20: GQA: 32 query / 8 KV heads · head 128layer 20: MoE: 8 experts, 2 active · expert 14336layer 21: GQA: 32 query / 8 KV heads · head 128layer 21: MoE: 8 experts, 2 active · expert 14336layer 22: GQA: 32 query / 8 KV heads · head 128layer 22: MoE: 8 experts, 2 active · expert 14336layer 23: GQA: 32 query / 8 KV heads · head 128layer 23: MoE: 8 experts, 2 active · expert 14336layer 24: GQA: 32 query / 8 KV heads · head 128layer 24: MoE: 8 experts, 2 active · expert 14336layer 25: GQA: 32 query / 8 KV heads · head 128layer 25: MoE: 8 experts, 2 active · expert 14336layer 26: GQA: 32 query / 8 KV heads · head 128layer 26: MoE: 8 experts, 2 active · expert 14336layer 27: GQA: 32 query / 8 KV heads · head 128layer 27: MoE: 8 experts, 2 active · expert 14336layer 28: GQA: 32 query / 8 KV heads · head 128layer 28: MoE: 8 experts, 2 active · expert 14336layer 29: GQA: 32 query / 8 KV heads · head 128layer 29: MoE: 8 experts, 2 active · expert 14336layer 30: GQA: 32 query / 8 KV heads · head 128layer 30: MoE: 8 experts, 2 active · expert 14336layer 31: GQA: 32 query / 8 KV heads · head 128layer 31: MoE: 8 experts, 2 active · expert 1433601631× 32normGQA: 32 query / 8 KV heads · head 128+normMoE: 8 experts, 2 active · expert 14336+full attentionMoE FFN

Modelled costs

From the cost model, batch size 1. Totals the lab states are in the table above; differences come from rounding, from what a lab counts, or from parts the model does not describe (listed on the about page).

Parameters (modelled)46.7B
Active per token (modelled)12.9B
Without embeddings and output head46.4B total, 12.6B active
Published weights (Hugging Face count)46.7B
KV cache per token, BF16 (layers that grow with context)128 KiB
KV cache + state at 32K tokens, BF164 GiB
Decode FLOPs per token at 4K context27.6 GFLOP
Prefill FLOPs for a 4K prompt108 TFLOP

KV cache against context

Mixtral 8x7B: KV cache bytes against context length101001,00010,000980 KiB9.5 MiB95 MiB950 MiBcontext (tokens)KV cache + state (BF16)Mixtral 8x7B

Compare with other models →

Every architecture field

FieldValueSource
d_model4,096config.jsonconfig.jsonhidden_size
vocab32,000config.jsonconfig.jsonvocab_size
tied_embeddingsfalseconfig.jsonconfig.jsontie_word_embeddings
mixers.full.typeattncodemodelling codeattention block
mixers.full.heads32config.jsonconfig.jsonnum_attention_heads
mixers.full.kv_heads8config.jsonconfig.jsonnum_key_value_heads
mixers.full.head_dim128codemodelling codetransformers 5.18.0: head_dim = hidden_size / num_attention_heads
ffns.moe.typemoecodemodelling codeMoE block
ffns.moe.experts8config.jsonconfig.jsonnum_local_experts
ffns.moe.active2config.jsonconfig.jsonnum_experts_per_tok
ffns.moe.d_expert14,336config.jsonconfig.jsonintermediate_size
ffns.moe.gatedtruecodemodelling codeexperts are gated MLPs
layout32× full/moeconfig.jsonconfig.jsonnum_hidden_layers

Sources