llm-architectures-explained

/models

DeepSeekMoE 16B

DeepSeek · DeepSeekMoE · open weights

Facts and where they come from

Released2024-01paperarXiv 2401.06066arXiv v1, January 2024
Licenceotherconfig.jsonconfig.jsonREADME metadata: license
Total parameters16BpaperarXiv 2401.06066abstract: we scale up DeepSeekMoE to 16B parameters
Active parametersnot disclosednot disclosed
Context length4K tokensconfig.jsonconfig.jsonmax_position_embeddings
Norm placementprecodemodelling codetransformers 5.18.0 / repo modelling code for deepseek: input_layernorm before attention, post_attention_layernorm before the MLP
Norm typeRMSNormcodemodelling codetransformers 5.18.0 / repo modelling code for deepseek: input_layernorm before attention, post_attention_layernorm before the MLP
QK-normnocodemodelling codeno q/k normalisation in the attention block
Positional encodingRoPEcodemodelling coderotary on the full head (default)
Parallel attention and MLPnocodemodelling codetransformers 5.18.0 / repo modelling code for deepseek: input_layernorm before attention, post_attention_layernorm before the MLP

Architecture, drawn from the data

MHA 16q/16kv. Each column is one layer: its token mixer above, its feed-forward block below. Paler columns reuse another layer’s keys and values.

DeepSeekMoE 16B: layer stack and blockslayers (28)mixer / FFNlayer 0: MHA: 16 query / 16 KV heads · head 128layer 0: Gated MLP: 10944layer 1: MHA: 16 query / 16 KV heads · head 128layer 1: MoE: 64 experts, 6 active · expert 1408 · 2 sharedlayer 2: MHA: 16 query / 16 KV heads · head 128layer 2: MoE: 64 experts, 6 active · expert 1408 · 2 sharedlayer 3: MHA: 16 query / 16 KV heads · head 128layer 3: MoE: 64 experts, 6 active · expert 1408 · 2 sharedlayer 4: MHA: 16 query / 16 KV heads · head 128layer 4: MoE: 64 experts, 6 active · expert 1408 · 2 sharedlayer 5: MHA: 16 query / 16 KV heads · head 128layer 5: MoE: 64 experts, 6 active · expert 1408 · 2 sharedlayer 6: MHA: 16 query / 16 KV heads · head 128layer 6: MoE: 64 experts, 6 active · expert 1408 · 2 sharedlayer 7: MHA: 16 query / 16 KV heads · head 128layer 7: MoE: 64 experts, 6 active · expert 1408 · 2 sharedlayer 8: MHA: 16 query / 16 KV heads · head 128layer 8: MoE: 64 experts, 6 active · expert 1408 · 2 sharedlayer 9: MHA: 16 query / 16 KV heads · head 128layer 9: MoE: 64 experts, 6 active · expert 1408 · 2 sharedlayer 10: MHA: 16 query / 16 KV heads · head 128layer 10: MoE: 64 experts, 6 active · expert 1408 · 2 sharedlayer 11: MHA: 16 query / 16 KV heads · head 128layer 11: MoE: 64 experts, 6 active · expert 1408 · 2 sharedlayer 12: MHA: 16 query / 16 KV heads · head 128layer 12: MoE: 64 experts, 6 active · expert 1408 · 2 sharedlayer 13: MHA: 16 query / 16 KV heads · head 128layer 13: MoE: 64 experts, 6 active · expert 1408 · 2 sharedlayer 14: MHA: 16 query / 16 KV heads · head 128layer 14: MoE: 64 experts, 6 active · expert 1408 · 2 sharedlayer 15: MHA: 16 query / 16 KV heads · head 128layer 15: MoE: 64 experts, 6 active · expert 1408 · 2 sharedlayer 16: MHA: 16 query / 16 KV heads · head 128layer 16: MoE: 64 experts, 6 active · expert 1408 · 2 sharedlayer 17: MHA: 16 query / 16 KV heads · head 128layer 17: MoE: 64 experts, 6 active · expert 1408 · 2 sharedlayer 18: MHA: 16 query / 16 KV heads · head 128layer 18: MoE: 64 experts, 6 active · expert 1408 · 2 sharedlayer 19: MHA: 16 query / 16 KV heads · head 128layer 19: MoE: 64 experts, 6 active · expert 1408 · 2 sharedlayer 20: MHA: 16 query / 16 KV heads · head 128layer 20: MoE: 64 experts, 6 active · expert 1408 · 2 sharedlayer 21: MHA: 16 query / 16 KV heads · head 128layer 21: MoE: 64 experts, 6 active · expert 1408 · 2 sharedlayer 22: MHA: 16 query / 16 KV heads · head 128layer 22: MoE: 64 experts, 6 active · expert 1408 · 2 sharedlayer 23: MHA: 16 query / 16 KV heads · head 128layer 23: MoE: 64 experts, 6 active · expert 1408 · 2 sharedlayer 24: MHA: 16 query / 16 KV heads · head 128layer 24: MoE: 64 experts, 6 active · expert 1408 · 2 sharedlayer 25: MHA: 16 query / 16 KV heads · head 128layer 25: MoE: 64 experts, 6 active · expert 1408 · 2 sharedlayer 26: MHA: 16 query / 16 KV heads · head 128layer 26: MoE: 64 experts, 6 active · expert 1408 · 2 sharedlayer 27: MHA: 16 query / 16 KV heads · head 128layer 27: MoE: 64 experts, 6 active · expert 1408 · 2 shared01427× 27normMHA: 16 query / 16 KV heads · head 128+normMoE: 64 experts, 6 active · expert 1408 · 2 shared+× 1normMHA: 16 query / 16 KV heads · head 128+normGated MLP: 10944+full attentiondense FFNMoE FFN

Modelled costs

From the cost model, batch size 1. Totals the lab states are in the table above; differences come from rounding, from what a lab counts, or from parts the model does not describe (listed on the about page).

Parameters (modelled)16.4B
Active per token (modelled)2.83B
Without embeddings and output head16B total, 2.41B active
Published weights (Hugging Face count)16.4B
KV cache per token, BF16 (layers that grow with context)224 KiB
KV cache + state at 4K tokens, BF16896 MiB
Decode FLOPs per token at 4K context6.18 GFLOP
Prefill FLOPs for a 4K prompt21.7 TFLOP

KV cache against context

DeepSeekMoE 16B: KV cache bytes against context length101001,000980 KiB9.5 MiB95 MiB950 MiBcontext (tokens)KV cache + state (BF16)DeepSeekMoE 16B

Compare with other models →

Every architecture field

FieldValueSource
d_model2,048config.jsonconfig.jsonhidden_size
vocab102,400config.jsonconfig.jsonvocab_size
tied_embeddingsfalseconfig.jsonconfig.jsontie_word_embeddings
mixers.full.typeattncodemodelling codeattention block
mixers.full.heads16config.jsonconfig.jsonnum_attention_heads
mixers.full.kv_heads16config.jsonconfig.jsonnum_key_value_heads
mixers.full.head_dim128codemodelling codetransformers 5.18.0: head_dim = hidden_size / num_attention_heads
ffns.dense.typedensecodemodelling codeMLP block
ffns.dense.d_ff10,944config.jsonconfig.jsonintermediate_size
ffns.dense.gatedtruecodemodelling codeMLP: gated (SwiGLU/GeGLU)
ffns.moe.typemoecodemodelling codeMoE block
ffns.moe.experts64config.jsonconfig.jsonn_routed_experts
ffns.moe.active6config.jsonconfig.jsonnum_experts_per_tok
ffns.moe.d_expert1,408config.jsonconfig.jsonmoe_intermediate_size
ffns.moe.gatedtruecodemodelling codeexperts are gated MLPs
ffns.moe.shared2config.jsonconfig.jsonn_shared_experts
ffns.moe.d_shared1,408config.jsonconfig.jsonmoe_intermediate_size
layout1× full/dense · 27× full/moeconfig.jsonconfig.jsonnum_hidden_layers, first_k_dense_replace

Sources