llm-architectures-explained

/models

DeepSeek-V2

DeepSeek · DeepSeek V2 · open weights

Facts and where they come from

Released2024-04config.jsonconfig.jsonHugging Face repository creation date (api.createdAt)
Licenceotherconfig.jsonconfig.jsonREADME metadata: license
Total parameters236Blabmodel cardREADME: 236B total parameters, of which 21B are activated
Active parameters21Blabmodel cardREADME: 21B activated
Context length160K tokensconfig.jsonconfig.jsonmax_position_embeddings
Norm placementprecodemodelling codetransformers 5.18.0 / repo modelling code for deepseek_v2: input_layernorm before attention, post_attention_layernorm before the MLP
Norm typeRMSNormcodemodelling codetransformers 5.18.0 / repo modelling code for deepseek_v2: input_layernorm before attention, post_attention_layernorm before the MLP
QK-normnocodemodelling codeno q/k normalisation in the attention block (MLA normalises its latent vectors, which is not QK-norm)
Positional encodingRoPE on 33.3% of each headconfig.jsonconfig.jsonqk_rope_head_dim / (qk_nope_head_dim + qk_rope_head_dim): decoupled RoPE
Parallel attention and MLPnocodemodelling codetransformers 5.18.0 / repo modelling code for deepseek_v2: input_layernorm before attention, post_attention_layernorm before the MLP

Architecture, drawn from the data

MLA 128h, latent 512. Each column is one layer: its token mixer above, its feed-forward block below. Paler columns reuse another layer’s keys and values.

DeepSeek-V2: layer stack and blockslayers (60)mixer / FFNlayer 0: MLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 0: Gated MLP: 12288layer 1: MLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 1: MoE: 160 experts, 6 active · expert 1536 · 2 sharedlayer 2: MLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 2: MoE: 160 experts, 6 active · expert 1536 · 2 sharedlayer 3: MLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 3: MoE: 160 experts, 6 active · expert 1536 · 2 sharedlayer 4: MLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 4: MoE: 160 experts, 6 active · expert 1536 · 2 sharedlayer 5: MLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 5: MoE: 160 experts, 6 active · expert 1536 · 2 sharedlayer 6: MLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 6: MoE: 160 experts, 6 active · expert 1536 · 2 sharedlayer 7: MLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 7: MoE: 160 experts, 6 active · expert 1536 · 2 sharedlayer 8: MLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 8: MoE: 160 experts, 6 active · expert 1536 · 2 sharedlayer 9: MLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 9: MoE: 160 experts, 6 active · expert 1536 · 2 sharedlayer 10: MLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 10: MoE: 160 experts, 6 active · expert 1536 · 2 sharedlayer 11: MLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 11: MoE: 160 experts, 6 active · expert 1536 · 2 sharedlayer 12: MLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 12: MoE: 160 experts, 6 active · expert 1536 · 2 sharedlayer 13: MLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 13: MoE: 160 experts, 6 active · expert 1536 · 2 sharedlayer 14: MLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 14: MoE: 160 experts, 6 active · expert 1536 · 2 sharedlayer 15: MLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 15: MoE: 160 experts, 6 active · expert 1536 · 2 sharedlayer 16: MLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 16: MoE: 160 experts, 6 active · expert 1536 · 2 sharedlayer 17: MLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 17: MoE: 160 experts, 6 active · expert 1536 · 2 sharedlayer 18: MLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 18: MoE: 160 experts, 6 active · expert 1536 · 2 sharedlayer 19: MLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 19: MoE: 160 experts, 6 active · expert 1536 · 2 sharedlayer 20: MLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 20: MoE: 160 experts, 6 active · expert 1536 · 2 sharedlayer 21: MLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 21: MoE: 160 experts, 6 active · expert 1536 · 2 sharedlayer 22: MLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 22: MoE: 160 experts, 6 active · expert 1536 · 2 sharedlayer 23: MLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 23: MoE: 160 experts, 6 active · expert 1536 · 2 sharedlayer 24: MLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 24: MoE: 160 experts, 6 active · expert 1536 · 2 sharedlayer 25: MLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 25: MoE: 160 experts, 6 active · expert 1536 · 2 sharedlayer 26: MLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 26: MoE: 160 experts, 6 active · expert 1536 · 2 sharedlayer 27: MLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 27: MoE: 160 experts, 6 active · expert 1536 · 2 sharedlayer 28: MLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 28: MoE: 160 experts, 6 active · expert 1536 · 2 sharedlayer 29: MLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 29: MoE: 160 experts, 6 active · expert 1536 · 2 sharedlayer 30: MLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 30: MoE: 160 experts, 6 active · expert 1536 · 2 sharedlayer 31: MLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 31: MoE: 160 experts, 6 active · expert 1536 · 2 sharedlayer 32: MLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 32: MoE: 160 experts, 6 active · expert 1536 · 2 sharedlayer 33: MLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 33: MoE: 160 experts, 6 active · expert 1536 · 2 sharedlayer 34: MLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 34: MoE: 160 experts, 6 active · expert 1536 · 2 sharedlayer 35: MLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 35: MoE: 160 experts, 6 active · expert 1536 · 2 sharedlayer 36: MLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 36: MoE: 160 experts, 6 active · expert 1536 · 2 sharedlayer 37: MLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 37: MoE: 160 experts, 6 active · expert 1536 · 2 sharedlayer 38: MLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 38: MoE: 160 experts, 6 active · expert 1536 · 2 sharedlayer 39: MLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 39: MoE: 160 experts, 6 active · expert 1536 · 2 sharedlayer 40: MLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 40: MoE: 160 experts, 6 active · expert 1536 · 2 sharedlayer 41: MLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 41: MoE: 160 experts, 6 active · expert 1536 · 2 sharedlayer 42: MLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 42: MoE: 160 experts, 6 active · expert 1536 · 2 sharedlayer 43: MLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 43: MoE: 160 experts, 6 active · expert 1536 · 2 sharedlayer 44: MLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 44: MoE: 160 experts, 6 active · expert 1536 · 2 sharedlayer 45: MLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 45: MoE: 160 experts, 6 active · expert 1536 · 2 sharedlayer 46: MLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 46: MoE: 160 experts, 6 active · expert 1536 · 2 sharedlayer 47: MLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 47: MoE: 160 experts, 6 active · expert 1536 · 2 sharedlayer 48: MLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 48: MoE: 160 experts, 6 active · expert 1536 · 2 sharedlayer 49: MLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 49: MoE: 160 experts, 6 active · expert 1536 · 2 sharedlayer 50: MLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 50: MoE: 160 experts, 6 active · expert 1536 · 2 sharedlayer 51: MLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 51: MoE: 160 experts, 6 active · expert 1536 · 2 sharedlayer 52: MLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 52: MoE: 160 experts, 6 active · expert 1536 · 2 sharedlayer 53: MLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 53: MoE: 160 experts, 6 active · expert 1536 · 2 sharedlayer 54: MLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 54: MoE: 160 experts, 6 active · expert 1536 · 2 sharedlayer 55: MLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 55: MoE: 160 experts, 6 active · expert 1536 · 2 sharedlayer 56: MLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 56: MoE: 160 experts, 6 active · expert 1536 · 2 sharedlayer 57: MLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 57: MoE: 160 experts, 6 active · expert 1536 · 2 sharedlayer 58: MLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 58: MoE: 160 experts, 6 active · expert 1536 · 2 sharedlayer 59: MLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 59: MoE: 160 experts, 6 active · expert 1536 · 2 shared03059× 59normMLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536+normMoE: 160 experts, 6 active · expert 1536 · 2 shared+× 1normMLA: 128 heads · KV latent 512 + RoPE 64 · Q latent 1536+normGated MLP: 12288+MLAdense FFNMoE FFN

Modelled costs

From the cost model, batch size 1. Totals the lab states are in the table above; differences come from rounding, from what a lab counts, or from parts the model does not describe (listed on the about page).

Parameters (modelled)236B
Active per token (modelled)21.4B
Without embeddings and output head235B total, 20.3B active
Published weights (Hugging Face count)236B
KV cache per token, BF16 (layers that grow with context)67.5 KiB
KV cache + state at 160K tokens, BF1610.5 GiB
Decode FLOPs per token at 4K context61.8 GFLOP
Prefill FLOPs for a 4K prompt208 TFLOP

KV cache against context

DeepSeek-V2: KV cache bytes against context length101001,00010,000100,000980 KiB9.5 MiB95 MiB950 MiB9.3 GiBcontext (tokens)KV cache + state (BF16)DeepSeek-V2

Compare with other models →

Every architecture field

FieldValueSource
d_model5,120config.jsonconfig.jsonhidden_size
vocab102,400config.jsonconfig.jsonvocab_size
tied_embeddingsfalseconfig.jsonconfig.jsontie_word_embeddings
mixers.mla.typemlacodemodelling codemulti-head latent attention
mixers.mla.heads128config.jsonconfig.jsonnum_attention_heads
mixers.mla.q_lora_rank1,536config.jsonconfig.jsonq_lora_rank
mixers.mla.kv_lora_rank512config.jsonconfig.jsonkv_lora_rank
mixers.mla.qk_nope128config.jsonconfig.jsonqk_nope_head_dim
mixers.mla.qk_rope64config.jsonconfig.jsonqk_rope_head_dim
mixers.mla.v_head_dim128config.jsonconfig.jsonv_head_dim
ffns.dense.typedensecodemodelling codeMLP block
ffns.dense.d_ff12,288config.jsonconfig.jsonintermediate_size
ffns.dense.gatedtruecodemodelling codeMLP: gated (SwiGLU/GeGLU)
ffns.moe.typemoecodemodelling codeMoE block
ffns.moe.experts160config.jsonconfig.jsonn_routed_experts
ffns.moe.active6config.jsonconfig.jsonnum_experts_per_tok
ffns.moe.d_expert1,536config.jsonconfig.jsonmoe_intermediate_size
ffns.moe.gatedtruecodemodelling codeexperts are gated MLPs
ffns.moe.shared2config.jsonconfig.jsonn_shared_experts
ffns.moe.d_shared1,536config.jsonconfig.jsonmoe_intermediate_size
layout1× mla/dense · 59× mla/moeconfig.jsonconfig.jsonnum_hidden_layers, first_k_dense_replace

Sources