DeepSeek-V4.1-Flash
DeepSeek · DeepSeek V4 · open weights · causal encoder-decoder
- Sliding window
- Sparse attention
- Compressed KV
- RoPE
- Partial RoPE
- MoE
- Shared expert
- MTP
- Cross-layer KV sharing
- Causal encoder-decoder
Facts and where they come from
| Released | 2026-09 | config.jsonconfig.jsonHugging Face repository creation date (api.createdAt) |
|---|---|---|
| Licence | mit | config.jsonconfig.jsonREADME metadata: license |
| Total parameters | 552B | paperarXiv 2609.19969abstract p.1; Table 1 p.24: 552B backbone parameters (plus 196B Engram parameters) |
| Active parameters | 16B | paperarXiv 2609.19969abstract p.1: 8B activated per token at prefill, 16B at decode |
| Context length | 1M tokens | config.jsonconfig.jsontext_config.max_position_embeddings |
| Norm placement | not disclosed | not disclosednot checked in the modelling code |
| Norm type | not disclosed | not disclosednot checked in the modelling code |
| QK-norm | no | codemodelling codeno q/k normalisation in the attention block |
| Positional encoding | RoPE on 12.5% of each head | config.jsonconfig.jsonqk_rope_head_dim / head_dim |
Architecture, drawn from the data
compressed 64h, window 128. Each column is one layer: its token mixer above, its feed-forward block below. Paler columns reuse another layer’s keys and values.
Modelled costs
From the cost model, batch size 1. Totals the lab states are in the table above; differences come from rounding, from what a lab counts, or from parts the model does not describe (listed on the about page).
| Parameters (modelled) | 552B |
|---|---|
| Active per token (modelled) | 16.6B |
| Without embeddings and output head | 550B total, 15.3B active |
| Multi-token-prediction layers (extra) | 41.4B |
| Published weights (Hugging Face count) | 763B (packed low-bit tensors, so not comparable) |
| KV cache per token, BF16 (layers that grow with context) | 3.13 KiB |
| KV cache + state at 1M tokens, BF16 | 3.13 GiB |
| Decode FLOPs per token at 4K context | 45.5 GFLOP |
| Prefill FLOPs for a 4K prompt | 75.2 TFLOP |
KV cache against context
Every architecture field
| Field | Value | Source |
|---|---|---|
| d_model | 5,120 | config.jsonconfig.jsontext_config.hidden_size |
| vocab | 129,280 | config.jsonconfig.jsontext_config.vocab_size |
| tied_embeddings | false | config.jsonconfig.jsontext_config.tie_word_embeddings |
| mixers.csa.type | csa | codemodelling codeDeepSeek V4 compressed attention (CSA/HCA) with a sliding window |
| mixers.csa.heads | 64 | config.jsonconfig.jsontext_config.num_attention_heads |
| mixers.csa.head_dim | 512 | config.jsonconfig.jsontext_config.head_dim |
| mixers.csa.q_lora_rank | 1,280 | config.jsonconfig.jsontext_config.q_lora_rank |
| mixers.csa.o_lora_rank | 1,024 | config.jsonconfig.jsontext_config.o_lora_rank |
| mixers.csa.o_groups | 8 | config.jsonconfig.jsontext_config.o_groups |
| mixers.csa.window | 128 | config.jsonconfig.jsontext_config.sliding_window |
| mixers.csa.indexer.heads | 32 | config.jsonconfig.jsontext_config.index_n_heads |
| mixers.csa.indexer.head_dim | 128 | config.jsonconfig.jsontext_config.index_head_dim |
| mixers.csa.indexer.topk | 512 | config.jsonconfig.jsontext_config.index_topk |
| ffns.moe.type | moe | codemodelling codeMoE block |
| ffns.moe.experts | 384 | config.jsonconfig.jsontext_config.n_routed_experts |
| ffns.moe.active | 6 | config.jsonconfig.jsontext_config.num_experts_per_tok |
| ffns.moe.d_expert | 2,304 | config.jsonconfig.jsontext_config.moe_intermediate_size |
| ffns.moe.gated | true | codemodelling codeexperts are gated MLPs |
| ffns.moe.shared | 1 | config.jsonconfig.jsontext_config.n_shared_experts |
| ffns.moe.d_shared | 2,304 | config.jsonconfig.jsontext_config.moe_intermediate_size |
| layout | 2× csa/moe (ratio=0,kv_source=false,indexer=false) · 1× csa/moe (ratio=2,kv_source=true,indexer=true) · 5× csa/moe (ratio=2,kv_source=false,indexer=false) · 1× csa/moe (ratio=2,kv_source=true,indexer=true) · 5× csa/moe (ratio=2,kv_source=false,indexer=false) · 1× csa/moe (ratio=2,kv_source=true,indexer=true) · 5× csa/moe (ratio=2,kv_source=false,indexer=false) · 1× csa/moe (ratio=1,kv_source=true,indexer=true) · 3× csa/moe (ratio=1,kv_source=false,indexer=false) · 1× csa/moe (ratio=1,kv_source=false,indexer=true) · 3× csa/moe (ratio=1,kv_source=false,indexer=false) · 1× csa/moe (ratio=1,kv_source=false,indexer=true) · 3× csa/moe (ratio=1,kv_source=false,indexer=false) · 1× csa/moe (ratio=1,kv_source=false,indexer=true) · 3× csa/moe (ratio=1,kv_source=false,indexer=false) · 1× csa/moe (ratio=1,kv_source=false,indexer=true) · 3× csa/moe (ratio=1,kv_source=false,indexer=false) | config.jsonconfig.jsontext_config.compress_ratios, kv_source_layer_ids, index_source_layer_ids |
| mtp_layers | 3 | config.jsonconfig.jsontext_config.num_nextn_predict_layers |
| ced_encoder_layers | 20 | paperarXiv 2609.19969Fig. 3 caption p.7; §2.1: a 20-layer causal encoder and a 20-layer decoder |
| ced_window | 128 | paperarXiv 2609.19969§4.2.1 p.22: n_win = 128 (Decoder SWA Bounded Replay) |
Sources
- config.json @ 2cba9e4
- model card
- arXiv 2609.19969
- modelling code · transformers 5.18.0 modelling code, or the model repository's own modelling file at the pinned revision
Listed in the LLM Architecture Gallery checklist as “DeepSeek V4.1-Flash (552B)” (name only; see about).