Qwen3.5 397B-A17B
Alibaba Cloud (Qwen) · Qwen3.5 · open weights · multimodal (text stack modelled)
- GQA
- Linear attention
- DeltaNet / KDA
- Hybrid
- RoPE
- Partial RoPE
- Pre-norm
- QK-norm
- MoE
- Shared expert
- MTP
Facts and where they come from
| Released | 2026-02 | config.jsonconfig.jsonHugging Face repository creation date (api.createdAt) |
|---|---|---|
| Licence | apache-2.0 | config.jsonconfig.jsonREADME metadata: license |
| Total parameters | 397B | labmodel cardREADME: 397B in total and 17B activated |
| Active parameters | 17B | labmodel cardREADME: 17B activated |
| Context length | 256K tokens | labmodel cardREADME: 262,144 natively |
| Norm placement | pre | codemodelling codetransformers 5.18.0 / repo modelling code for qwen3_5_moe_text: input_layernorm before attention, post_attention_layernorm before the MLP |
| Norm type | RMSNorm | codemodelling codetransformers 5.18.0 / repo modelling code for qwen3_5_moe_text: input_layernorm before attention, post_attention_layernorm before the MLP |
| QK-norm | yes | codemodelling codetransformers 5.18.0 qwen3_5_moe: q_norm and k_norm |
| Positional encoding | multimodal RoPE on 25% of each head | config.jsonconfig.jsontext_config.rope_parameters.partial_rotary_factor |
| Parallel attention and MLP | no | codemodelling codetransformers 5.18.0 / repo modelling code for qwen3_5_moe_text: input_layernorm before attention, post_attention_layernorm before the MLP |
Architecture, drawn from the data
45× Gated DeltaNet + 15× GQA 32q/2kv. Each column is one layer: its token mixer above, its feed-forward block below. Paler columns reuse another layer’s keys and values.
Modelled costs
From the cost model, batch size 1. Totals the lab states are in the table above; differences come from rounding, from what a lab counts, or from parts the model does not describe (listed on the about page).
| Parameters (modelled) | 396B |
|---|---|
| Active per token (modelled) | 17.3B |
| Without embeddings and output head | 394B total, 15.3B active |
| Multi-token-prediction layers (extra) | 6.6B |
| Published weights (Hugging Face count) | 403B |
| KV cache per token, BF16 (layers that grow with context) | 30 KiB |
| KV cache + state at 256K tokens, BF16 | 7.59 GiB |
| Decode FLOPs per token at 4K context | 35 GFLOP |
| Prefill FLOPs for a 4K prompt | 131 TFLOP |
KV cache against context
Every architecture field
| Field | Value | Source |
|---|---|---|
| d_model | 4,096 | config.jsonconfig.jsontext_config.hidden_size |
| vocab | 248,320 | config.jsonconfig.jsontext_config.vocab_size |
| tied_embeddings | false | codemodelling codetransformers 5.18.0: PretrainedConfig.tie_word_embeddings defaultnot in config |
| mixers.full.type | attn | codemodelling codeattention block |
| mixers.full.heads | 32 | config.jsonconfig.jsontext_config.num_attention_heads |
| mixers.full.kv_heads | 2 | config.jsonconfig.jsontext_config.num_key_value_heads |
| mixers.full.head_dim | 256 | config.jsonconfig.jsontext_config.head_dim |
| mixers.full.gate | elementwise | codemodelling codeattn_output_gate: q_proj also produces a sigmoid output gate |
| mixers.full.qk_norm | true | codemodelling codetransformers 5.18.0 qwen3_5_moe_text: q_norm and k_norm |
| mixers.linear.type | deltanet | codemodelling codeGated DeltaNet linear attention |
| mixers.linear.k_heads | 16 | config.jsonconfig.jsontext_config.linear_num_key_heads |
| mixers.linear.v_heads | 64 | config.jsonconfig.jsontext_config.linear_num_value_heads |
| mixers.linear.k_head_dim | 128 | config.jsonconfig.jsontext_config.linear_key_head_dim |
| mixers.linear.v_head_dim | 128 | config.jsonconfig.jsontext_config.linear_value_head_dim |
| mixers.linear.conv_kernel | 4 | config.jsonconfig.jsontext_config.linear_conv_kernel_dim |
| ffns.moe.type | moe | codemodelling codeMoE block |
| ffns.moe.experts | 512 | config.jsonconfig.jsontext_config.num_experts |
| ffns.moe.active | 10 | config.jsonconfig.jsontext_config.num_experts_per_tok |
| ffns.moe.d_expert | 1,024 | config.jsonconfig.jsontext_config.moe_intermediate_size |
| ffns.moe.gated | true | codemodelling codeexperts are gated MLPs |
| ffns.moe.shared | 1 | codemodelling codeone shared expert of shared_expert_intermediate_size |
| ffns.moe.d_shared | 1,024 | config.jsonconfig.jsontext_config.shared_expert_intermediate_size |
| layout | 3× linear/moe · 1× full/moe · 3× linear/moe · 1× full/moe · 3× linear/moe · 1× full/moe · 3× linear/moe · 1× full/moe · 3× linear/moe · 1× full/moe · 3× linear/moe · 1× full/moe · 3× linear/moe · 1× full/moe · 3× linear/moe · 1× full/moe · 3× linear/moe · 1× full/moe · 3× linear/moe · 1× full/moe · 3× linear/moe · 1× full/moe · 3× linear/moe · 1× full/moe · 3× linear/moe · 1× full/moe · 3× linear/moe · 1× full/moe · 3× linear/moe · 1× full/moe | config.jsonconfig.jsontext_config.layer_types |
| mtp_layers | 1 | config.jsonconfig.jsontext_config.mtp_num_hidden_layers |
| mtp_layer | {"mixer":"full","ffn":"moe","n":1} | codemodelling codeMTP layer: a full-attention block |
Sources
- config.json @ 8472618
- model card
- modelling code · transformers 5.18.0 modelling code, or the model repository's own modelling file at the pinned revision
Listed in the LLM Architecture Gallery checklist as “Qwen3.5 (397B)” (name only; see about).