GLM-5.3-Flash
Z.ai · GLM-5 · open weights · multimodal (text stack modelled)
- MLA
- Sparse attention
- Linear attention
- DeltaNet / KDA
- Hybrid
- RoPE
- Partial RoPE
- NoPE layers
- Pre-norm
- MoE
- Shared expert
- Dense first layers
- MTP
Facts and where they come from
| Released | 2026-08 | config.jsonconfig.jsonHugging Face repository creation date (api.createdAt) |
|---|---|---|
| Licence | mit | config.jsonconfig.jsonREADME metadata: license |
| Total parameters | 320B | labmodel cardREADME: 320B total parameters and just 18B active |
| Active parameters | 18B | labmodel cardREADME: 18B active |
| Context length | 1M tokens | config.jsonconfig.jsontext_config.max_position_embeddings |
| Norm placement | pre | codemodelling codetransformers 5.18.0 / repo modelling code for glm5_next_text: input_layernorm before attention, post_attention_layernorm before the MLP |
| Norm type | RMSNorm | codemodelling codetransformers 5.18.0 / repo modelling code for glm5_next_text: input_layernorm before attention, post_attention_layernorm before the MLP |
| QK-norm | no | codemodelling codeno q/k normalisation in the attention block (MLA normalises its latent vectors, which is not QK-norm) |
| Positional encoding | RoPE on 0% of each head; MLA layers use no RoPE; the linear-attention layers carry order | config.jsonconfig.jsontext_config.mla_use_nope |
| Parallel attention and MLP | no | codemodelling codetransformers 5.18.0 / repo modelling code for glm5_next_text: input_layernorm before attention, post_attention_layernorm before the MLP |
Architecture, drawn from the data
34× Kimi Delta Attention + 11× MLA 64h, latent 512, top-2048. Each column is one layer: its token mixer above, its feed-forward block below. Paler columns reuse another layer’s keys and values.
Modelled costs
From the cost model, batch size 1. Totals the lab states are in the table above; differences come from rounding, from what a lab counts, or from parts the model does not describe (listed on the about page).
| Parameters (modelled) | 314B |
|---|---|
| Active per token (modelled) | 18.5B |
| Without embeddings and output head | 313B total, 17.2B active |
| Multi-token-prediction layers (extra) | 7.48B |
| Published weights (Hugging Face count) | 321B |
| KV cache per token, BF16 (layers that grow with context) | 13.8 KiB |
| KV cache + state at 1M tokens, BF16 | 13.8 GiB |
| Decode FLOPs per token at 4K context | 37.7 GFLOP |
| Prefill FLOPs for a 4K prompt | 147 TFLOP |
KV cache against context
Every architecture field
| Field | Value | Source |
|---|---|---|
| d_model | 4,096 | config.jsonconfig.jsontext_config.hidden_size |
| vocab | 154,880 | config.jsonconfig.jsontext_config.vocab_size |
| tied_embeddings | false | config.jsonconfig.jsontext_config.tie_word_embeddings |
| mixers.mla.type | mla | codemodelling codemulti-head latent attention |
| mixers.mla.heads | 64 | config.jsonconfig.jsontext_config.num_attention_heads |
| mixers.mla.q_lora_rank | 1,536 | config.jsonconfig.jsontext_config.q_lora_rank |
| mixers.mla.kv_lora_rank | 512 | config.jsonconfig.jsontext_config.kv_lora_rank |
| mixers.mla.qk_nope | 256 | config.jsonconfig.jsontext_config.qk_nope_head_dim |
| mixers.mla.qk_rope | 0 | config.jsonconfig.jsontext_config.qk_rope_head_dim |
| mixers.mla.v_head_dim | 256 | config.jsonconfig.jsontext_config.v_head_dim |
| mixers.mla.indexer.heads | 32 | config.jsonconfig.jsontext_config.index_n_heads |
| mixers.mla.indexer.head_dim | 128 | config.jsonconfig.jsontext_config.index_head_dim |
| mixers.mla.indexer.topk | 2,048 | config.jsonconfig.jsontext_config.index_topk |
| mixers.kda.type | kda | codemodelling codeKimi Delta Attention |
| mixers.kda.k_heads | 64 | config.jsonconfig.jsontext_config.linear_attn_config.num_heads |
| mixers.kda.v_heads | 64 | config.jsonconfig.jsontext_config.linear_attn_config.num_heads |
| mixers.kda.k_head_dim | 128 | config.jsonconfig.jsontext_config.linear_attn_config.head_dim |
| mixers.kda.v_head_dim | 128 | config.jsonconfig.jsontext_config.linear_attn_config.head_dim |
| mixers.kda.conv_kernel | 4 | config.jsonconfig.jsontext_config.linear_attn_config.short_conv_kernel_size |
| ffns.dense.type | dense | codemodelling codeMLP block |
| ffns.dense.d_ff | 12,288 | config.jsonconfig.jsontext_config.intermediate_size |
| ffns.dense.gated | true | codemodelling codeMLP: gated (SwiGLU/GeGLU) |
| ffns.moe.type | moe | codemodelling codeMoE block |
| ffns.moe.experts | 288 | config.jsonconfig.jsontext_config.n_routed_experts |
| ffns.moe.active | 8 | config.jsonconfig.jsontext_config.num_experts_per_tok |
| ffns.moe.d_expert | 2,048 | config.jsonconfig.jsontext_config.moe_intermediate_size |
| ffns.moe.gated | true | codemodelling codeexperts are gated MLPs |
| ffns.moe.shared | 1 | config.jsonconfig.jsontext_config.n_shared_experts |
| ffns.moe.d_shared | 2,048 | config.jsonconfig.jsontext_config.moe_intermediate_size |
| layout | 3× kda/dense · 1× mla/moe · 3× kda/moe · 1× mla/moe · 3× kda/moe · 1× mla/moe · 3× kda/moe · 1× mla/moe · 3× kda/moe · 1× mla/moe · 3× kda/moe · 1× mla/moe · 3× kda/moe · 1× mla/moe · 3× kda/moe · 1× mla/moe · 3× kda/moe · 1× mla/moe · 3× kda/moe · 1× mla/moe · 3× kda/moe · 1× mla/moe · 1× kda/moe | config.jsonconfig.jsontext_config.layer_types, mlp_layer_types |
| mtp_layers | 1 | config.jsonconfig.jsontext_config.num_nextn_predict_layers |
Sources
- config.json @ eb9eb20
- model card
- modelling code · transformers 5.18.0 modelling code, or the model repository's own modelling file at the pinned revision
Listed in the LLM Architecture Gallery checklist as “GLM-5.3-Flash (320B)” (name only; see about).