ZAYA1-8B
Zyphra · ZAYA · open weights
- GQA
- RoPE
- Partial RoPE
- Pre-norm
- MoE
Facts and where they come from
| Released | 2026-05 | config.jsonconfig.jsonHugging Face repository creation date (api.createdAt) |
|---|---|---|
| Licence | apache-2.0 | config.jsonconfig.jsonREADME metadata: license |
| Total parameters | 8.4B | labmodel cardREADME: 760M active parameters and 8.4B total parameters |
| Active parameters | 760M | labmodel cardREADME: 760M active |
| Context length | 128K tokens | config.jsonconfig.jsonmax_position_embeddings |
| Norm placement | pre | codemodelling codetransformers 5.18.0 / repo modelling code for zaya: input_layernorm before attention, post_attention_layernorm before the MLP |
| Norm type | RMSNorm | codemodelling codetransformers 5.18.0 / repo modelling code for zaya: input_layernorm before attention, post_attention_layernorm before the MLP |
| QK-norm | no | codemodelling codeno q/k normalisation in the attention block |
| Positional encoding | RoPE on 50% of each head | config.jsonconfig.jsonpartial_rotary_factor |
| Parallel attention and MLP | no | codemodelling codetransformers 5.18.0 / repo modelling code for zaya: input_layernorm before attention, post_attention_layernorm before the MLP |
Architecture, drawn from the data
GQA 8q/2kv. Each column is one layer: its token mixer above, its feed-forward block below. Paler columns reuse another layer’s keys and values.
Modelled costs
From the cost model, batch size 1. Totals the lab states are in the table above; differences come from rounding, from what a lab counts, or from parts the model does not describe (listed on the about page).
| Parameters (modelled) | 8.8B |
|---|---|
| Active per token (modelled) | 1.25B |
| Without embeddings and output head | 8.26B total, 715M active |
| Published weights (Hugging Face count) | 8.84B |
| KV cache per token, BF16 (layers that grow with context) | 40 KiB |
| KV cache + state at 128K tokens, BF16 | 5 GiB |
| Decode FLOPs per token at 4K context | 3.17 GFLOP |
| Prefill FLOPs for a 4K prompt | 7.23 TFLOP |
KV cache against context
Every architecture field
| Field | Value | Source |
|---|---|---|
| d_model | 2,048 | config.jsonconfig.jsonhidden_size |
| vocab | 262,272 | config.jsonconfig.jsonvocab_size |
| tied_embeddings | true | config.jsonconfig.jsontie_word_embeddings |
| mixers.full.type | attn | codemodelling codeCCA (compressed convolutional attention) approximated as GQA attentionapproximation |
| mixers.full.heads | 8 | config.jsonconfig.jsonnum_attention_heads |
| mixers.full.kv_heads | 2 | config.jsonconfig.jsonnum_key_value_heads |
| mixers.full.head_dim | 128 | config.jsonconfig.jsonhead_dim |
| ffns.moe.type | moe | codemodelling codeMoE block |
| ffns.moe.experts | 16 | config.jsonconfig.jsonnum_experts |
| ffns.moe.active | 1 | config.jsonconfig.jsonnum_experts_per_tok |
| ffns.moe.d_expert | 2,048 | config.jsonconfig.jsonmoe_intermediate_size |
| ffns.moe.gated | true | codemodelling codeexperts are gated MLPs |
| layout | 40× full/moe | config.jsonconfig.jsonnum_hidden_layers |
Sources
- config.json @ 67d34da
- model card
- arXiv 2605.05365
- modelling code · transformers 5.18.0 modelling code, or the model repository's own modelling file at the pinned revision
Listed in the LLM Architecture Gallery checklist as “ZAYA1-8B (8.4B)” (name only; see about).