Tiny Aya
Cohere · Aya · open weights
- GQA
- Sliding window
- RoPE
- NoPE layers
- Parallel block
Facts and where they come from
| Released | 2026-02 | config.jsonconfig.jsonHugging Face repository creation date (api.createdAt) |
|---|---|---|
| Licence | cc-by-nc-4.0 | config.jsonconfig.jsonREADME metadata: license |
| Total parameters | 3.35B | labmodel cardREADME: a pretrained 3.35 billion parameter model |
| Active parameters | not disclosed | not disclosed |
| Context length | 8K tokens | paperpaperTable 2: input context 8192 |
| Norm placement | parallel | codemodelling codetransformers 5.18.0 cohere2: a single input_layernorm feeds attention and the MLP |
| Norm type | LayerNorm | codemodelling codetransformers 5.18.0 cohere2: a single input_layernorm feeds attention and the MLP |
| QK-norm | no | codemodelling codeno q/k normalisation in the attention block |
| Positional encoding | RoPE; full-attention layers have no RoPE; sliding-window layers use it | paperpaper§3.1: RoPE on sliding-window layers, NoPE on full-attention layers |
| Parallel attention and MLP | yes | codemodelling codetransformers 5.18.0 cohere2: a single input_layernorm feeds attention and the MLP |
Architecture, drawn from the data
27× GQA 16q/4kv, window 4096 + 9× GQA 16q/4kv. Each column is one layer: its token mixer above, its feed-forward block below. Paler columns reuse another layer’s keys and values.
Modelled costs
From the cost model, batch size 1. Totals the lab states are in the table above; differences come from rounding, from what a lab counts, or from parts the model does not describe (listed on the about page).
| Parameters (modelled) | 3.35B |
|---|---|
| Active per token (modelled) | 3.35B |
| Without embeddings and output head | 2.81B total, 2.81B active |
| Published weights (Hugging Face count) | 3.35B |
| KV cache per token, BF16 (layers that grow with context) | 18 KiB |
| KV cache + state at 8K tokens, BF16 | 360 MiB |
| Decode FLOPs per token at 4K context | 7.91 GFLOP |
| Prefill FLOPs for a 4K prompt | 25.5 TFLOP |
KV cache against context
Every architecture field
| Field | Value | Source |
|---|---|---|
| d_model | 2,048 | paperpaperTable 2 (p. 9): embedding dims 2048 |
| vocab | 262,144 | paperpaperTable 2: vocab size 262k (262,144 assumed) |
| tied_embeddings | true | paperpaperTable 2: 0.5B embedding parameters counted once (tied) |
| mixers.full.type | attn | codemodelling codeattention block |
| mixers.full.heads | 16 | paperpaperTable 2: num heads 16 |
| mixers.full.kv_heads | 4 | paperpaperTable 2: num KV heads 4 |
| mixers.full.head_dim | 128 | codemodelling codetransformers 5.18.0: head_dim = hidden_size / num_attention_heads |
| mixers.sliding.type | attn | codemodelling codeattention block |
| mixers.sliding.heads | 16 | paperpaperTable 2: num heads 16 |
| mixers.sliding.kv_heads | 4 | paperpaperTable 2: num KV heads 4 |
| mixers.sliding.head_dim | 128 | codemodelling codetransformers 5.18.0: head_dim = hidden_size / num_attention_heads |
| mixers.sliding.window | 4,096 | paperpaperTable 2: sliding window 4096 |
| ffns.dense.type | dense | codemodelling codeMLP block |
| ffns.dense.d_ff | 11,008 | paperpaperTable 2: FFN hidden dims 11008 |
| ffns.dense.gated | true | codemodelling codeMLP: gated (SwiGLU/GeGLU) |
| layout | 3× sliding/dense · 1× full/dense · 3× sliding/dense · 1× full/dense · 3× sliding/dense · 1× full/dense · 3× sliding/dense · 1× full/dense · 3× sliding/dense · 1× full/dense · 3× sliding/dense · 1× full/dense · 3× sliding/dense · 1× full/dense · 3× sliding/dense · 1× full/dense · 3× sliding/dense · 1× full/dense | paperpaper§3.1: sliding window and full attention in a 3:1 ratio |
| norms_per_layer | 1 | codemodelling codeparallel block: one norm feeds attention and MLP |
Sources
- config.json @ d62ac17
- model card
- paper · Tiny Aya: Bridging Scale and Multilingual Depth
- modelling code · transformers 5.18.0 modelling code, or the model repository's own modelling file at the pinned revision
Listed in the LLM Architecture Gallery checklist as “Tiny Aya (3.35B)” (name only; see about).