Kolibri-1
Aleph Alpha · Kolibri · open weights
- GQA
- Sliding window
- RoPE
- MoE
- Shared expert
Facts and where they come from
| Released | 2026-10 | config.jsonconfig.jsonHugging Face repository creation date (api.createdAt) |
|---|---|---|
| Licence | apache-2.0 | config.jsonconfig.jsonREADME metadata: license |
| Total parameters | 78B | labmodel cardREADME: Total parameters 78B (78,103,074,560) |
| Active parameters | not disclosed | not disclosed |
| Context length | 1M tokens | labmodel cardREADME: Context length 1,048,576 tokens |
| Norm placement | not disclosed | not disclosednot checked in the modelling code |
| Norm type | not disclosed | not disclosednot checked in the modelling code |
| QK-norm | no | codemodelling codeno q/k normalisation in the attention block |
| Positional encoding | RoPE | codemodelling coderotary on the full head (default) |
Architecture, drawn from the data
40× GQA 48q/4kv, window 513 + 10× GQA 48q/4kv. Each column is one layer: its token mixer above, its feed-forward block below. Paler columns reuse another layer’s keys and values.
Modelled costs
From the cost model, batch size 1. Totals the lab states are in the table above; differences come from rounding, from what a lab counts, or from parts the model does not describe (listed on the about page).
| Parameters (modelled) | 78.1B |
|---|---|
| Active per token (modelled) | 3.78B |
| Without embeddings and output head | 77.4B total, 3.13B active |
| Published weights (Hugging Face count) | 78.1B |
| KV cache per token, BF16 (layers that grow with context) | 20 KiB |
| KV cache + state at 1M tokens, BF16 | 20 GiB |
| Decode FLOPs per token at 4K context | 8.42 GFLOP |
| Prefill FLOPs for a 4K prompt | 29.6 TFLOP |
KV cache against context
Every architecture field
| Field | Value | Source |
|---|---|---|
| d_model | 2,560 | config.jsonconfig.jsonhidden_size |
| vocab | 128,000 | config.jsonconfig.jsonvocab_size |
| tied_embeddings | false | config.jsonconfig.jsontie_word_embeddings |
| mixers.full.type | attn | codemodelling codeattention block |
| mixers.full.heads | 48 | config.jsonconfig.jsonnum_attention_heads |
| mixers.full.kv_heads | 4 | config.jsonconfig.jsonnum_key_value_heads |
| mixers.full.head_dim | 128 | config.jsonconfig.jsonhead_dim |
| mixers.sliding.type | attn | codemodelling codeattention block |
| mixers.sliding.heads | 48 | config.jsonconfig.jsonnum_attention_heads |
| mixers.sliding.kv_heads | 4 | config.jsonconfig.jsonnum_key_value_heads |
| mixers.sliding.head_dim | 128 | config.jsonconfig.jsonhead_dim |
| mixers.sliding.window | 513 | config.jsonconfig.jsonsliding_window |
| ffns.moe.type | moe | codemodelling codeMoE block |
| ffns.moe.experts | 384 | config.jsonconfig.jsonnum_experts |
| ffns.moe.active | 6 | config.jsonconfig.jsonnum_experts_per_tok |
| ffns.moe.d_expert | 512 | config.jsonconfig.jsonmoe_intermediate_size |
| ffns.moe.gated | true | codemodelling codeexperts are gated MLPs |
| ffns.moe.shared | 1 | codemodelling codeone shared expert of shared_expert_intermediate_size |
| ffns.moe.d_shared | 512 | config.jsonconfig.jsonshared_expert_intermediate_size |
| layout | 4× sliding/moe · 1× full/moe · 4× sliding/moe · 1× full/moe · 4× sliding/moe · 1× full/moe · 4× sliding/moe · 1× full/moe · 4× sliding/moe · 1× full/moe · 4× sliding/moe · 1× full/moe · 4× sliding/moe · 1× full/moe · 4× sliding/moe · 1× full/moe · 4× sliding/moe · 1× full/moe · 4× sliding/moe · 1× full/moe | config.jsonconfig.jsonlayer_types |
Sources
- config.json @ e52eb46
- model card
- modelling code · transformers 5.18.0 modelling code, or the model repository's own modelling file at the pinned revision
Listed in the LLM Architecture Gallery checklist as “Kolibri-1 (78B)” (name only; see about).