North Mini Code 1.0
Cohere · Command A · open weights
- GQA
- Sliding window
- RoPE
- NoPE layers
- MoE
- Dense first layers
- Parallel block
Facts and where they come from
| Released | 2026-06 | config.jsonconfig.jsonHugging Face repository creation date (api.createdAt) |
|---|---|---|
| Licence | apache-2.0 | config.jsonconfig.jsonREADME metadata: license |
| Total parameters | 30B | labmodel cardREADME: 30B total; 3B active |
| Active parameters | 3B | labmodel cardREADME: 3B active |
| Context length | 256K tokens | labmodel cardREADME: context length 256K |
| Norm placement | parallel | codemodelling codetransformers 5.18.0 cohere2_moe: a single input_layernorm feeds attention and the MoE |
| Norm type | LayerNorm | codemodelling codetransformers 5.18.0 cohere2_moe: a single input_layernorm feeds attention and the MoE |
| QK-norm | no | config.jsonconfig.jsonuse_qk_norm |
| Positional encoding | RoPE; full-attention layers have no RoPE; sliding-window layers use it | codemodelling codetransformers 5.18.0 cohere2_moe: RoPE applied only when the layer has a sliding window |
| Parallel attention and MLP | yes | codemodelling codetransformers 5.18.0 cohere2_moe: a single input_layernorm feeds attention and the MoE |
Architecture, drawn from the data
13× GQA 32q/4kv + 36× GQA 32q/4kv, window 4096. Each column is one layer: its token mixer above, its feed-forward block below. Paler columns reuse another layer’s keys and values.
Modelled costs
From the cost model, batch size 1. Totals the lab states are in the table above; differences come from rounding, from what a lab counts, or from parts the model does not describe (listed on the about page).
| Parameters (modelled) | 30.5B |
|---|---|
| Active per token (modelled) | 3.31B |
| Without embeddings and output head | 29.9B total, 2.77B active |
| Published weights (Hugging Face count) | 30.5B |
| KV cache per token, BF16 (layers that grow with context) | 26 KiB |
| KV cache + state at 256K tokens, BF16 | 6.78 GiB |
| Decode FLOPs per token at 4K context | 9.9 GFLOP |
| Prefill FLOPs for a 4K prompt | 29.4 TFLOP |
KV cache against context
Every architecture field
| Field | Value | Source |
|---|---|---|
| d_model | 2,048 | config.jsonconfig.jsonhidden_size |
| vocab | 262,144 | config.jsonconfig.jsonvocab_size |
| tied_embeddings | true | codemodelling codetransformers 5.18.0 cohere2_moe: tie_word_embeddings default true |
| mixers.full.type | attn | codemodelling codeattention block |
| mixers.full.heads | 32 | config.jsonconfig.jsonnum_attention_heads |
| mixers.full.kv_heads | 4 | config.jsonconfig.jsonnum_key_value_heads |
| mixers.full.head_dim | 128 | config.jsonconfig.jsonhead_dim |
| mixers.sliding.type | attn | codemodelling codeattention block |
| mixers.sliding.heads | 32 | config.jsonconfig.jsonnum_attention_heads |
| mixers.sliding.kv_heads | 4 | config.jsonconfig.jsonnum_key_value_heads |
| mixers.sliding.head_dim | 128 | config.jsonconfig.jsonhead_dim |
| mixers.sliding.window | 4,096 | config.jsonconfig.jsonsliding_window |
| ffns.dense.type | dense | codemodelling codeMLP block |
| ffns.dense.d_ff | 3,072 | config.jsonconfig.jsonprefix_dense_intermediate_size |
| ffns.dense.gated | true | codemodelling codeMLP: gated (SwiGLU/GeGLU) |
| ffns.moe.type | moe | codemodelling codeMoE block |
| ffns.moe.experts | 128 | config.jsonconfig.jsonnum_experts |
| ffns.moe.active | 8 | config.jsonconfig.jsonnum_experts_per_tok |
| ffns.moe.d_expert | 768 | config.jsonconfig.jsonintermediate_size |
| ffns.moe.gated | true | codemodelling codeexperts are gated MLPs |
| layout | 1× full/dense · 3× sliding/moe · 1× full/moe · 3× sliding/moe · 1× full/moe · 3× sliding/moe · 1× full/moe · 3× sliding/moe · 1× full/moe · 3× sliding/moe · 1× full/moe · 3× sliding/moe · 1× full/moe · 3× sliding/moe · 1× full/moe · 3× sliding/moe · 1× full/moe · 3× sliding/moe · 1× full/moe · 3× sliding/moe · 1× full/moe · 3× sliding/moe · 1× full/moe · 3× sliding/moe · 1× full/moe | config.jsonconfig.jsonlayer_types, first_k_dense_replace |
| norms_per_layer | 1 | codemodelling codetransformers 5.18.0 cohere2_moe: one LayerNorm per parallel block |
Sources
- config.json @ d11e61a
- model card
- modelling code · transformers 5.18.0 modelling code, or the model repository's own modelling file at the pinned revision
Listed in the LLM Architecture Gallery checklist as “North Mini Code (30B-A3B)” (name only; see about).