Nemotron 3 Ultra
NVIDIA · Nemotron 3 · open weights
- GQA
- Mamba (SSM)
- Hybrid
- RoPE
- Pre-norm
- MoE
- Shared expert
- Latent MoE
- MTP
Facts and where they come from
| Released | 2026-06 | config.jsonconfig.jsonHugging Face repository creation date (api.createdAt) |
|---|---|---|
| Licence | other | config.jsonconfig.jsonREADME metadata: license |
| Total parameters | 550B | labmodel cardREADME: 550B Total / 55B Active |
| Active parameters | 55B | labmodel cardREADME: 55B Active |
| Context length | 1M tokens | labmodel cardREADME: up to 1M tokens |
| Norm placement | pre | codemodelling codetransformers 5.18.0 nemotron_h: one norm before each mixer or MLP block |
| Norm type | RMSNorm | codemodelling codetransformers 5.18.0 nemotron_h: one norm before each mixer or MLP block |
| QK-norm | no | codemodelling codeno q/k normalisation in the attention block |
| Positional encoding | RoPE | config.jsonconfig.jsonpartial_rotary_factor |
| Parallel attention and MLP | no | codemodelling codetransformers 5.18.0 nemotron_h: one norm before each mixer or MLP block |
Architecture, drawn from the data
48× Mamba-2 + 12× GQA 64q/2kv. Each column is one layer: its token mixer above, its feed-forward block below. Paler columns reuse another layer’s keys and values.
Modelled costs
From the cost model, batch size 1. Totals the lab states are in the table above; differences come from rounding, from what a lab counts, or from parts the model does not describe (listed on the about page).
| Parameters (modelled) | 549B |
|---|---|
| Active per token (modelled) | 56.1B |
| Without embeddings and output head | 547B total, 53.9B active |
| Multi-token-prediction layers (extra) | 11.2B |
| Published weights (Hugging Face count) | 303B (packed low-bit tensors, so not comparable) |
| KV cache per token, BF16 (layers that grow with context) | 12 KiB |
| KV cache + state at 1M tokens, BF16 | 12.2 GiB |
| Decode FLOPs per token at 4K context | 112 GFLOP |
| Prefill FLOPs for a 4K prompt | 447 TFLOP |
KV cache against context
Every architecture field
| Field | Value | Source |
|---|---|---|
| d_model | 8,192 | config.jsonconfig.jsonhidden_size |
| vocab | 131,072 | config.jsonconfig.jsonvocab_size |
| tied_embeddings | false | config.jsonconfig.jsontie_word_embeddings |
| mixers.full.type | attn | codemodelling codeattention block |
| mixers.full.heads | 64 | config.jsonconfig.jsonnum_attention_heads |
| mixers.full.kv_heads | 2 | config.jsonconfig.jsonnum_key_value_heads |
| mixers.full.head_dim | 128 | config.jsonconfig.jsonhead_dim |
| mixers.mamba.type | mamba2 | codemodelling codeMamba-2 mixer |
| mixers.mamba.heads | 256 | config.jsonconfig.jsonmamba_num_heads |
| mixers.mamba.head_dim | 64 | config.jsonconfig.jsonmamba_head_dim |
| mixers.mamba.state | 128 | config.jsonconfig.jsonssm_state_size |
| mixers.mamba.groups | 8 | config.jsonconfig.jsonn_groups |
| mixers.mamba.conv_kernel | 4 | config.jsonconfig.jsonconv_kernel |
| ffns.moe.type | moe | codemodelling codeMoE block |
| ffns.moe.experts | 512 | config.jsonconfig.jsonn_routed_experts |
| ffns.moe.active | 22 | config.jsonconfig.jsonnum_experts_per_tok |
| ffns.moe.d_expert | 5,120 | config.jsonconfig.jsonmoe_intermediate_size |
| ffns.moe.gated | false | codemodelling codeexperts are two-matrix MLPs |
| ffns.moe.shared | 1 | config.jsonconfig.jsonn_shared_experts |
| ffns.moe.d_shared | 10,240 | config.jsonconfig.jsonmoe_shared_expert_intermediate_size |
| ffns.moe.latent | 2,048 | config.jsonconfig.jsonmoe_latent_size |
| ffns.dense.type | dense | codemodelling codeMLP block |
| ffns.dense.d_ff | 5,120 | config.jsonconfig.jsonintermediate_size |
| ffns.dense.gated | false | codemodelling codeMLP: two matrices, no gate |
| layout | 1× mamba/none · 1× none/moe · 1× mamba/none · 1× none/moe · 1× mamba/none · 1× none/moe · 1× mamba/none · 1× full/none · 1× none/moe · 1× mamba/none · 1× none/moe · 1× mamba/none · 1× none/moe · 1× mamba/none · 1× full/none · 1× none/moe · 1× mamba/none · 1× none/moe · 1× mamba/none · 1× none/moe · 1× mamba/none · 1× none/moe · 1× mamba/none · 1× full/none · 1× none/moe · 1× mamba/none · 1× none/moe · 1× mamba/none · 1× none/moe · 1× mamba/none · 1× none/moe · 1× mamba/none · 1× full/none · 1× none/moe · 1× mamba/none · 1× none/moe · 1× mamba/none · 1× none/moe · 1× mamba/none · 1× full/none · 1× none/moe · 1× mamba/none · 1× none/moe · 1× mamba/none · 1× none/moe · 1× mamba/none · 1× none/moe · 1× mamba/none · 1× full/none · 1× none/moe · 1× mamba/none · 1× none/moe · 1× mamba/none · 1× none/moe · 1× mamba/none · 1× none/moe · 1× mamba/none · 1× full/none · 1× none/moe · 1× mamba/none · 1× none/moe · 1× mamba/none · 1× none/moe · 1× mamba/none · 1× full/none · 1× none/moe · 1× mamba/none · 1× none/moe · 1× mamba/none · 1× none/moe · 1× mamba/none · 1× none/moe · 1× mamba/none · 1× full/none · 1× none/moe · 1× mamba/none · 1× none/moe · 1× mamba/none · 1× none/moe · 1× mamba/none · 1× none/moe · 1× mamba/none · 1× full/none · 1× none/moe · 1× mamba/none · 1× none/moe · 1× mamba/none · 1× none/moe · 1× mamba/none · 1× full/none · 1× none/moe · 1× mamba/none · 1× none/moe · 1× mamba/none · 1× none/moe · 1× mamba/none · 1× none/moe · 1× mamba/none · 1× full/none · 1× none/moe · 1× mamba/none · 1× none/moe · 1× mamba/none · 1× none/moe · 1× mamba/none · 1× none/moe · 1× mamba/none · 1× none/moe | config.jsonconfig.jsonlayers_block_type |
| norms_per_layer | 1 | codemodelling codetransformers 5.18.0 nemotron_h: one pre-norm per block (each block is a mixer or an MLP) |
| mtp_layers | 1 | config.jsonconfig.jsonnum_nextn_predict_layers |
| mtp_layer | {"mixer":"full","ffn":"moe","n":1} | codemodelling codeMTP: attention + MoE blocks |
Sources
- config.json @ 0246264
- model card
- modelling code · transformers 5.18.0 modelling code, or the model repository's own modelling file at the pinned revision
Listed in the LLM Architecture Gallery checklist as “Nemotron 3 Ultra (550B-A55B)” (name only; see about).