llm-architectures-explained

/models

Ember-1

Fireworks AI · Kimi K3 · closed weights

Architecture taken from Kimi K3: announcement: Ember-1 is built on Kimi K3

Facts and where they come from

Released2026-09labannouncementannouncement dated 23 September 2026
Licenceproprietarylabannouncementweights not published
Total parametersnot disclosednot disclosed
Active parametersnot disclosednot disclosed
Context lengthnot disclosednot disclosed
Norm placementprecodemodelling codetransformers 5.18.0 / repo modelling code for kimi_linear: input_layernorm before attention, post_attention_layernorm before the MLP
Norm typeRMSNormcodemodelling codetransformers 5.18.0 / repo modelling code for kimi_linear: input_layernorm before attention, post_attention_layernorm before the MLP
QK-normnocodemodelling codeno q/k normalisation in the attention block (MLA normalises its latent vectors, which is not QK-norm)
Positional encodingRoPE on 33.3% of each head; MLA layers use no RoPE; the linear-attention layers carry orderconfig.jsonconfig.jsontext_config.mla_use_nope
Parallel attention and MLPnocodemodelling codetransformers 5.18.0 / repo modelling code for kimi_linear: input_layernorm before attention, post_attention_layernorm before the MLP

Architecture, drawn from the data

69× Kimi Delta Attention + 24× MLA 96h, latent 512. Each column is one layer: its token mixer above, its feed-forward block below. Paler columns reuse another layer’s keys and values.

Ember-1: layer stack and blockslayers (93)mixer / FFNlayer 0: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 0: Gated MLP: 33792layer 1: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 1: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 2: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 2: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 3: MLA: 96 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 3: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 4: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 4: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 5: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 5: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 6: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 6: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 7: MLA: 96 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 7: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 8: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 8: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 9: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 9: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 10: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 10: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 11: MLA: 96 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 11: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 12: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 12: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 13: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 13: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 14: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 14: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 15: MLA: 96 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 15: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 16: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 16: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 17: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 17: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 18: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 18: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 19: MLA: 96 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 19: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 20: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 20: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 21: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 21: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 22: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 22: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 23: MLA: 96 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 23: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 24: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 24: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 25: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 25: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 26: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 26: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 27: MLA: 96 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 27: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 28: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 28: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 29: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 29: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 30: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 30: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 31: MLA: 96 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 31: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 32: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 32: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 33: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 33: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 34: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 34: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 35: MLA: 96 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 35: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 36: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 36: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 37: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 37: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 38: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 38: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 39: MLA: 96 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 39: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 40: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 40: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 41: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 41: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 42: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 42: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 43: MLA: 96 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 43: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 44: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 44: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 45: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 45: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 46: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 46: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 47: MLA: 96 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 47: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 48: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 48: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 49: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 49: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 50: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 50: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 51: MLA: 96 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 51: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 52: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 52: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 53: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 53: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 54: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 54: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 55: MLA: 96 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 55: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 56: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 56: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 57: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 57: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 58: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 58: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 59: MLA: 96 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 59: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 60: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 60: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 61: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 61: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 62: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 62: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 63: MLA: 96 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 63: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 64: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 64: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 65: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 65: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 66: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 66: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 67: MLA: 96 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 67: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 68: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 68: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 69: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 69: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 70: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 70: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 71: MLA: 96 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 71: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 72: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 72: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 73: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 73: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 74: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 74: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 75: MLA: 96 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 75: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 76: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 76: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 77: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 77: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 78: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 78: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 79: MLA: 96 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 79: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 80: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 80: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 81: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 81: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 82: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 82: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 83: MLA: 96 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 83: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 84: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 84: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 85: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 85: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 86: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 86: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 87: MLA: 96 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 87: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 88: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 88: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 89: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 89: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 90: Kimi Delta Attention: 96 heads · state 128×128 per headlayer 90: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 91: MLA: 96 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 91: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584layer 92: MLA: 96 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 92: MoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 358404692× 68normKimi Delta Attention: 96 heads · state 128×128 per head+normMoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584+× 24normMLA: 96 heads · KV latent 512 + RoPE 64 · Q latent 1536+normMoE: 896 experts, 16 active · expert 3072 · 2 shared · latent 3584+× 1normKimi Delta Attention: 96 heads · state 128×128 per head+normGated MLP: 33792+Kimi Delta AttentionMLAdense FFNMoE FFN

Modelled costs

From the cost model, batch size 1. Totals the lab states are in the table above; differences come from rounding, from what a lab counts, or from parts the model does not describe (listed on the about page).

Parameters (modelled)2.78T
Active per token (modelled)106B
Without embeddings and output head2.78T total, 103B active
KV cache per token, BF16 (layers that grow with context)27 KiB
KV cache + state at 128K tokens, BF163.59 GiB
Decode FLOPs per token at 4K context215 GFLOP
Prefill FLOPs for a 4K prompt860 TFLOP

KV cache against context

Ember-1: KV cache bytes against context length101001,00010,000100,000950 MiBcontext (tokens)KV cache + state (BF16)Ember-1

Compare with other models →

Every architecture field

FieldValueSource
d_model7,168config.jsonconfig.jsontext_config.hidden_size
vocab163,840config.jsonconfig.jsontext_config.vocab_size
tied_embeddingsfalseconfig.jsonconfig.jsontext_config.tie_word_embeddings
mixers.mla.typemlacodemodelling codemulti-head latent attention
mixers.mla.heads96config.jsonconfig.jsontext_config.num_attention_heads
mixers.mla.q_lora_rank1,536config.jsonconfig.jsontext_config.q_lora_rank
mixers.mla.kv_lora_rank512config.jsonconfig.jsontext_config.kv_lora_rank
mixers.mla.qk_nope128config.jsonconfig.jsontext_config.qk_nope_head_dim
mixers.mla.qk_rope64config.jsonconfig.jsontext_config.qk_rope_head_dim
mixers.mla.v_head_dim128config.jsonconfig.jsontext_config.v_head_dim
mixers.mla.gateelementwisecodemodelling codemla_use_output_gate: true
mixers.kda.typekdacodemodelling codeKimi Delta Attention
mixers.kda.k_heads96config.jsonconfig.jsontext_config.linear_attn_config.num_heads
mixers.kda.v_heads96config.jsonconfig.jsontext_config.linear_attn_config.num_heads
mixers.kda.k_head_dim128config.jsonconfig.jsontext_config.linear_attn_config.head_dim
mixers.kda.v_head_dim128config.jsonconfig.jsontext_config.linear_attn_config.head_dim
mixers.kda.conv_kernel4config.jsonconfig.jsontext_config.linear_attn_config.short_conv_kernel_size
ffns.dense.typedensecodemodelling codeMLP block
ffns.dense.d_ff33,792config.jsonconfig.jsontext_config.intermediate_size
ffns.dense.gatedtruecodemodelling codeMLP: gated (SwiGLU/GeGLU)
ffns.moe.typemoecodemodelling codeMoE block
ffns.moe.experts896config.jsonconfig.jsontext_config.num_experts
ffns.moe.active16config.jsonconfig.jsontext_config.num_experts_per_token
ffns.moe.d_expert3,072config.jsonconfig.jsontext_config.moe_intermediate_size
ffns.moe.gatedtruecodemodelling codeexperts are gated MLPs
ffns.moe.shared2config.jsonconfig.jsontext_config.num_shared_experts
ffns.moe.d_shared3,072config.jsonconfig.jsontext_config.moe_intermediate_size
ffns.moe.latent3,584config.jsonconfig.jsontext_config.routed_expert_hidden_size
layout1× kda/dense · 2× kda/moe · 1× mla/moe · 3× kda/moe · 1× mla/moe · 3× kda/moe · 1× mla/moe · 3× kda/moe · 1× mla/moe · 3× kda/moe · 1× mla/moe · 3× kda/moe · 1× mla/moe · 3× kda/moe · 1× mla/moe · 3× kda/moe · 1× mla/moe · 3× kda/moe · 1× mla/moe · 3× kda/moe · 1× mla/moe · 3× kda/moe · 1× mla/moe · 3× kda/moe · 1× mla/moe · 3× kda/moe · 1× mla/moe · 3× kda/moe · 1× mla/moe · 3× kda/moe · 1× mla/moe · 3× kda/moe · 1× mla/moe · 3× kda/moe · 1× mla/moe · 3× kda/moe · 1× mla/moe · 3× kda/moe · 1× mla/moe · 3× kda/moe · 1× mla/moe · 3× kda/moe · 1× mla/moe · 3× kda/moe · 1× mla/moe · 3× kda/moe · 2× mla/moeconfig.jsonconfig.jsontext_config.linear_attn_config.full_attn_layers (1-based), first_k_dense_replace

Sources

Listed in the LLM Architecture Gallery checklist as “Ember-1 (2.8T)” (name only; see about).