llm-architectures-explained

/models

Inkling

Thinking Machines Lab · Inkling · open weights · multimodal (text stack modelled)

Facts and where they come from

Released2026-07config.jsonconfig.jsonHugging Face repository creation date (api.createdAt)
Licenceapache-2.0config.jsonconfig.jsonREADME metadata: license
Total parameters975Blabmodel cardREADME: 975B total, 41B active
Active parameters41Blabmodel cardREADME: 41B active
Context lengthnot disclosednot disclosed
Norm placementnot disclosednot disclosednot checked in the modelling code
Norm typenot disclosednot disclosednot checked in the modelling code
QK-normnocodemodelling codeno q/k normalisation in the attention block
Positional encodingRoPEcodemodelling coderotary on the full head (default)

Architecture, drawn from the data

55× GQA 64q/16kv, window 512 + 11× GQA 64q/8kv. Each column is one layer: its token mixer above, its feed-forward block below. Paler columns reuse another layer’s keys and values.

Inkling: layer stack and blockslayers (66)mixer / FFNlayer 0: GQA: 64 query / 16 KV heads · head 128 · window 512layer 0: Gated MLP: 24576layer 1: GQA: 64 query / 16 KV heads · head 128 · window 512layer 1: Gated MLP: 24576layer 2: GQA: 64 query / 16 KV heads · head 128 · window 512layer 2: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 3: GQA: 64 query / 16 KV heads · head 128 · window 512layer 3: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 4: GQA: 64 query / 16 KV heads · head 128 · window 512layer 4: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 5: GQA: 64 query / 8 KV heads · head 128layer 5: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 6: GQA: 64 query / 16 KV heads · head 128 · window 512layer 6: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 7: GQA: 64 query / 16 KV heads · head 128 · window 512layer 7: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 8: GQA: 64 query / 16 KV heads · head 128 · window 512layer 8: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 9: GQA: 64 query / 16 KV heads · head 128 · window 512layer 9: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 10: GQA: 64 query / 16 KV heads · head 128 · window 512layer 10: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 11: GQA: 64 query / 8 KV heads · head 128layer 11: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 12: GQA: 64 query / 16 KV heads · head 128 · window 512layer 12: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 13: GQA: 64 query / 16 KV heads · head 128 · window 512layer 13: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 14: GQA: 64 query / 16 KV heads · head 128 · window 512layer 14: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 15: GQA: 64 query / 16 KV heads · head 128 · window 512layer 15: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 16: GQA: 64 query / 16 KV heads · head 128 · window 512layer 16: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 17: GQA: 64 query / 8 KV heads · head 128layer 17: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 18: GQA: 64 query / 16 KV heads · head 128 · window 512layer 18: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 19: GQA: 64 query / 16 KV heads · head 128 · window 512layer 19: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 20: GQA: 64 query / 16 KV heads · head 128 · window 512layer 20: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 21: GQA: 64 query / 16 KV heads · head 128 · window 512layer 21: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 22: GQA: 64 query / 16 KV heads · head 128 · window 512layer 22: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 23: GQA: 64 query / 8 KV heads · head 128layer 23: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 24: GQA: 64 query / 16 KV heads · head 128 · window 512layer 24: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 25: GQA: 64 query / 16 KV heads · head 128 · window 512layer 25: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 26: GQA: 64 query / 16 KV heads · head 128 · window 512layer 26: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 27: GQA: 64 query / 16 KV heads · head 128 · window 512layer 27: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 28: GQA: 64 query / 16 KV heads · head 128 · window 512layer 28: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 29: GQA: 64 query / 8 KV heads · head 128layer 29: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 30: GQA: 64 query / 16 KV heads · head 128 · window 512layer 30: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 31: GQA: 64 query / 16 KV heads · head 128 · window 512layer 31: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 32: GQA: 64 query / 16 KV heads · head 128 · window 512layer 32: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 33: GQA: 64 query / 16 KV heads · head 128 · window 512layer 33: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 34: GQA: 64 query / 16 KV heads · head 128 · window 512layer 34: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 35: GQA: 64 query / 8 KV heads · head 128layer 35: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 36: GQA: 64 query / 16 KV heads · head 128 · window 512layer 36: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 37: GQA: 64 query / 16 KV heads · head 128 · window 512layer 37: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 38: GQA: 64 query / 16 KV heads · head 128 · window 512layer 38: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 39: GQA: 64 query / 16 KV heads · head 128 · window 512layer 39: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 40: GQA: 64 query / 16 KV heads · head 128 · window 512layer 40: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 41: GQA: 64 query / 8 KV heads · head 128layer 41: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 42: GQA: 64 query / 16 KV heads · head 128 · window 512layer 42: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 43: GQA: 64 query / 16 KV heads · head 128 · window 512layer 43: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 44: GQA: 64 query / 16 KV heads · head 128 · window 512layer 44: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 45: GQA: 64 query / 16 KV heads · head 128 · window 512layer 45: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 46: GQA: 64 query / 16 KV heads · head 128 · window 512layer 46: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 47: GQA: 64 query / 8 KV heads · head 128layer 47: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 48: GQA: 64 query / 16 KV heads · head 128 · window 512layer 48: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 49: GQA: 64 query / 16 KV heads · head 128 · window 512layer 49: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 50: GQA: 64 query / 16 KV heads · head 128 · window 512layer 50: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 51: GQA: 64 query / 16 KV heads · head 128 · window 512layer 51: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 52: GQA: 64 query / 16 KV heads · head 128 · window 512layer 52: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 53: GQA: 64 query / 8 KV heads · head 128layer 53: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 54: GQA: 64 query / 16 KV heads · head 128 · window 512layer 54: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 55: GQA: 64 query / 16 KV heads · head 128 · window 512layer 55: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 56: GQA: 64 query / 16 KV heads · head 128 · window 512layer 56: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 57: GQA: 64 query / 16 KV heads · head 128 · window 512layer 57: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 58: GQA: 64 query / 16 KV heads · head 128 · window 512layer 58: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 59: GQA: 64 query / 8 KV heads · head 128layer 59: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 60: GQA: 64 query / 16 KV heads · head 128 · window 512layer 60: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 61: GQA: 64 query / 16 KV heads · head 128 · window 512layer 61: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 62: GQA: 64 query / 16 KV heads · head 128 · window 512layer 62: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 63: GQA: 64 query / 16 KV heads · head 128 · window 512layer 63: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 64: GQA: 64 query / 16 KV heads · head 128 · window 512layer 64: MoE: 256 experts, 6 active · expert 3072 · 2 sharedlayer 65: GQA: 64 query / 8 KV heads · head 128layer 65: MoE: 256 experts, 6 active · expert 3072 · 2 shared03365× 53GQA: 64 query / 16 KV heads · head 128 · window 512+MoE: 256 experts, 6 active · expert 3072 · 2 shared+× 11GQA: 64 query / 8 KV heads · head 128+MoE: 256 experts, 6 active · expert 3072 · 2 shared+× 2GQA: 64 query / 16 KV heads · head 128 · window 512+Gated MLP: 24576+sliding windowfull attentiondense FFNMoE FFN

Modelled costs

From the cost model, batch size 1. Totals the lab states are in the table above; differences come from rounding, from what a lab counts, or from parts the model does not describe (listed on the about page).

Parameters (modelled)947B
Active per token (modelled)40.6B
Without embeddings and output head944B total, 38.2B active
Multi-token-prediction layers (extra)118B
Published weights (Hugging Face count)952B
KV cache per token, BF16 (layers that grow with context)44 KiB
KV cache + state at 128K tokens, BF165.71 GiB
Decode FLOPs per token at 4K context81.2 GFLOP
Prefill FLOPs for a 4K prompt319 TFLOP

KV cache against context

Inkling: KV cache bytes against context length101001,00010,000100,000980 KiB9.5 MiB95 MiB950 MiBcontext (tokens)KV cache + state (BF16)Inkling

Compare with other models →

Every architecture field

FieldValueSource
d_model6,144config.jsonconfig.jsontext_config.hidden_size
vocab201,024config.jsonconfig.jsontext_config.vocab_size
tied_embeddingsfalsecodemodelling codetransformers 5.18.0: PretrainedConfig.tie_word_embeddings defaultnot in config
mixers.full.typeattncodemodelling codeattention block
mixers.full.heads64config.jsonconfig.jsontext_config.num_attention_heads
mixers.full.kv_heads8config.jsonconfig.jsontext_config.num_key_value_heads
mixers.full.head_dim128config.jsonconfig.jsontext_config.head_dim
mixers.sliding.typeattncodemodelling codeattention block
mixers.sliding.heads64config.jsonconfig.jsontext_config.swa_num_attention_heads
mixers.sliding.kv_heads16config.jsonconfig.jsontext_config.swa_num_key_value_heads
mixers.sliding.head_dim128config.jsonconfig.jsontext_config.swa_head_dim
mixers.sliding.window512config.jsonconfig.jsontext_config.sliding_window_size
ffns.dense.typedensecodemodelling codeMLP block
ffns.dense.d_ff24,576config.jsonconfig.jsontext_config.dense_intermediate_size
ffns.dense.gatedtruecodemodelling codeMLP: gated (SwiGLU/GeGLU)
ffns.moe.typemoecodemodelling codeMoE block
ffns.moe.experts256config.jsonconfig.jsontext_config.n_routed_experts
ffns.moe.active6config.jsonconfig.jsontext_config.num_experts_per_tok
ffns.moe.d_expert3,072config.jsonconfig.jsontext_config.intermediate_size
ffns.moe.gatedtruecodemodelling codeexperts are gated MLPs
ffns.moe.shared2config.jsonconfig.jsontext_config.n_shared_experts
ffns.moe.d_shared3,072config.jsonconfig.jsontext_config.intermediate_size
layout2× sliding/dense · 3× sliding/moe · 1× full/moe · 5× sliding/moe · 1× full/moe · 5× sliding/moe · 1× full/moe · 5× sliding/moe · 1× full/moe · 5× sliding/moe · 1× full/moe · 5× sliding/moe · 1× full/moe · 5× sliding/moe · 1× full/moe · 5× sliding/moe · 1× full/moe · 5× sliding/moe · 1× full/moe · 5× sliding/moe · 1× full/moe · 5× sliding/moe · 1× full/moeconfig.jsonconfig.jsontext_config.local_layer_ids, dense_mlp_idx
mtp_layers8config.jsonconfig.jsonmtp_config.num_nextn_predict_layers

Sources

Listed in the LLM Architecture Gallery checklist as “Inkling (975B)” (name only; see about).