llm-architectures-explained

/models

LongCat-Flash-Lite

Meituan · LongCat · open weights

Facts and where they come from

Released2026-01config.jsonconfig.jsonHugging Face repository creation date (api.createdAt)
Licencemitconfig.jsonconfig.jsonREADME metadata: license
Total parameters68.5Blabmodel cardREADME: a non-thinking 68.5B parameter MoE model with approximately 3B activated parameters
Active parameters3Blabmodel cardREADME: approximately 3B activated
Context length320K tokensconfig.jsonconfig.jsonmax_position_embeddings
Norm placementnot disclosednot disclosednot checked in the modelling code
Norm typenot disclosednot disclosednot checked in the modelling code
QK-normnocodemodelling codeno q/k normalisation in the attention block (MLA normalises its latent vectors, which is not QK-norm)
Positional encodingRoPE on 33.3% of each headconfig.jsonconfig.jsonqk_rope_head_dim / (qk_nope_head_dim + qk_rope_head_dim): decoupled RoPE

Architecture, drawn from the data

MLA 32h, latent 512. Each column is one layer: its token mixer above, its feed-forward block below. Paler columns reuse another layer’s keys and values.

LongCat-Flash-Lite: layer stack and blockslayers (42)mixer / FFNlayer 0: MLA: 32 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 0: Gated MLP: 6144layer 1: MLA: 32 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 1: Gated MLP: 6144layer 2: no mixerlayer 2: MoE: 256 experts, 12 active · expert 1024layer 3: MLA: 32 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 3: Gated MLP: 6144layer 4: MLA: 32 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 4: Gated MLP: 6144layer 5: no mixerlayer 5: MoE: 256 experts, 12 active · expert 1024layer 6: MLA: 32 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 6: Gated MLP: 6144layer 7: MLA: 32 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 7: Gated MLP: 6144layer 8: no mixerlayer 8: MoE: 256 experts, 12 active · expert 1024layer 9: MLA: 32 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 9: Gated MLP: 6144layer 10: MLA: 32 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 10: Gated MLP: 6144layer 11: no mixerlayer 11: MoE: 256 experts, 12 active · expert 1024layer 12: MLA: 32 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 12: Gated MLP: 6144layer 13: MLA: 32 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 13: Gated MLP: 6144layer 14: no mixerlayer 14: MoE: 256 experts, 12 active · expert 1024layer 15: MLA: 32 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 15: Gated MLP: 6144layer 16: MLA: 32 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 16: Gated MLP: 6144layer 17: no mixerlayer 17: MoE: 256 experts, 12 active · expert 1024layer 18: MLA: 32 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 18: Gated MLP: 6144layer 19: MLA: 32 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 19: Gated MLP: 6144layer 20: no mixerlayer 20: MoE: 256 experts, 12 active · expert 1024layer 21: MLA: 32 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 21: Gated MLP: 6144layer 22: MLA: 32 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 22: Gated MLP: 6144layer 23: no mixerlayer 23: MoE: 256 experts, 12 active · expert 1024layer 24: MLA: 32 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 24: Gated MLP: 6144layer 25: MLA: 32 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 25: Gated MLP: 6144layer 26: no mixerlayer 26: MoE: 256 experts, 12 active · expert 1024layer 27: MLA: 32 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 27: Gated MLP: 6144layer 28: MLA: 32 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 28: Gated MLP: 6144layer 29: no mixerlayer 29: MoE: 256 experts, 12 active · expert 1024layer 30: MLA: 32 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 30: Gated MLP: 6144layer 31: MLA: 32 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 31: Gated MLP: 6144layer 32: no mixerlayer 32: MoE: 256 experts, 12 active · expert 1024layer 33: MLA: 32 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 33: Gated MLP: 6144layer 34: MLA: 32 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 34: Gated MLP: 6144layer 35: no mixerlayer 35: MoE: 256 experts, 12 active · expert 1024layer 36: MLA: 32 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 36: Gated MLP: 6144layer 37: MLA: 32 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 37: Gated MLP: 6144layer 38: no mixerlayer 38: MoE: 256 experts, 12 active · expert 1024layer 39: MLA: 32 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 39: Gated MLP: 6144layer 40: MLA: 32 heads · KV latent 512 + RoPE 64 · Q latent 1536layer 40: Gated MLP: 6144layer 41: no mixerlayer 41: MoE: 256 experts, 12 active · expert 102402141× 28MLA: 32 heads · KV latent 512 + RoPE 64 · Q latent 1536+Gated MLP: 6144+× 14MoE: 256 experts, 12 active · expert 1024+MLAdense FFNMoE FFN

Modelled costs

From the cost model, batch size 1. Totals the lab states are in the table above; differences come from rounding, from what a lab counts, or from parts the model does not describe (listed on the about page).

Parameters (modelled)68.5B
Active per token (modelled)36.3B
Without embeddings and output head36.3B total, 4.1B active
Published weights (Hugging Face count)69.1B
KV cache per token, BF16 (layers that grow with context)31.5 KiB
KV cache + state at 320K tokens, BF169.84 GiB
Decode FLOPs per token at 4K context11.3 GFLOP
Prefill FLOPs for a 4K prompt38.4 TFLOP

KV cache against context

LongCat-Flash-Lite: KV cache bytes against context length101001,00010,000100,00098 KiB980 KiB9.5 MiB95 MiB950 MiBcontext (tokens)KV cache + state (BF16)LongCat-Flash-Lite

Compare with other models →

Every architecture field

FieldValueSource
d_model3,072config.jsonconfig.jsonhidden_size
vocab131,072config.jsonconfig.jsonvocab_size
tied_embeddingsfalsecodemodelling codetransformers 5.18.0: PretrainedConfig.tie_word_embeddings defaultnot in config
mixers.mla.typemlacodemodelling codemulti-head latent attention
mixers.mla.heads32config.jsonconfig.jsonnum_attention_heads
mixers.mla.q_lora_rank1,536config.jsonconfig.jsonq_lora_rank
mixers.mla.kv_lora_rank512config.jsonconfig.jsonkv_lora_rank
mixers.mla.qk_nope128config.jsonconfig.jsonqk_nope_head_dim
mixers.mla.qk_rope64config.jsonconfig.jsonqk_rope_head_dim
mixers.mla.v_head_dim128config.jsonconfig.jsonv_head_dim
ffns.dense.typedensecodemodelling codeMLP block
ffns.dense.d_ff6,144config.jsonconfig.jsonffn_hidden_size
ffns.dense.gatedtruecodemodelling codeMLP: gated (SwiGLU/GeGLU)
ffns.moe.typemoecodemodelling codeMoE block
ffns.moe.experts256config.jsonconfig.jsonn_routed_experts
ffns.moe.active12config.jsonconfig.jsonmoe_topk
ffns.moe.d_expert1,024config.jsonconfig.jsonexpert_ffn_hidden_size
ffns.moe.gatedtruecodemodelling codeexperts are gated MLPs
layout2× mla/dense · 1× none/moe · 2× mla/dense · 1× none/moe · 2× mla/dense · 1× none/moe · 2× mla/dense · 1× none/moe · 2× mla/dense · 1× none/moe · 2× mla/dense · 1× none/moe · 2× mla/dense · 1× none/moe · 2× mla/dense · 1× none/moe · 2× mla/dense · 1× none/moe · 2× mla/dense · 1× none/moe · 2× mla/dense · 1× none/moe · 2× mla/dense · 1× none/moe · 2× mla/dense · 1× none/moe · 2× mla/dense · 1× none/moecodemodelling codetransformers 5.18.0 longcat_flash: each of num_layers layers has 2 MLA blocks, 2 dense MLPs and 1 MoE block
extra_embedding_params31,406,948,352codemodelling coden-gram embedding table: vocab_size x ngram_vocab_size_ratio x hidden_size (approximate)

Sources

Listed in the LLM Architecture Gallery checklist as “LongCat-Flash-Lite (68.5B-A3B)” (name only; see about).