LFM2.5 350M
Liquid AI · LFM2 · open weights
- GQA
- Short convolution
- Hybrid
- RoPE
- Pre-norm
- QK-norm
Facts and where they come from
| Released | 2026-03 | config.jsonconfig.jsonHugging Face repository creation date (api.createdAt) |
|---|---|---|
| Licence | other | config.jsonconfig.jsonREADME metadata: license |
| Total parameters | 350M | labmodel cardREADME: Number of parameters: 350M |
| Active parameters | not disclosed | not disclosed |
| Context length | 32K tokens | labmodel cardREADME: Context length: 32,768 tokens |
| Norm placement | pre | codemodelling codetransformers 5.18.0 lfm2: operator_norm and ffn_norm |
| Norm type | RMSNorm | codemodelling codetransformers 5.18.0 lfm2: operator_norm and ffn_norm |
| QK-norm | yes | codemodelling codetransformers 5.18.0 lfm2: q_layernorm and k_layernorm |
| Positional encoding | RoPE | codemodelling coderotary on the full head (default) |
| Parallel attention and MLP | no | codemodelling codetransformers 5.18.0 lfm2: operator_norm and ffn_norm |
Architecture, drawn from the data
10× Short convolution + 6× GQA 16q/8kv. Each column is one layer: its token mixer above, its feed-forward block below. Paler columns reuse another layer’s keys and values.
Modelled costs
From the cost model, batch size 1. Totals the lab states are in the table above; differences come from rounding, from what a lab counts, or from parts the model does not describe (listed on the about page).
| Parameters (modelled) | 354M |
|---|---|
| Active per token (modelled) | 354M |
| Without embeddings and output head | 287M total, 287M active |
| Published weights (Hugging Face count) | 354M |
| KV cache per token, BF16 (layers that grow with context) | 12 KiB |
| KV cache + state at 32K tokens, BF16 | 384 MiB |
| Decode FLOPs per token at 4K context | 810 MFLOP |
| Prefill FLOPs for a 4K prompt | 2.56 TFLOP |
KV cache against context
Every architecture field
| Field | Value | Source |
|---|---|---|
| d_model | 1,024 | config.jsonconfig.jsonhidden_size |
| vocab | 65,536 | config.jsonconfig.jsonvocab_size |
| tied_embeddings | true | config.jsonconfig.jsontie_embedding |
| mixers.full.type | attn | codemodelling codeattention block |
| mixers.full.heads | 16 | config.jsonconfig.jsonnum_attention_heads |
| mixers.full.kv_heads | 8 | config.jsonconfig.jsonnum_key_value_heads |
| mixers.full.head_dim | 64 | codemodelling codetransformers 5.18.0: head_dim = hidden_size / num_attention_heads |
| mixers.full.qk_norm | true | codemodelling codetransformers 5.18.0 lfm2: q_layernorm and k_layernorm |
| mixers.conv.type | conv | codemodelling codegated short convolution |
| mixers.conv.kernel | 3 | config.jsonconfig.jsonconv_L_cache |
| ffns.dense.type | dense | codemodelling codeMLP |
| ffns.dense.d_ff | 4,608 | codemodelling codetransformers 5.18.0 lfm2: block_auto_adjust_ff_dim: int(2/3 * block_ff_dim * multiplier) rounded up to block_multiple_of |
| ffns.dense.gated | true | codemodelling codeSwiGLU |
| layout | 2× conv/dense · 1× full/dense · 2× conv/dense · 1× full/dense · 2× conv/dense · 1× full/dense · 1× conv/dense · 1× full/dense · 1× conv/dense · 1× full/dense · 1× conv/dense · 1× full/dense · 1× conv/dense | config.jsonconfig.jsonlayer_types |
Sources
- config.json @ 9e6c6cc
- model card
- modelling code · transformers 5.18.0 modelling code, or the model repository's own modelling file at the pinned revision
Listed in the LLM Architecture Gallery checklist as “LFM2.5 (350M)” (name only; see about).