llm-architectures-explained

/models

Laguna S 2.1

Poolside · Laguna · open weights

Facts and where they come from

Released2026-07config.jsonconfig.jsonHugging Face repository creation date (api.createdAt)
Licenceopenmdw-1.1config.jsonconfig.jsonREADME metadata: license
Total parameters118Blabmodel cardREADME: 118B total, ~8B activated per token
Active parameters8Blabmodel cardREADME: ~8B activated
Context length1M tokenslabmodel cardREADME: 1,048,576-token context window
Norm placementprecodemodelling codetransformers 5.18.0 / repo modelling code for laguna: input_layernorm before attention, post_attention_layernorm before the MLP
Norm typeRMSNormcodemodelling codetransformers 5.18.0 / repo modelling code for laguna: input_layernorm before attention, post_attention_layernorm before the MLP
QK-normyescodemodelling coderepo modeling_laguna.py: q/k norms
Positional encodingRoPE on 50% of each headconfig.jsonconfig.jsonrope_parameters.full_attention.partial_rotary_factor (global layers)
Parallel attention and MLPnocodemodelling codetransformers 5.18.0 / repo modelling code for laguna: input_layernorm before attention, post_attention_layernorm before the MLP

Architecture, drawn from the data

12× GQA 48q/8kv + 36× GQA 72q/8kv, window 512. Each column is one layer: its token mixer above, its feed-forward block below. Paler columns reuse another layer’s keys and values.

Laguna S 2.1: layer stack and blockslayers (48)mixer / FFNlayer 0: GQA: 48 query / 8 KV heads · head 128 · headwise output gatelayer 0: Gated MLP: 12288layer 1: GQA: 72 query / 8 KV heads · head 128 · window 512 · headwise output gatelayer 1: MoE: 256 experts, 10 active · expert 1024 · 1 sharedlayer 2: GQA: 72 query / 8 KV heads · head 128 · window 512 · headwise output gatelayer 2: MoE: 256 experts, 10 active · expert 1024 · 1 sharedlayer 3: GQA: 72 query / 8 KV heads · head 128 · window 512 · headwise output gatelayer 3: MoE: 256 experts, 10 active · expert 1024 · 1 sharedlayer 4: GQA: 48 query / 8 KV heads · head 128 · headwise output gatelayer 4: MoE: 256 experts, 10 active · expert 1024 · 1 sharedlayer 5: GQA: 72 query / 8 KV heads · head 128 · window 512 · headwise output gatelayer 5: MoE: 256 experts, 10 active · expert 1024 · 1 sharedlayer 6: GQA: 72 query / 8 KV heads · head 128 · window 512 · headwise output gatelayer 6: MoE: 256 experts, 10 active · expert 1024 · 1 sharedlayer 7: GQA: 72 query / 8 KV heads · head 128 · window 512 · headwise output gatelayer 7: MoE: 256 experts, 10 active · expert 1024 · 1 sharedlayer 8: GQA: 48 query / 8 KV heads · head 128 · headwise output gatelayer 8: MoE: 256 experts, 10 active · expert 1024 · 1 sharedlayer 9: GQA: 72 query / 8 KV heads · head 128 · window 512 · headwise output gatelayer 9: MoE: 256 experts, 10 active · expert 1024 · 1 sharedlayer 10: GQA: 72 query / 8 KV heads · head 128 · window 512 · headwise output gatelayer 10: MoE: 256 experts, 10 active · expert 1024 · 1 sharedlayer 11: GQA: 72 query / 8 KV heads · head 128 · window 512 · headwise output gatelayer 11: MoE: 256 experts, 10 active · expert 1024 · 1 sharedlayer 12: GQA: 48 query / 8 KV heads · head 128 · headwise output gatelayer 12: MoE: 256 experts, 10 active · expert 1024 · 1 sharedlayer 13: GQA: 72 query / 8 KV heads · head 128 · window 512 · headwise output gatelayer 13: MoE: 256 experts, 10 active · expert 1024 · 1 sharedlayer 14: GQA: 72 query / 8 KV heads · head 128 · window 512 · headwise output gatelayer 14: MoE: 256 experts, 10 active · expert 1024 · 1 sharedlayer 15: GQA: 72 query / 8 KV heads · head 128 · window 512 · headwise output gatelayer 15: MoE: 256 experts, 10 active · expert 1024 · 1 sharedlayer 16: GQA: 48 query / 8 KV heads · head 128 · headwise output gatelayer 16: MoE: 256 experts, 10 active · expert 1024 · 1 sharedlayer 17: GQA: 72 query / 8 KV heads · head 128 · window 512 · headwise output gatelayer 17: MoE: 256 experts, 10 active · expert 1024 · 1 sharedlayer 18: GQA: 72 query / 8 KV heads · head 128 · window 512 · headwise output gatelayer 18: MoE: 256 experts, 10 active · expert 1024 · 1 sharedlayer 19: GQA: 72 query / 8 KV heads · head 128 · window 512 · headwise output gatelayer 19: MoE: 256 experts, 10 active · expert 1024 · 1 sharedlayer 20: GQA: 48 query / 8 KV heads · head 128 · headwise output gatelayer 20: MoE: 256 experts, 10 active · expert 1024 · 1 sharedlayer 21: GQA: 72 query / 8 KV heads · head 128 · window 512 · headwise output gatelayer 21: MoE: 256 experts, 10 active · expert 1024 · 1 sharedlayer 22: GQA: 72 query / 8 KV heads · head 128 · window 512 · headwise output gatelayer 22: MoE: 256 experts, 10 active · expert 1024 · 1 sharedlayer 23: GQA: 72 query / 8 KV heads · head 128 · window 512 · headwise output gatelayer 23: MoE: 256 experts, 10 active · expert 1024 · 1 sharedlayer 24: GQA: 48 query / 8 KV heads · head 128 · headwise output gatelayer 24: MoE: 256 experts, 10 active · expert 1024 · 1 sharedlayer 25: GQA: 72 query / 8 KV heads · head 128 · window 512 · headwise output gatelayer 25: MoE: 256 experts, 10 active · expert 1024 · 1 sharedlayer 26: GQA: 72 query / 8 KV heads · head 128 · window 512 · headwise output gatelayer 26: MoE: 256 experts, 10 active · expert 1024 · 1 sharedlayer 27: GQA: 72 query / 8 KV heads · head 128 · window 512 · headwise output gatelayer 27: MoE: 256 experts, 10 active · expert 1024 · 1 sharedlayer 28: GQA: 48 query / 8 KV heads · head 128 · headwise output gatelayer 28: MoE: 256 experts, 10 active · expert 1024 · 1 sharedlayer 29: GQA: 72 query / 8 KV heads · head 128 · window 512 · headwise output gatelayer 29: MoE: 256 experts, 10 active · expert 1024 · 1 sharedlayer 30: GQA: 72 query / 8 KV heads · head 128 · window 512 · headwise output gatelayer 30: MoE: 256 experts, 10 active · expert 1024 · 1 sharedlayer 31: GQA: 72 query / 8 KV heads · head 128 · window 512 · headwise output gatelayer 31: MoE: 256 experts, 10 active · expert 1024 · 1 sharedlayer 32: GQA: 48 query / 8 KV heads · head 128 · headwise output gatelayer 32: MoE: 256 experts, 10 active · expert 1024 · 1 sharedlayer 33: GQA: 72 query / 8 KV heads · head 128 · window 512 · headwise output gatelayer 33: MoE: 256 experts, 10 active · expert 1024 · 1 sharedlayer 34: GQA: 72 query / 8 KV heads · head 128 · window 512 · headwise output gatelayer 34: MoE: 256 experts, 10 active · expert 1024 · 1 sharedlayer 35: GQA: 72 query / 8 KV heads · head 128 · window 512 · headwise output gatelayer 35: MoE: 256 experts, 10 active · expert 1024 · 1 sharedlayer 36: GQA: 48 query / 8 KV heads · head 128 · headwise output gatelayer 36: MoE: 256 experts, 10 active · expert 1024 · 1 sharedlayer 37: GQA: 72 query / 8 KV heads · head 128 · window 512 · headwise output gatelayer 37: MoE: 256 experts, 10 active · expert 1024 · 1 sharedlayer 38: GQA: 72 query / 8 KV heads · head 128 · window 512 · headwise output gatelayer 38: MoE: 256 experts, 10 active · expert 1024 · 1 sharedlayer 39: GQA: 72 query / 8 KV heads · head 128 · window 512 · headwise output gatelayer 39: MoE: 256 experts, 10 active · expert 1024 · 1 sharedlayer 40: GQA: 48 query / 8 KV heads · head 128 · headwise output gatelayer 40: MoE: 256 experts, 10 active · expert 1024 · 1 sharedlayer 41: GQA: 72 query / 8 KV heads · head 128 · window 512 · headwise output gatelayer 41: MoE: 256 experts, 10 active · expert 1024 · 1 sharedlayer 42: GQA: 72 query / 8 KV heads · head 128 · window 512 · headwise output gatelayer 42: MoE: 256 experts, 10 active · expert 1024 · 1 sharedlayer 43: GQA: 72 query / 8 KV heads · head 128 · window 512 · headwise output gatelayer 43: MoE: 256 experts, 10 active · expert 1024 · 1 sharedlayer 44: GQA: 48 query / 8 KV heads · head 128 · headwise output gatelayer 44: MoE: 256 experts, 10 active · expert 1024 · 1 sharedlayer 45: GQA: 72 query / 8 KV heads · head 128 · window 512 · headwise output gatelayer 45: MoE: 256 experts, 10 active · expert 1024 · 1 sharedlayer 46: GQA: 72 query / 8 KV heads · head 128 · window 512 · headwise output gatelayer 46: MoE: 256 experts, 10 active · expert 1024 · 1 sharedlayer 47: GQA: 72 query / 8 KV heads · head 128 · window 512 · headwise output gatelayer 47: MoE: 256 experts, 10 active · expert 1024 · 1 shared02447× 36normGQA: 72 query / 8 KV heads · head 128 · window 512 · headwise output gate+normMoE: 256 experts, 10 active · expert 1024 · 1 shared+× 11normGQA: 48 query / 8 KV heads · head 128 · headwise output gate+normMoE: 256 experts, 10 active · expert 1024 · 1 shared+× 1normGQA: 48 query / 8 KV heads · head 128 · headwise output gate+normGated MLP: 12288+full attentionsliding windowdense FFNMoE FFN

Modelled costs

From the cost model, batch size 1. Totals the lab states are in the table above; differences come from rounding, from what a lab counts, or from parts the model does not describe (listed on the about page).

Parameters (modelled)118B
Active per token (modelled)8.45B
Without embeddings and output head117B total, 7.83B active
Published weights (Hugging Face count)118B
KV cache per token, BF16 (layers that grow with context)48 KiB
KV cache + state at 1M tokens, BF1648.1 GiB
Decode FLOPs per token at 4K context18.2 GFLOP
Prefill FLOPs for a 4K prompt69.2 TFLOP

KV cache against context

Laguna S 2.1: KV cache bytes against context length101001,00010,000100,0001,000,000980 KiB9.5 MiB95 MiB950 MiB9.3 GiBcontext (tokens)KV cache + state (BF16)Laguna S 2.1

Compare with other models →

Every architecture field

FieldValueSource
d_model3,072config.jsonconfig.jsonhidden_size
vocab100,352config.jsonconfig.jsonvocab_size
tied_embeddingsfalseconfig.jsonconfig.jsontie_word_embeddings
mixers.full.typeattncodemodelling codeattention block
mixers.full.heads48config.jsonconfig.jsonnum_attention_heads_per_layer (full layers)
mixers.full.kv_heads8config.jsonconfig.jsonnum_key_value_heads
mixers.full.head_dim128config.jsonconfig.jsonhead_dim
mixers.full.gateheadwisecodemodelling codegating: per-head output gate
mixers.full.qk_normtruecodemodelling codemodeling_laguna.py: q/k norms
mixers.sliding.typeattncodemodelling codeattention block
mixers.sliding.heads72config.jsonconfig.jsonnum_attention_heads_per_layer (sliding layers)
mixers.sliding.kv_heads8config.jsonconfig.jsonnum_key_value_heads
mixers.sliding.head_dim128config.jsonconfig.jsonhead_dim
mixers.sliding.window512config.jsonconfig.jsonsliding_window
mixers.sliding.gateheadwisecodemodelling codegating: per-head output gate
mixers.sliding.qk_normtruecodemodelling codemodeling_laguna.py: q/k norms
ffns.dense.typedensecodemodelling codeMLP block
ffns.dense.d_ff12,288config.jsonconfig.jsonintermediate_size
ffns.dense.gatedtruecodemodelling codeMLP: gated (SwiGLU/GeGLU)
ffns.moe.typemoecodemodelling codeMoE block
ffns.moe.experts256config.jsonconfig.jsonnum_experts
ffns.moe.active10config.jsonconfig.jsonnum_experts_per_tok
ffns.moe.d_expert1,024config.jsonconfig.jsonmoe_intermediate_size
ffns.moe.gatedtruecodemodelling codeexperts are gated MLPs
ffns.moe.shared1codemodelling codeone shared expert of shared_expert_intermediate_size
ffns.moe.d_shared1,024config.jsonconfig.jsonshared_expert_intermediate_size
layout1× full/dense · 3× sliding/moe · 1× full/moe · 3× sliding/moe · 1× full/moe · 3× sliding/moe · 1× full/moe · 3× sliding/moe · 1× full/moe · 3× sliding/moe · 1× full/moe · 3× sliding/moe · 1× full/moe · 3× sliding/moe · 1× full/moe · 3× sliding/moe · 1× full/moe · 3× sliding/moe · 1× full/moe · 3× sliding/moe · 1× full/moe · 3× sliding/moe · 1× full/moe · 3× sliding/moeconfig.jsonconfig.jsonlayer_types, mlp_layer_types

Sources

Listed in the LLM Architecture Gallery checklist as “Laguna S 2.1 (118B)” (name only; see about).