llm-architectures-explained

/models

Mamba-2 2.7B

Carnegie Mellon / Princeton · Mamba · open weights

Facts and where they come from

Released2024-05config.jsonconfig.jsonHugging Face repository creation date (api.createdAt)
Licenceapache-2.0config.jsonconfig.jsonREADME metadata: license
Total parameters2.7Blabmodel cardmodel name mamba2-2.7b
Active parametersnot disclosednot disclosed
Context lengthnot disclosednot disclosed
Norm placementprecodemodelling codemamba_ssm: one norm before each block
Norm typeRMSNormcodemodelling codemamba_ssm: one norm before each block
QK-normnocodemodelling codeno q/k normalisation in the attention block
Positional encodingnone; all layers: the recurrence carries ordercodemodelling codetransformers 5.18.0 mamba2_ssm: no positional encoding
Parallel attention and MLPnocodemodelling codemamba_ssm: one norm before each block

Architecture, drawn from the data

Mamba-2. Each column is one layer: its token mixer above, its feed-forward block below. Paler columns reuse another layer’s keys and values.

Mamba-2 2.7B: layer stack and blockslayers (64)mixer / FFNlayer 0: Mamba-2: 80 heads × 64 · state 128layer 0: no FFNlayer 1: Mamba-2: 80 heads × 64 · state 128layer 1: no FFNlayer 2: Mamba-2: 80 heads × 64 · state 128layer 2: no FFNlayer 3: Mamba-2: 80 heads × 64 · state 128layer 3: no FFNlayer 4: Mamba-2: 80 heads × 64 · state 128layer 4: no FFNlayer 5: Mamba-2: 80 heads × 64 · state 128layer 5: no FFNlayer 6: Mamba-2: 80 heads × 64 · state 128layer 6: no FFNlayer 7: Mamba-2: 80 heads × 64 · state 128layer 7: no FFNlayer 8: Mamba-2: 80 heads × 64 · state 128layer 8: no FFNlayer 9: Mamba-2: 80 heads × 64 · state 128layer 9: no FFNlayer 10: Mamba-2: 80 heads × 64 · state 128layer 10: no FFNlayer 11: Mamba-2: 80 heads × 64 · state 128layer 11: no FFNlayer 12: Mamba-2: 80 heads × 64 · state 128layer 12: no FFNlayer 13: Mamba-2: 80 heads × 64 · state 128layer 13: no FFNlayer 14: Mamba-2: 80 heads × 64 · state 128layer 14: no FFNlayer 15: Mamba-2: 80 heads × 64 · state 128layer 15: no FFNlayer 16: Mamba-2: 80 heads × 64 · state 128layer 16: no FFNlayer 17: Mamba-2: 80 heads × 64 · state 128layer 17: no FFNlayer 18: Mamba-2: 80 heads × 64 · state 128layer 18: no FFNlayer 19: Mamba-2: 80 heads × 64 · state 128layer 19: no FFNlayer 20: Mamba-2: 80 heads × 64 · state 128layer 20: no FFNlayer 21: Mamba-2: 80 heads × 64 · state 128layer 21: no FFNlayer 22: Mamba-2: 80 heads × 64 · state 128layer 22: no FFNlayer 23: Mamba-2: 80 heads × 64 · state 128layer 23: no FFNlayer 24: Mamba-2: 80 heads × 64 · state 128layer 24: no FFNlayer 25: Mamba-2: 80 heads × 64 · state 128layer 25: no FFNlayer 26: Mamba-2: 80 heads × 64 · state 128layer 26: no FFNlayer 27: Mamba-2: 80 heads × 64 · state 128layer 27: no FFNlayer 28: Mamba-2: 80 heads × 64 · state 128layer 28: no FFNlayer 29: Mamba-2: 80 heads × 64 · state 128layer 29: no FFNlayer 30: Mamba-2: 80 heads × 64 · state 128layer 30: no FFNlayer 31: Mamba-2: 80 heads × 64 · state 128layer 31: no FFNlayer 32: Mamba-2: 80 heads × 64 · state 128layer 32: no FFNlayer 33: Mamba-2: 80 heads × 64 · state 128layer 33: no FFNlayer 34: Mamba-2: 80 heads × 64 · state 128layer 34: no FFNlayer 35: Mamba-2: 80 heads × 64 · state 128layer 35: no FFNlayer 36: Mamba-2: 80 heads × 64 · state 128layer 36: no FFNlayer 37: Mamba-2: 80 heads × 64 · state 128layer 37: no FFNlayer 38: Mamba-2: 80 heads × 64 · state 128layer 38: no FFNlayer 39: Mamba-2: 80 heads × 64 · state 128layer 39: no FFNlayer 40: Mamba-2: 80 heads × 64 · state 128layer 40: no FFNlayer 41: Mamba-2: 80 heads × 64 · state 128layer 41: no FFNlayer 42: Mamba-2: 80 heads × 64 · state 128layer 42: no FFNlayer 43: Mamba-2: 80 heads × 64 · state 128layer 43: no FFNlayer 44: Mamba-2: 80 heads × 64 · state 128layer 44: no FFNlayer 45: Mamba-2: 80 heads × 64 · state 128layer 45: no FFNlayer 46: Mamba-2: 80 heads × 64 · state 128layer 46: no FFNlayer 47: Mamba-2: 80 heads × 64 · state 128layer 47: no FFNlayer 48: Mamba-2: 80 heads × 64 · state 128layer 48: no FFNlayer 49: Mamba-2: 80 heads × 64 · state 128layer 49: no FFNlayer 50: Mamba-2: 80 heads × 64 · state 128layer 50: no FFNlayer 51: Mamba-2: 80 heads × 64 · state 128layer 51: no FFNlayer 52: Mamba-2: 80 heads × 64 · state 128layer 52: no FFNlayer 53: Mamba-2: 80 heads × 64 · state 128layer 53: no FFNlayer 54: Mamba-2: 80 heads × 64 · state 128layer 54: no FFNlayer 55: Mamba-2: 80 heads × 64 · state 128layer 55: no FFNlayer 56: Mamba-2: 80 heads × 64 · state 128layer 56: no FFNlayer 57: Mamba-2: 80 heads × 64 · state 128layer 57: no FFNlayer 58: Mamba-2: 80 heads × 64 · state 128layer 58: no FFNlayer 59: Mamba-2: 80 heads × 64 · state 128layer 59: no FFNlayer 60: Mamba-2: 80 heads × 64 · state 128layer 60: no FFNlayer 61: Mamba-2: 80 heads × 64 · state 128layer 61: no FFNlayer 62: Mamba-2: 80 heads × 64 · state 128layer 62: no FFNlayer 63: Mamba-2: 80 heads × 64 · state 128layer 63: no FFN03263× 64normMamba-2: 80 heads × 64 · state 128+Mamba-2

Modelled costs

From the cost model, batch size 1. Totals the lab states are in the table above; differences come from rounding, from what a lab counts, or from parts the model does not describe (listed on the about page).

Parameters (modelled)2.7B
Active per token (modelled)2.7B
Without embeddings and output head2.57B total, 2.57B active
KV cache per token, BF16 (layers that grow with context)0 B
KV cache + state at 128K tokens, BF1682 MiB
Decode FLOPs per token at 4K context5.66 GFLOP
Prefill FLOPs for a 4K prompt22.1 TFLOP

KV cache against context

Mamba-2 2.7B: KV cache bytes against context length101001,00010,000100,00095 MiBcontext (tokens)KV cache + state (BF16)Mamba-2 2.7B

Compare with other models →

Every architecture field

FieldValueSource
d_model2,560config.jsonconfig.jsond_model
vocab50,277config.jsonconfig.jsonvocab_size
tied_embeddingstrueconfig.jsonconfig.jsontie_embeddings
mixers.mamba.typemamba2codemodelling codeMamba-2
mixers.mamba.heads80codemodelling codemamba_ssm Mamba2: d_inner = expand(2) * d_model; nheads = d_inner / headdim(64)
mixers.mamba.head_dim64codemodelling codemamba_ssm Mamba2 default headdim 64
mixers.mamba.state128codemodelling codemamba_ssm Mamba2 default d_state 128
mixers.mamba.groups1codemodelling codemamba_ssm Mamba2 default ngroups 1
mixers.mamba.conv_kernel4codemodelling codemamba_ssm Mamba2 default d_conv 4
layout64× mamba/noneconfig.jsonconfig.jsonn_layer
norms_per_layer1codemodelling codeone RMSNorm per block

Sources