llm-architectures-explained

/models

Mamba 2.8B

Carnegie Mellon / Princeton · Mamba · open weights

Facts and where they come from

Released2023-12paperarXiv 2312.00752arXiv v1, December 2023
Licencenot disclosednot disclosed
Total parameters2.8Blabmodel cardmodel name mamba-2.8b
Active parametersnot disclosednot disclosed
Context lengthnot disclosednot disclosed
Norm placementprecodemodelling codetransformers 5.18.0 mamba: one norm before each block
Norm typeRMSNormcodemodelling codetransformers 5.18.0 mamba: one norm before each block
QK-normnocodemodelling codeno q/k normalisation in the attention block
Positional encodingnone; all layers: the recurrence carries ordercodemodelling codetransformers 5.18.0 mamba: no positional encoding
Parallel attention and MLPnocodemodelling codetransformers 5.18.0 mamba: one norm before each block

Architecture, drawn from the data

Mamba. Each column is one layer: its token mixer above, its feed-forward block below. Paler columns reuse another layer’s keys and values.

Mamba 2.8B: layer stack and blockslayers (64)mixer / FFNlayer 0: Mamba: inner 5120 · state 16layer 0: no FFNlayer 1: Mamba: inner 5120 · state 16layer 1: no FFNlayer 2: Mamba: inner 5120 · state 16layer 2: no FFNlayer 3: Mamba: inner 5120 · state 16layer 3: no FFNlayer 4: Mamba: inner 5120 · state 16layer 4: no FFNlayer 5: Mamba: inner 5120 · state 16layer 5: no FFNlayer 6: Mamba: inner 5120 · state 16layer 6: no FFNlayer 7: Mamba: inner 5120 · state 16layer 7: no FFNlayer 8: Mamba: inner 5120 · state 16layer 8: no FFNlayer 9: Mamba: inner 5120 · state 16layer 9: no FFNlayer 10: Mamba: inner 5120 · state 16layer 10: no FFNlayer 11: Mamba: inner 5120 · state 16layer 11: no FFNlayer 12: Mamba: inner 5120 · state 16layer 12: no FFNlayer 13: Mamba: inner 5120 · state 16layer 13: no FFNlayer 14: Mamba: inner 5120 · state 16layer 14: no FFNlayer 15: Mamba: inner 5120 · state 16layer 15: no FFNlayer 16: Mamba: inner 5120 · state 16layer 16: no FFNlayer 17: Mamba: inner 5120 · state 16layer 17: no FFNlayer 18: Mamba: inner 5120 · state 16layer 18: no FFNlayer 19: Mamba: inner 5120 · state 16layer 19: no FFNlayer 20: Mamba: inner 5120 · state 16layer 20: no FFNlayer 21: Mamba: inner 5120 · state 16layer 21: no FFNlayer 22: Mamba: inner 5120 · state 16layer 22: no FFNlayer 23: Mamba: inner 5120 · state 16layer 23: no FFNlayer 24: Mamba: inner 5120 · state 16layer 24: no FFNlayer 25: Mamba: inner 5120 · state 16layer 25: no FFNlayer 26: Mamba: inner 5120 · state 16layer 26: no FFNlayer 27: Mamba: inner 5120 · state 16layer 27: no FFNlayer 28: Mamba: inner 5120 · state 16layer 28: no FFNlayer 29: Mamba: inner 5120 · state 16layer 29: no FFNlayer 30: Mamba: inner 5120 · state 16layer 30: no FFNlayer 31: Mamba: inner 5120 · state 16layer 31: no FFNlayer 32: Mamba: inner 5120 · state 16layer 32: no FFNlayer 33: Mamba: inner 5120 · state 16layer 33: no FFNlayer 34: Mamba: inner 5120 · state 16layer 34: no FFNlayer 35: Mamba: inner 5120 · state 16layer 35: no FFNlayer 36: Mamba: inner 5120 · state 16layer 36: no FFNlayer 37: Mamba: inner 5120 · state 16layer 37: no FFNlayer 38: Mamba: inner 5120 · state 16layer 38: no FFNlayer 39: Mamba: inner 5120 · state 16layer 39: no FFNlayer 40: Mamba: inner 5120 · state 16layer 40: no FFNlayer 41: Mamba: inner 5120 · state 16layer 41: no FFNlayer 42: Mamba: inner 5120 · state 16layer 42: no FFNlayer 43: Mamba: inner 5120 · state 16layer 43: no FFNlayer 44: Mamba: inner 5120 · state 16layer 44: no FFNlayer 45: Mamba: inner 5120 · state 16layer 45: no FFNlayer 46: Mamba: inner 5120 · state 16layer 46: no FFNlayer 47: Mamba: inner 5120 · state 16layer 47: no FFNlayer 48: Mamba: inner 5120 · state 16layer 48: no FFNlayer 49: Mamba: inner 5120 · state 16layer 49: no FFNlayer 50: Mamba: inner 5120 · state 16layer 50: no FFNlayer 51: Mamba: inner 5120 · state 16layer 51: no FFNlayer 52: Mamba: inner 5120 · state 16layer 52: no FFNlayer 53: Mamba: inner 5120 · state 16layer 53: no FFNlayer 54: Mamba: inner 5120 · state 16layer 54: no FFNlayer 55: Mamba: inner 5120 · state 16layer 55: no FFNlayer 56: Mamba: inner 5120 · state 16layer 56: no FFNlayer 57: Mamba: inner 5120 · state 16layer 57: no FFNlayer 58: Mamba: inner 5120 · state 16layer 58: no FFNlayer 59: Mamba: inner 5120 · state 16layer 59: no FFNlayer 60: Mamba: inner 5120 · state 16layer 60: no FFNlayer 61: Mamba: inner 5120 · state 16layer 61: no FFNlayer 62: Mamba: inner 5120 · state 16layer 62: no FFNlayer 63: Mamba: inner 5120 · state 16layer 63: no FFN03263× 64normMamba: inner 5120 · state 16+Mamba

Modelled costs

From the cost model, batch size 1. Totals the lab states are in the table above; differences come from rounding, from what a lab counts, or from parts the model does not describe (listed on the about page).

Parameters (modelled)2.77B
Active per token (modelled)2.77B
Without embeddings and output head2.64B total, 2.64B active
Published weights (Hugging Face count)2.77B
KV cache per token, BF16 (layers that grow with context)0 B
KV cache + state at 128K tokens, BF1611.9 MiB
Decode FLOPs per token at 4K context5.57 GFLOP
Prefill FLOPs for a 4K prompt21.7 TFLOP

KV cache against context

Mamba 2.8B: KV cache bytes against context length101001,00010,000100,0009.5 MiBcontext (tokens)KV cache + state (BF16)Mamba 2.8B

Compare with other models →

Every architecture field

FieldValueSource
d_model2,560config.jsonconfig.jsonhidden_size
vocab50,280config.jsonconfig.jsonvocab_size
tied_embeddingstruecodemodelling codetransformers 5.18.0 mamba: tied
mixers.mamba.typemamba1codemodelling codeMamba (selective SSM)
mixers.mamba.d_inner5,120config.jsonconfig.jsonintermediate_size
mixers.mamba.state16config.jsonconfig.jsonstate_size
mixers.mamba.conv_kernel4config.jsonconfig.jsonconv_kernel
mixers.mamba.dt_rank160config.jsonconfig.jsontime_step_rank
layout64× mamba/noneconfig.jsonconfig.jsonnum_hidden_layers
norms_per_layer1codemodelling codeone RMSNorm per Mamba block

Sources