llm-architectures-explained

/models

BLOOM

BigScience · BLOOM · open weights

Facts and where they come from

Released2022-11paperarXiv 2211.05100arXiv v1, November 2022
Licencebigscience-bloom-rail-1.0config.jsonconfig.jsonREADME metadata: license
Total parameters176Blabmodel cardREADME: BLOOM, a 176B parameter language model
Active parametersnot disclosednot disclosed
Context lengthnot disclosednot disclosed
Norm placementprecodemodelling codetransformers 5.18.0 bloom: input_layernorm and post_attention_layernorm (pre-MLP), plus an embedding LayerNorm
Norm typeLayerNormcodemodelling codetransformers 5.18.0 bloom: input_layernorm and post_attention_layernorm (pre-MLP), plus an embedding LayerNorm
QK-normnocodemodelling codeno q/k normalisation in the attention block
Positional encodingALiBicodemodelling codetransformers 5.18.0 bloom: ALiBi attention biases
Parallel attention and MLPnocodemodelling codetransformers 5.18.0 bloom: input_layernorm and post_attention_layernorm (pre-MLP), plus an embedding LayerNorm

Architecture, drawn from the data

MHA 112q/112kv. Each column is one layer: its token mixer above, its feed-forward block below. Paler columns reuse another layer’s keys and values.

BLOOM: layer stack and blockslayers (70)mixer / FFNlayer 0: MHA: 112 query / 112 KV heads · head 128layer 0: MLP: 57344layer 1: MHA: 112 query / 112 KV heads · head 128layer 1: MLP: 57344layer 2: MHA: 112 query / 112 KV heads · head 128layer 2: MLP: 57344layer 3: MHA: 112 query / 112 KV heads · head 128layer 3: MLP: 57344layer 4: MHA: 112 query / 112 KV heads · head 128layer 4: MLP: 57344layer 5: MHA: 112 query / 112 KV heads · head 128layer 5: MLP: 57344layer 6: MHA: 112 query / 112 KV heads · head 128layer 6: MLP: 57344layer 7: MHA: 112 query / 112 KV heads · head 128layer 7: MLP: 57344layer 8: MHA: 112 query / 112 KV heads · head 128layer 8: MLP: 57344layer 9: MHA: 112 query / 112 KV heads · head 128layer 9: MLP: 57344layer 10: MHA: 112 query / 112 KV heads · head 128layer 10: MLP: 57344layer 11: MHA: 112 query / 112 KV heads · head 128layer 11: MLP: 57344layer 12: MHA: 112 query / 112 KV heads · head 128layer 12: MLP: 57344layer 13: MHA: 112 query / 112 KV heads · head 128layer 13: MLP: 57344layer 14: MHA: 112 query / 112 KV heads · head 128layer 14: MLP: 57344layer 15: MHA: 112 query / 112 KV heads · head 128layer 15: MLP: 57344layer 16: MHA: 112 query / 112 KV heads · head 128layer 16: MLP: 57344layer 17: MHA: 112 query / 112 KV heads · head 128layer 17: MLP: 57344layer 18: MHA: 112 query / 112 KV heads · head 128layer 18: MLP: 57344layer 19: MHA: 112 query / 112 KV heads · head 128layer 19: MLP: 57344layer 20: MHA: 112 query / 112 KV heads · head 128layer 20: MLP: 57344layer 21: MHA: 112 query / 112 KV heads · head 128layer 21: MLP: 57344layer 22: MHA: 112 query / 112 KV heads · head 128layer 22: MLP: 57344layer 23: MHA: 112 query / 112 KV heads · head 128layer 23: MLP: 57344layer 24: MHA: 112 query / 112 KV heads · head 128layer 24: MLP: 57344layer 25: MHA: 112 query / 112 KV heads · head 128layer 25: MLP: 57344layer 26: MHA: 112 query / 112 KV heads · head 128layer 26: MLP: 57344layer 27: MHA: 112 query / 112 KV heads · head 128layer 27: MLP: 57344layer 28: MHA: 112 query / 112 KV heads · head 128layer 28: MLP: 57344layer 29: MHA: 112 query / 112 KV heads · head 128layer 29: MLP: 57344layer 30: MHA: 112 query / 112 KV heads · head 128layer 30: MLP: 57344layer 31: MHA: 112 query / 112 KV heads · head 128layer 31: MLP: 57344layer 32: MHA: 112 query / 112 KV heads · head 128layer 32: MLP: 57344layer 33: MHA: 112 query / 112 KV heads · head 128layer 33: MLP: 57344layer 34: MHA: 112 query / 112 KV heads · head 128layer 34: MLP: 57344layer 35: MHA: 112 query / 112 KV heads · head 128layer 35: MLP: 57344layer 36: MHA: 112 query / 112 KV heads · head 128layer 36: MLP: 57344layer 37: MHA: 112 query / 112 KV heads · head 128layer 37: MLP: 57344layer 38: MHA: 112 query / 112 KV heads · head 128layer 38: MLP: 57344layer 39: MHA: 112 query / 112 KV heads · head 128layer 39: MLP: 57344layer 40: MHA: 112 query / 112 KV heads · head 128layer 40: MLP: 57344layer 41: MHA: 112 query / 112 KV heads · head 128layer 41: MLP: 57344layer 42: MHA: 112 query / 112 KV heads · head 128layer 42: MLP: 57344layer 43: MHA: 112 query / 112 KV heads · head 128layer 43: MLP: 57344layer 44: MHA: 112 query / 112 KV heads · head 128layer 44: MLP: 57344layer 45: MHA: 112 query / 112 KV heads · head 128layer 45: MLP: 57344layer 46: MHA: 112 query / 112 KV heads · head 128layer 46: MLP: 57344layer 47: MHA: 112 query / 112 KV heads · head 128layer 47: MLP: 57344layer 48: MHA: 112 query / 112 KV heads · head 128layer 48: MLP: 57344layer 49: MHA: 112 query / 112 KV heads · head 128layer 49: MLP: 57344layer 50: MHA: 112 query / 112 KV heads · head 128layer 50: MLP: 57344layer 51: MHA: 112 query / 112 KV heads · head 128layer 51: MLP: 57344layer 52: MHA: 112 query / 112 KV heads · head 128layer 52: MLP: 57344layer 53: MHA: 112 query / 112 KV heads · head 128layer 53: MLP: 57344layer 54: MHA: 112 query / 112 KV heads · head 128layer 54: MLP: 57344layer 55: MHA: 112 query / 112 KV heads · head 128layer 55: MLP: 57344layer 56: MHA: 112 query / 112 KV heads · head 128layer 56: MLP: 57344layer 57: MHA: 112 query / 112 KV heads · head 128layer 57: MLP: 57344layer 58: MHA: 112 query / 112 KV heads · head 128layer 58: MLP: 57344layer 59: MHA: 112 query / 112 KV heads · head 128layer 59: MLP: 57344layer 60: MHA: 112 query / 112 KV heads · head 128layer 60: MLP: 57344layer 61: MHA: 112 query / 112 KV heads · head 128layer 61: MLP: 57344layer 62: MHA: 112 query / 112 KV heads · head 128layer 62: MLP: 57344layer 63: MHA: 112 query / 112 KV heads · head 128layer 63: MLP: 57344layer 64: MHA: 112 query / 112 KV heads · head 128layer 64: MLP: 57344layer 65: MHA: 112 query / 112 KV heads · head 128layer 65: MLP: 57344layer 66: MHA: 112 query / 112 KV heads · head 128layer 66: MLP: 57344layer 67: MHA: 112 query / 112 KV heads · head 128layer 67: MLP: 57344layer 68: MHA: 112 query / 112 KV heads · head 128layer 68: MLP: 57344layer 69: MHA: 112 query / 112 KV heads · head 128layer 69: MLP: 5734403569× 70normMHA: 112 query / 112 KV heads · head 128+normMLP: 57344+full attentiondense FFN

Modelled costs

From the cost model, batch size 1. Totals the lab states are in the table above; differences come from rounding, from what a lab counts, or from parts the model does not describe (listed on the about page).

Parameters (modelled)176B
Active per token (modelled)176B
Without embeddings and output head173B total, 173B active
Published weights (Hugging Face count)176B
KV cache per token, BF16 (layers that grow with context)3.83 MiB
KV cache + state at 128K tokens, BF16490 GiB
Decode FLOPs per token at 4K context369 GFLOP
Prefill FLOPs for a 4K prompt1.45 PFLOP

KV cache against context

BLOOM: KV cache bytes against context length101001,00010,000100,0009.5 MiB95 MiB950 MiB9.3 GiB93 GiBcontext (tokens)KV cache + state (BF16)BLOOM

Compare with other models →

Every architecture field

FieldValueSource
d_model14,336config.jsonconfig.jsonn_embed
vocab250,880config.jsonconfig.jsonvocab_size
mixers.full.typeattncodemodelling codeattention
mixers.full.heads112config.jsonconfig.jsonnum_attention_heads
mixers.full.kv_heads112config.jsonconfig.jsonnum_attention_heads
mixers.full.head_dim128codemodelling coden_embed / num_attention_heads
mixers.full.biastruecodemodelling codeqkv biases
ffns.dense.typedensecodemodelling codeMLP
ffns.dense.d_ff57,344codemodelling codetransformers 5.18.0 bloom: 4 * hidden_size
ffns.dense.gatedfalsecodemodelling codeGELU MLP
ffns.dense.biastruecodemodelling codebiases
layout70× full/denseconfig.jsonconfig.jsonn_layer
tied_embeddingstruecodemodelling codetransformers 5.18.0 bloom: tied

Sources