llm-architectures-explained

/models

T5 11B

Google · T5 · open weights · encoder-decoder

Facts and where they come from

Released2019-10paperarXiv 1910.10683arXiv v1, October 2019
Licenceapache-2.0config.jsonconfig.jsonREADME metadata: license
Total parameters11Blabmodel cardREADME: T5-11B is the checkpoint with 11 billion parameters
Active parametersnot disclosednot disclosed
Context length512 tokensconfig.jsonconfig.jsonn_positions
Norm placementprecodemodelling codetransformers 5.18.0 t5: layer_norm before each sub-layer (T5LayerNorm, no bias or mean)
Norm typeRMSNormcodemodelling codetransformers 5.18.0 t5: layer_norm before each sub-layer (T5LayerNorm, no bias or mean)
QK-normnocodemodelling codeno q/k normalisation in the attention block
Positional encodingrelative position biascodemodelling codetransformers 5.18.0 t5: bucketed relative position bias
Parallel attention and MLPnocodemodelling codetransformers 5.18.0 t5: layer_norm before each sub-layer (T5LayerNorm, no bias or mean)

Architecture, drawn from the data

MHA 128q/128kv. Each column is one layer: its token mixer above, its feed-forward block below. Paler columns reuse another layer’s keys and values.

T5 11B: layer stack and blocksencoder (24)mixer / FFNlayer 0: MHA: 128 query / 128 KV heads · head 128layer 0: MLP: 65536layer 1: MHA: 128 query / 128 KV heads · head 128layer 1: MLP: 65536layer 2: MHA: 128 query / 128 KV heads · head 128layer 2: MLP: 65536layer 3: MHA: 128 query / 128 KV heads · head 128layer 3: MLP: 65536layer 4: MHA: 128 query / 128 KV heads · head 128layer 4: MLP: 65536layer 5: MHA: 128 query / 128 KV heads · head 128layer 5: MLP: 65536layer 6: MHA: 128 query / 128 KV heads · head 128layer 6: MLP: 65536layer 7: MHA: 128 query / 128 KV heads · head 128layer 7: MLP: 65536layer 8: MHA: 128 query / 128 KV heads · head 128layer 8: MLP: 65536layer 9: MHA: 128 query / 128 KV heads · head 128layer 9: MLP: 65536layer 10: MHA: 128 query / 128 KV heads · head 128layer 10: MLP: 65536layer 11: MHA: 128 query / 128 KV heads · head 128layer 11: MLP: 65536layer 12: MHA: 128 query / 128 KV heads · head 128layer 12: MLP: 65536layer 13: MHA: 128 query / 128 KV heads · head 128layer 13: MLP: 65536layer 14: MHA: 128 query / 128 KV heads · head 128layer 14: MLP: 65536layer 15: MHA: 128 query / 128 KV heads · head 128layer 15: MLP: 65536layer 16: MHA: 128 query / 128 KV heads · head 128layer 16: MLP: 65536layer 17: MHA: 128 query / 128 KV heads · head 128layer 17: MLP: 65536layer 18: MHA: 128 query / 128 KV heads · head 128layer 18: MLP: 65536layer 19: MHA: 128 query / 128 KV heads · head 128layer 19: MLP: 65536layer 20: MHA: 128 query / 128 KV heads · head 128layer 20: MLP: 65536layer 21: MHA: 128 query / 128 KV heads · head 128layer 21: MLP: 65536layer 22: MHA: 128 query / 128 KV heads · head 128layer 22: MLP: 65536layer 23: MHA: 128 query / 128 KV heads · head 128layer 23: MLP: 6553601223decoder (24)mixer / FFNlayer 0: MHA: 128 query / 128 KV heads · head 128layer 0: MLP: 65536layer 1: MHA: 128 query / 128 KV heads · head 128layer 1: MLP: 65536layer 2: MHA: 128 query / 128 KV heads · head 128layer 2: MLP: 65536layer 3: MHA: 128 query / 128 KV heads · head 128layer 3: MLP: 65536layer 4: MHA: 128 query / 128 KV heads · head 128layer 4: MLP: 65536layer 5: MHA: 128 query / 128 KV heads · head 128layer 5: MLP: 65536layer 6: MHA: 128 query / 128 KV heads · head 128layer 6: MLP: 65536layer 7: MHA: 128 query / 128 KV heads · head 128layer 7: MLP: 65536layer 8: MHA: 128 query / 128 KV heads · head 128layer 8: MLP: 65536layer 9: MHA: 128 query / 128 KV heads · head 128layer 9: MLP: 65536layer 10: MHA: 128 query / 128 KV heads · head 128layer 10: MLP: 65536layer 11: MHA: 128 query / 128 KV heads · head 128layer 11: MLP: 65536layer 12: MHA: 128 query / 128 KV heads · head 128layer 12: MLP: 65536layer 13: MHA: 128 query / 128 KV heads · head 128layer 13: MLP: 65536layer 14: MHA: 128 query / 128 KV heads · head 128layer 14: MLP: 65536layer 15: MHA: 128 query / 128 KV heads · head 128layer 15: MLP: 65536layer 16: MHA: 128 query / 128 KV heads · head 128layer 16: MLP: 65536layer 17: MHA: 128 query / 128 KV heads · head 128layer 17: MLP: 65536layer 18: MHA: 128 query / 128 KV heads · head 128layer 18: MLP: 65536layer 19: MHA: 128 query / 128 KV heads · head 128layer 19: MLP: 65536layer 20: MHA: 128 query / 128 KV heads · head 128layer 20: MLP: 65536layer 21: MHA: 128 query / 128 KV heads · head 128layer 21: MLP: 65536layer 22: MHA: 128 query / 128 KV heads · head 128layer 22: MLP: 65536layer 23: MHA: 128 query / 128 KV heads · head 128layer 23: MLP: 6553601223× 24normMHA: 128 query / 128 KV heads · head 128+normMLP: 65536+full attentiondense FFN

Modelled costs

From the cost model, batch size 1. Totals the lab states are in the table above; differences come from rounding, from what a lab counts, or from parts the model does not describe (listed on the about page).

Parameters (modelled)11.3B
Active per token (modelled)11.3B
Without embeddings and output head11.3B total, 11.3B active
KV cache per token, BF16 (layers that grow with context)3 MiB
KV cache + state at 512 tokens, BF161.5 GiB
Decode FLOPs per token at 4K context35.5 GFLOP
Prefill FLOPs for a 4K prompt106 TFLOP

KV cache against context

T5 11B: KV cache bytes against context length101001,0009.5 MiB95 MiB950 MiB9.3 GiBcontext (tokens)KV cache + state (BF16)T5 11B

Compare with other models →

Every architecture field

FieldValueSource
d_model1,024config.jsonconfig.jsond_model
vocab32,128config.jsonconfig.jsonvocab_size
tied_embeddingstruecodemodelling codetransformers 5.18.0 t5: tie_word_embeddings default true
mixers.full.typeattncodemodelling codeattention
mixers.full.heads128config.jsonconfig.jsonnum_heads
mixers.full.kv_heads128config.jsonconfig.jsonnum_heads
mixers.full.head_dim128config.jsonconfig.jsond_kv
ffns.dense.typedensecodemodelling codeMLP
ffns.dense.d_ff65,536config.jsonconfig.jsond_ff
ffns.dense.gatedfalsecodemodelling codefeed_forward_proj
layout24× full/denseconfig.jsonconfig.jsonnum_decoder_layers
encoder_layout24× full/denseconfig.jsonconfig.jsonnum_layers
norms_per_layer2codemodelling codepre-norm RMSNorm

Sources