llm-architectures-explained

/models

OPT-66B

Meta · OPT · open weights

Facts and where they come from

Released2022-05paperarXiv 2205.01068arXiv v1, May 2022
Licenceotherconfig.jsonconfig.jsonREADME metadata: license
Total parameters66Blabmodel cardmodel name opt-66b
Active parametersnot disclosednot disclosed
Context length2K tokensconfig.jsonconfig.jsonmax_position_embeddings
Norm placementprecodemodelling codetransformers 5.18.0 opt: do_layer_norm_before: true
Norm typeLayerNormcodemodelling codetransformers 5.18.0 opt: do_layer_norm_before: true
QK-normnocodemodelling codeno q/k normalisation in the attention block
Positional encodinglearned absolutecodemodelling codetransformers 5.18.0 opt: learned absolute position embeddings
Parallel attention and MLPnocodemodelling codetransformers 5.18.0 opt: do_layer_norm_before: true

Architecture, drawn from the data

MHA 72q/72kv. Each column is one layer: its token mixer above, its feed-forward block below. Paler columns reuse another layer’s keys and values.

OPT-66B: layer stack and blockslayers (64)mixer / FFNlayer 0: MHA: 72 query / 72 KV heads · head 128layer 0: MLP: 36864layer 1: MHA: 72 query / 72 KV heads · head 128layer 1: MLP: 36864layer 2: MHA: 72 query / 72 KV heads · head 128layer 2: MLP: 36864layer 3: MHA: 72 query / 72 KV heads · head 128layer 3: MLP: 36864layer 4: MHA: 72 query / 72 KV heads · head 128layer 4: MLP: 36864layer 5: MHA: 72 query / 72 KV heads · head 128layer 5: MLP: 36864layer 6: MHA: 72 query / 72 KV heads · head 128layer 6: MLP: 36864layer 7: MHA: 72 query / 72 KV heads · head 128layer 7: MLP: 36864layer 8: MHA: 72 query / 72 KV heads · head 128layer 8: MLP: 36864layer 9: MHA: 72 query / 72 KV heads · head 128layer 9: MLP: 36864layer 10: MHA: 72 query / 72 KV heads · head 128layer 10: MLP: 36864layer 11: MHA: 72 query / 72 KV heads · head 128layer 11: MLP: 36864layer 12: MHA: 72 query / 72 KV heads · head 128layer 12: MLP: 36864layer 13: MHA: 72 query / 72 KV heads · head 128layer 13: MLP: 36864layer 14: MHA: 72 query / 72 KV heads · head 128layer 14: MLP: 36864layer 15: MHA: 72 query / 72 KV heads · head 128layer 15: MLP: 36864layer 16: MHA: 72 query / 72 KV heads · head 128layer 16: MLP: 36864layer 17: MHA: 72 query / 72 KV heads · head 128layer 17: MLP: 36864layer 18: MHA: 72 query / 72 KV heads · head 128layer 18: MLP: 36864layer 19: MHA: 72 query / 72 KV heads · head 128layer 19: MLP: 36864layer 20: MHA: 72 query / 72 KV heads · head 128layer 20: MLP: 36864layer 21: MHA: 72 query / 72 KV heads · head 128layer 21: MLP: 36864layer 22: MHA: 72 query / 72 KV heads · head 128layer 22: MLP: 36864layer 23: MHA: 72 query / 72 KV heads · head 128layer 23: MLP: 36864layer 24: MHA: 72 query / 72 KV heads · head 128layer 24: MLP: 36864layer 25: MHA: 72 query / 72 KV heads · head 128layer 25: MLP: 36864layer 26: MHA: 72 query / 72 KV heads · head 128layer 26: MLP: 36864layer 27: MHA: 72 query / 72 KV heads · head 128layer 27: MLP: 36864layer 28: MHA: 72 query / 72 KV heads · head 128layer 28: MLP: 36864layer 29: MHA: 72 query / 72 KV heads · head 128layer 29: MLP: 36864layer 30: MHA: 72 query / 72 KV heads · head 128layer 30: MLP: 36864layer 31: MHA: 72 query / 72 KV heads · head 128layer 31: MLP: 36864layer 32: MHA: 72 query / 72 KV heads · head 128layer 32: MLP: 36864layer 33: MHA: 72 query / 72 KV heads · head 128layer 33: MLP: 36864layer 34: MHA: 72 query / 72 KV heads · head 128layer 34: MLP: 36864layer 35: MHA: 72 query / 72 KV heads · head 128layer 35: MLP: 36864layer 36: MHA: 72 query / 72 KV heads · head 128layer 36: MLP: 36864layer 37: MHA: 72 query / 72 KV heads · head 128layer 37: MLP: 36864layer 38: MHA: 72 query / 72 KV heads · head 128layer 38: MLP: 36864layer 39: MHA: 72 query / 72 KV heads · head 128layer 39: MLP: 36864layer 40: MHA: 72 query / 72 KV heads · head 128layer 40: MLP: 36864layer 41: MHA: 72 query / 72 KV heads · head 128layer 41: MLP: 36864layer 42: MHA: 72 query / 72 KV heads · head 128layer 42: MLP: 36864layer 43: MHA: 72 query / 72 KV heads · head 128layer 43: MLP: 36864layer 44: MHA: 72 query / 72 KV heads · head 128layer 44: MLP: 36864layer 45: MHA: 72 query / 72 KV heads · head 128layer 45: MLP: 36864layer 46: MHA: 72 query / 72 KV heads · head 128layer 46: MLP: 36864layer 47: MHA: 72 query / 72 KV heads · head 128layer 47: MLP: 36864layer 48: MHA: 72 query / 72 KV heads · head 128layer 48: MLP: 36864layer 49: MHA: 72 query / 72 KV heads · head 128layer 49: MLP: 36864layer 50: MHA: 72 query / 72 KV heads · head 128layer 50: MLP: 36864layer 51: MHA: 72 query / 72 KV heads · head 128layer 51: MLP: 36864layer 52: MHA: 72 query / 72 KV heads · head 128layer 52: MLP: 36864layer 53: MHA: 72 query / 72 KV heads · head 128layer 53: MLP: 36864layer 54: MHA: 72 query / 72 KV heads · head 128layer 54: MLP: 36864layer 55: MHA: 72 query / 72 KV heads · head 128layer 55: MLP: 36864layer 56: MHA: 72 query / 72 KV heads · head 128layer 56: MLP: 36864layer 57: MHA: 72 query / 72 KV heads · head 128layer 57: MLP: 36864layer 58: MHA: 72 query / 72 KV heads · head 128layer 58: MLP: 36864layer 59: MHA: 72 query / 72 KV heads · head 128layer 59: MLP: 36864layer 60: MHA: 72 query / 72 KV heads · head 128layer 60: MLP: 36864layer 61: MHA: 72 query / 72 KV heads · head 128layer 61: MLP: 36864layer 62: MHA: 72 query / 72 KV heads · head 128layer 62: MLP: 36864layer 63: MHA: 72 query / 72 KV heads · head 128layer 63: MLP: 3686403263× 64normMHA: 72 query / 72 KV heads · head 128+normMLP: 36864+full attentiondense FFN

Modelled costs

From the cost model, batch size 1. Totals the lab states are in the table above; differences come from rounding, from what a lab counts, or from parts the model does not describe (listed on the about page).

Parameters (modelled)66.2B
Active per token (modelled)66.2B
Without embeddings and output head65.2B total, 65.2B active
KV cache per token, BF16 (layers that grow with context)2.25 MiB
KV cache + state at 2K tokens, BF164.5 GiB
Decode FLOPs per token at 4K context141 GFLOP
Prefill FLOPs for a 4K prompt554 TFLOP

KV cache against context

OPT-66B: KV cache bytes against context length101001,0009.5 MiB95 MiB950 MiB9.3 GiBcontext (tokens)KV cache + state (BF16)OPT-66B

Compare with other models →

Every architecture field

FieldValueSource
d_model9,216config.jsonconfig.jsonhidden_size
vocab50,272config.jsonconfig.jsonvocab_size
tied_embeddingsfalsecodemodelling codetransformers 5.18.0: PretrainedConfig.tie_word_embeddings defaultnot in config
mixers.full.typeattncodemodelling codeattention block
mixers.full.heads72config.jsonconfig.jsonnum_attention_heads
mixers.full.kv_heads72config.jsonconfig.jsonnum_attention_heads
mixers.full.head_dim128codemodelling codetransformers 5.18.0: head_dim = hidden_size / num_attention_heads
mixers.full.biastruecodemodelling codebiases
ffns.dense.typedensecodemodelling codeMLP block
ffns.dense.d_ff36,864config.jsonconfig.jsonffn_dim
ffns.dense.gatedfalsecodemodelling codeMLP: two matrices, no gate
ffns.dense.biastruecodemodelling codeMLP has biases
layout64× full/denseconfig.jsonconfig.jsonnum_hidden_layers
extra_embedding_params18,892,800codemodelling codelearned positions: (max_position_embeddings + 2) x hidden_size

Sources