llm-architectures-explained

/models

Falcon-7B

TII · Falcon · open weights

Facts and where they come from

Released2023-05labmodel cardHF repository created 2023-04; paper Nov 2023
Licenceapache-2.0config.jsonconfig.jsonREADME metadata: license
Total parameters7Blabmodel cardREADME: Falcon-7B is a 7B parameters causal decoder-only model
Active parametersnot disclosednot disclosed
Context lengthnot disclosednot disclosed
Norm placementparallelcodemodelling coderepo modeling_falcon.py: parallel attention and MLP
Norm typeLayerNormcodemodelling coderepo modeling_falcon.py: parallel attention and MLP
QK-normnocodemodelling codeno q/k normalisation in the attention block
Positional encodingRoPEcodemodelling coderotary on the full head (default)
Parallel attention and MLPyescodemodelling coderepo modeling_falcon.py: parallel attention and MLP

Architecture, drawn from the data

MQA 71q/1kv. Each column is one layer: its token mixer above, its feed-forward block below. Paler columns reuse another layer’s keys and values.

Falcon-7B: layer stack and blockslayers (32)mixer / FFNlayer 0: MQA: 71 query / 1 KV heads · head 64layer 0: MLP: 18176layer 1: MQA: 71 query / 1 KV heads · head 64layer 1: MLP: 18176layer 2: MQA: 71 query / 1 KV heads · head 64layer 2: MLP: 18176layer 3: MQA: 71 query / 1 KV heads · head 64layer 3: MLP: 18176layer 4: MQA: 71 query / 1 KV heads · head 64layer 4: MLP: 18176layer 5: MQA: 71 query / 1 KV heads · head 64layer 5: MLP: 18176layer 6: MQA: 71 query / 1 KV heads · head 64layer 6: MLP: 18176layer 7: MQA: 71 query / 1 KV heads · head 64layer 7: MLP: 18176layer 8: MQA: 71 query / 1 KV heads · head 64layer 8: MLP: 18176layer 9: MQA: 71 query / 1 KV heads · head 64layer 9: MLP: 18176layer 10: MQA: 71 query / 1 KV heads · head 64layer 10: MLP: 18176layer 11: MQA: 71 query / 1 KV heads · head 64layer 11: MLP: 18176layer 12: MQA: 71 query / 1 KV heads · head 64layer 12: MLP: 18176layer 13: MQA: 71 query / 1 KV heads · head 64layer 13: MLP: 18176layer 14: MQA: 71 query / 1 KV heads · head 64layer 14: MLP: 18176layer 15: MQA: 71 query / 1 KV heads · head 64layer 15: MLP: 18176layer 16: MQA: 71 query / 1 KV heads · head 64layer 16: MLP: 18176layer 17: MQA: 71 query / 1 KV heads · head 64layer 17: MLP: 18176layer 18: MQA: 71 query / 1 KV heads · head 64layer 18: MLP: 18176layer 19: MQA: 71 query / 1 KV heads · head 64layer 19: MLP: 18176layer 20: MQA: 71 query / 1 KV heads · head 64layer 20: MLP: 18176layer 21: MQA: 71 query / 1 KV heads · head 64layer 21: MLP: 18176layer 22: MQA: 71 query / 1 KV heads · head 64layer 22: MLP: 18176layer 23: MQA: 71 query / 1 KV heads · head 64layer 23: MLP: 18176layer 24: MQA: 71 query / 1 KV heads · head 64layer 24: MLP: 18176layer 25: MQA: 71 query / 1 KV heads · head 64layer 25: MLP: 18176layer 26: MQA: 71 query / 1 KV heads · head 64layer 26: MLP: 18176layer 27: MQA: 71 query / 1 KV heads · head 64layer 27: MLP: 18176layer 28: MQA: 71 query / 1 KV heads · head 64layer 28: MLP: 18176layer 29: MQA: 71 query / 1 KV heads · head 64layer 29: MLP: 18176layer 30: MQA: 71 query / 1 KV heads · head 64layer 30: MLP: 18176layer 31: MQA: 71 query / 1 KV heads · head 64layer 31: MLP: 1817601631× 32normMQA: 71 query / 1 KV heads · head 64MLP: 18176+full attentiondense FFN

Modelled costs

From the cost model, batch size 1. Totals the lab states are in the table above; differences come from rounding, from what a lab counts, or from parts the model does not describe (listed on the about page).

Parameters (modelled)6.92B
Active per token (modelled)6.92B
Without embeddings and output head6.63B total, 6.63B active
Published weights (Hugging Face count)7.22B
KV cache per token, BF16 (layers that grow with context)8 KiB
KV cache + state at 128K tokens, BF161 GiB
Decode FLOPs per token at 4K context16.2 GFLOP
Prefill FLOPs for a 4K prompt59.2 TFLOP

KV cache against context

Falcon-7B: KV cache bytes against context length101001,00010,000100,00098 KiB980 KiB9.5 MiB95 MiB950 MiBcontext (tokens)KV cache + state (BF16)Falcon-7B

Compare with other models →

Every architecture field

FieldValueSource
d_model4,544config.jsonconfig.jsonhidden_size
vocab65,024config.jsonconfig.jsonvocab_size
tied_embeddingstruecodemodelling codemodeling_falcon.py: lm_head tied to word_embeddings
mixers.full.typeattncodemodelling codeattention
mixers.full.heads71config.jsonconfig.jsonnum_attention_heads
mixers.full.kv_heads1codemodelling codemulti_query: true
mixers.full.head_dim64codemodelling codehidden_size / num_attention_heads
ffns.dense.typedensecodemodelling codeMLP
ffns.dense.d_ff18,176codemodelling codemodeling_falcon.py: 4 * hidden_size
ffns.dense.gatedfalsecodemodelling codeGELU MLP
layout32× full/denseconfig.jsonconfig.jsonnum_hidden_layers, n_layer
norms_per_layer1codemodelling codeparallel attention and MLP

Sources