llm-architectures-explained

/models

RWKV-6 Finch 14B

RWKV · RWKV · open weights

Facts and where they come from

Released2024-07config.jsonconfig.jsonHugging Face repository creation date (api.createdAt)
Licenceapache-2.0config.jsonconfig.jsonREADME metadata: license
Total parameters14Blabmodel cardmodel name v6-Finch-14B
Active parametersnot disclosednot disclosed
Context lengthnot disclosednot disclosed
Norm placementprecodemodelling coderepo modeling_rwkv6.py: ln1 and ln2 before each mixing block
Norm typeLayerNormcodemodelling coderepo modeling_rwkv6.py: ln1 and ln2 before each mixing block
QK-normnocodemodelling codeno q/k normalisation in the attention block
Positional encodingnone; all layers: the recurrence carries ordercodemodelling codetransformers 5.18.0 rwkv6: no positional encoding
Parallel attention and MLPnocodemodelling coderepo modeling_rwkv6.py: ln1 and ln2 before each mixing block

Architecture, drawn from the data

RWKV. Each column is one layer: its token mixer above, its feed-forward block below. Paler columns reuse another layer’s keys and values.

RWKV-6 Finch 14B: layer stack and blockslayers (61)mixer / FFNlayer 0: RWKV time mixing · head 64layer 0: MLP: 14336layer 1: RWKV time mixing · head 64layer 1: MLP: 14336layer 2: RWKV time mixing · head 64layer 2: MLP: 14336layer 3: RWKV time mixing · head 64layer 3: MLP: 14336layer 4: RWKV time mixing · head 64layer 4: MLP: 14336layer 5: RWKV time mixing · head 64layer 5: MLP: 14336layer 6: RWKV time mixing · head 64layer 6: MLP: 14336layer 7: RWKV time mixing · head 64layer 7: MLP: 14336layer 8: RWKV time mixing · head 64layer 8: MLP: 14336layer 9: RWKV time mixing · head 64layer 9: MLP: 14336layer 10: RWKV time mixing · head 64layer 10: MLP: 14336layer 11: RWKV time mixing · head 64layer 11: MLP: 14336layer 12: RWKV time mixing · head 64layer 12: MLP: 14336layer 13: RWKV time mixing · head 64layer 13: MLP: 14336layer 14: RWKV time mixing · head 64layer 14: MLP: 14336layer 15: RWKV time mixing · head 64layer 15: MLP: 14336layer 16: RWKV time mixing · head 64layer 16: MLP: 14336layer 17: RWKV time mixing · head 64layer 17: MLP: 14336layer 18: RWKV time mixing · head 64layer 18: MLP: 14336layer 19: RWKV time mixing · head 64layer 19: MLP: 14336layer 20: RWKV time mixing · head 64layer 20: MLP: 14336layer 21: RWKV time mixing · head 64layer 21: MLP: 14336layer 22: RWKV time mixing · head 64layer 22: MLP: 14336layer 23: RWKV time mixing · head 64layer 23: MLP: 14336layer 24: RWKV time mixing · head 64layer 24: MLP: 14336layer 25: RWKV time mixing · head 64layer 25: MLP: 14336layer 26: RWKV time mixing · head 64layer 26: MLP: 14336layer 27: RWKV time mixing · head 64layer 27: MLP: 14336layer 28: RWKV time mixing · head 64layer 28: MLP: 14336layer 29: RWKV time mixing · head 64layer 29: MLP: 14336layer 30: RWKV time mixing · head 64layer 30: MLP: 14336layer 31: RWKV time mixing · head 64layer 31: MLP: 14336layer 32: RWKV time mixing · head 64layer 32: MLP: 14336layer 33: RWKV time mixing · head 64layer 33: MLP: 14336layer 34: RWKV time mixing · head 64layer 34: MLP: 14336layer 35: RWKV time mixing · head 64layer 35: MLP: 14336layer 36: RWKV time mixing · head 64layer 36: MLP: 14336layer 37: RWKV time mixing · head 64layer 37: MLP: 14336layer 38: RWKV time mixing · head 64layer 38: MLP: 14336layer 39: RWKV time mixing · head 64layer 39: MLP: 14336layer 40: RWKV time mixing · head 64layer 40: MLP: 14336layer 41: RWKV time mixing · head 64layer 41: MLP: 14336layer 42: RWKV time mixing · head 64layer 42: MLP: 14336layer 43: RWKV time mixing · head 64layer 43: MLP: 14336layer 44: RWKV time mixing · head 64layer 44: MLP: 14336layer 45: RWKV time mixing · head 64layer 45: MLP: 14336layer 46: RWKV time mixing · head 64layer 46: MLP: 14336layer 47: RWKV time mixing · head 64layer 47: MLP: 14336layer 48: RWKV time mixing · head 64layer 48: MLP: 14336layer 49: RWKV time mixing · head 64layer 49: MLP: 14336layer 50: RWKV time mixing · head 64layer 50: MLP: 14336layer 51: RWKV time mixing · head 64layer 51: MLP: 14336layer 52: RWKV time mixing · head 64layer 52: MLP: 14336layer 53: RWKV time mixing · head 64layer 53: MLP: 14336layer 54: RWKV time mixing · head 64layer 54: MLP: 14336layer 55: RWKV time mixing · head 64layer 55: MLP: 14336layer 56: RWKV time mixing · head 64layer 56: MLP: 14336layer 57: RWKV time mixing · head 64layer 57: MLP: 14336layer 58: RWKV time mixing · head 64layer 58: MLP: 14336layer 59: RWKV time mixing · head 64layer 59: MLP: 14336layer 60: RWKV time mixing · head 64layer 60: MLP: 1433603060× 61normRWKV time mixing · head 64+normMLP: 14336+RWKVdense FFN

Modelled costs

From the cost model, batch size 1. Totals the lab states are in the table above; differences come from rounding, from what a lab counts, or from parts the model does not describe (listed on the about page).

Parameters (modelled)13.8B
Active per token (modelled)13.8B
Without embeddings and output head13.3B total, 13.3B active
KV cache per token, BF16 (layers that grow with context)0 B
KV cache + state at 128K tokens, BF1631.5 MiB
Decode FLOPs per token at 4K context27.2 GFLOP
Prefill FLOPs for a 4K prompt109 TFLOP

KV cache against context

RWKV-6 Finch 14B: KV cache bytes against context length101001,00010,000100,000context (tokens)KV cache + state (BF16)RWKV-6 Finch 14B

Compare with other models →

Every architecture field

FieldValueSource
d_model4,096config.jsonconfig.jsonhidden_size
vocab65,536config.jsonconfig.jsonvocab_size
tied_embeddingsfalsecodemodelling codeRWKV: separate head
mixers.rwkv.typerwkvcodemodelling codeRWKV time mixing
mixers.rwkv.head_dim64config.jsonconfig.jsonhead_size
mixers.rwkv.mats5codemodelling codetime mixing: receptance, key, value, output, gate (RWKV-6)
ffns.dense.typedensecodemodelling codechannel mixing
ffns.dense.d_ff14,336codemodelling codemodeling_rwkv6.py: intermediate_size = int(hidden_size * 3.5) // 32 * 32
ffns.dense.gatedfalsecodemodelling codechannel mixing: key and value matrices
ffns.dense.receptancetruecodemodelling codechannel mixing also has a d x d receptance matrix
layout61× rwkv/denseconfig.jsonconfig.jsonnum_hidden_layers

Sources