llm-architectures-explained

/models

Switch-C

Google · Switch Transformer · open weights · encoder-decoder

Facts and where they come from

Released2021-01paperpaperarXiv v1, January 2021
Licenceapache-2.0config.jsonconfig.jsonREADME metadata: license
Total parameters1.57TpaperpaperTable 9: Switch-C, 1571B parameters
Active parametersnot disclosednot disclosed
Context lengthnot disclosednot disclosed
Norm placementprecodemodelling codetransformers 5.18.0 switch_transformers: T5-style pre-norm
Norm typeRMSNormcodemodelling codetransformers 5.18.0 switch_transformers: T5-style pre-norm
QK-normnocodemodelling codeno q/k normalisation in the attention block
Positional encodingrelative position biascodemodelling codetransformers 5.18.0 t5: bucketed relative position bias
Parallel attention and MLPnocodemodelling codetransformers 5.18.0 switch_transformers: T5-style pre-norm

Architecture, drawn from the data

MHA 32q/32kv. Each column is one layer: its token mixer above, its feed-forward block below. Paler columns reuse another layer’s keys and values.

Switch-C: layer stack and blocksencoder (15)mixer / FFNlayer 0: MHA: 32 query / 32 KV heads · head 64layer 0: MoE: 2048 experts, 1 active · expert 6144layer 1: MHA: 32 query / 32 KV heads · head 64layer 1: MoE: 2048 experts, 1 active · expert 6144layer 2: MHA: 32 query / 32 KV heads · head 64layer 2: MoE: 2048 experts, 1 active · expert 6144layer 3: MHA: 32 query / 32 KV heads · head 64layer 3: MoE: 2048 experts, 1 active · expert 6144layer 4: MHA: 32 query / 32 KV heads · head 64layer 4: MoE: 2048 experts, 1 active · expert 6144layer 5: MHA: 32 query / 32 KV heads · head 64layer 5: MoE: 2048 experts, 1 active · expert 6144layer 6: MHA: 32 query / 32 KV heads · head 64layer 6: MoE: 2048 experts, 1 active · expert 6144layer 7: MHA: 32 query / 32 KV heads · head 64layer 7: MoE: 2048 experts, 1 active · expert 6144layer 8: MHA: 32 query / 32 KV heads · head 64layer 8: MoE: 2048 experts, 1 active · expert 6144layer 9: MHA: 32 query / 32 KV heads · head 64layer 9: MoE: 2048 experts, 1 active · expert 6144layer 10: MHA: 32 query / 32 KV heads · head 64layer 10: MoE: 2048 experts, 1 active · expert 6144layer 11: MHA: 32 query / 32 KV heads · head 64layer 11: MoE: 2048 experts, 1 active · expert 6144layer 12: MHA: 32 query / 32 KV heads · head 64layer 12: MoE: 2048 experts, 1 active · expert 6144layer 13: MHA: 32 query / 32 KV heads · head 64layer 13: MoE: 2048 experts, 1 active · expert 6144layer 14: MHA: 32 query / 32 KV heads · head 64layer 14: MoE: 2048 experts, 1 active · expert 61440714decoder (15)mixer / FFNlayer 0: MHA: 32 query / 32 KV heads · head 64layer 0: MoE: 2048 experts, 1 active · expert 6144layer 1: MHA: 32 query / 32 KV heads · head 64layer 1: MoE: 2048 experts, 1 active · expert 6144layer 2: MHA: 32 query / 32 KV heads · head 64layer 2: MoE: 2048 experts, 1 active · expert 6144layer 3: MHA: 32 query / 32 KV heads · head 64layer 3: MoE: 2048 experts, 1 active · expert 6144layer 4: MHA: 32 query / 32 KV heads · head 64layer 4: MoE: 2048 experts, 1 active · expert 6144layer 5: MHA: 32 query / 32 KV heads · head 64layer 5: MoE: 2048 experts, 1 active · expert 6144layer 6: MHA: 32 query / 32 KV heads · head 64layer 6: MoE: 2048 experts, 1 active · expert 6144layer 7: MHA: 32 query / 32 KV heads · head 64layer 7: MoE: 2048 experts, 1 active · expert 6144layer 8: MHA: 32 query / 32 KV heads · head 64layer 8: MoE: 2048 experts, 1 active · expert 6144layer 9: MHA: 32 query / 32 KV heads · head 64layer 9: MoE: 2048 experts, 1 active · expert 6144layer 10: MHA: 32 query / 32 KV heads · head 64layer 10: MoE: 2048 experts, 1 active · expert 6144layer 11: MHA: 32 query / 32 KV heads · head 64layer 11: MoE: 2048 experts, 1 active · expert 6144layer 12: MHA: 32 query / 32 KV heads · head 64layer 12: MoE: 2048 experts, 1 active · expert 6144layer 13: MHA: 32 query / 32 KV heads · head 64layer 13: MoE: 2048 experts, 1 active · expert 6144layer 14: MHA: 32 query / 32 KV heads · head 64layer 14: MoE: 2048 experts, 1 active · expert 61440714× 15normMHA: 32 query / 32 KV heads · head 64+normMoE: 2048 experts, 1 active · expert 6144+full attentionMoE FFN

Modelled costs

From the cost model, batch size 1. Totals the lab states are in the table above; differences come from rounding, from what a lab counts, or from parts the model does not describe (listed on the about page).

Parameters (modelled)1.57T
Active per token (modelled)1.73B
Without embeddings and output head1.57T total, 1.66B active
KV cache per token, BF16 (layers that grow with context)240 KiB
KV cache + state at 128K tokens, BF1630 GiB
Decode FLOPs per token at 4K context4.46 GFLOP
Prefill FLOPs for a 4K prompt13.6 TFLOP

KV cache against context

Switch-C: KV cache bytes against context length101001,00010,000100,000980 KiB9.5 MiB95 MiB950 MiB9.3 GiBcontext (tokens)KV cache + state (BF16)Switch-C

Compare with other models →

Every architecture field

FieldValueSource
d_model2,080paperpaperTable 9: d_model 2080
vocab32,128paperpapergoogle/switch-c-2048 config: vocab_size 32128 (T5 vocabulary)
tied_embeddingstruecodemodelling codetransformers 5.18.0 t5: tie_word_embeddings default true
mixers.full.typeattncodemodelling codeattention
mixers.full.heads32paperpaperTable 9: 32 heads
mixers.full.kv_heads32paperpaperTable 9: 32 heads
mixers.full.head_dim64paperpaperTable 9: d_kv 64
ffns.dense.typedensecodemodelling codeMLP
ffns.dense.d_ff6,144paperpaperTable 9: d_ff 6144
ffns.dense.gatedfalsecodemodelling codeReLU MLP
ffns.moe.typemoecodemodelling codeSwitch MoE
ffns.moe.experts2,048paperpaperTable 9: 2048 experts
ffns.moe.active1codemodelling codeSwitch routing: top-1
ffns.moe.d_expert6,144paperpaperTable 9: d_ff 6144
ffns.moe.gatedfalsecodemodelling codeReLU experts
layout15× full/moepaperpaperTable 9: 15 layers
encoder_layout15× full/moepaperpaperTable 9: 15 layers
norms_per_layer2codemodelling codepre-norm RMSNorm

Sources