GLaM (64B/64E)
Google · GLaM · closed weights
- MHA
- Relative bias
- Pre-norm
- MoE
Facts and where they come from
| Released | 2021-12 | paperpaperarXiv v1, December 2021 |
|---|---|---|
| Licence | proprietary | labpaperweights not published |
| Total parameters | 1.2T | paperpaperTable 4: 64B/64E, 1.2T parameters |
| Active parameters | 96.6B | paperpaperTable 4: 96.6B activated |
| Context length | not disclosed | not disclosed |
| Norm placement | pre | codemodelling codeGLaM paper: standard Transformer layers |
| Norm type | LayerNorm | codemodelling codeGLaM paper: standard Transformer layers |
| QK-norm | no | codemodelling codeno q/k normalisation in the attention block |
| Positional encoding | relative position bias | paperpaperGLaM §4: per-layer relative positional bias |
| Parallel attention and MLP | no | codemodelling codeGLaM paper: standard Transformer layers |
Architecture, drawn from the data
MHA 128q/128kv. Each column is one layer: its token mixer above, its feed-forward block below. Paler columns reuse another layer’s keys and values.
Modelled costs
From the cost model, batch size 1. Totals the lab states are in the table above; differences come from rounding, from what a lab counts, or from parts the model does not describe (listed on the about page).
| Parameters (modelled) | 1.16T |
|---|---|
| Active per token (modelled) | 98.7B |
| Without embeddings and output head | 1.16T total, 94.5B active |
| KV cache per token, BF16 (layers that grow with context) | 4 MiB |
| KV cache + state at 128K tokens, BF16 | 512 GiB |
| Decode FLOPs per token at 4K context | 210 GFLOP |
| Prefill FLOPs for a 4K prompt | 809 TFLOP |
KV cache against context
Every architecture field
| Field | Value | Source |
|---|---|---|
| d_model | 8,192 | paperpaperTable 4: d_model 8,192 |
| vocab | 256,000 | paperpaper§5: vocabulary of size 256K (256,000 assumed) |
| tied_embeddings | false | paperpaperdata/transcribed/glam-64b-64e.json: tie_word_embeddings |
| mixers.full.type | attn | codemodelling codeattention block |
| mixers.full.heads | 128 | paperpaperTable 4: heads 128 |
| mixers.full.kv_heads | 128 | paperpaperdata/transcribed/glam-64b-64e.json: num_key_value_heads |
| mixers.full.head_dim | 128 | paperpaperTable 4: d_head 128 |
| ffns.dense.type | dense | codemodelling codeMLP block |
| ffns.dense.d_ff | 32,768 | paperpaperTable 4: d_ffn 32,768 |
| ffns.dense.gated | true | codemodelling codeMLP: gated (SwiGLU/GeGLU) |
| ffns.moe.type | moe | codemodelling codeMoE block |
| ffns.moe.experts | 64 | paperpaperTable 4: 64 experts |
| ffns.moe.active | 2 | paperpaper§3: top-2 gating |
| ffns.moe.d_expert | 32,768 | paperpaperTable 4: d_ffn 32,768 |
| ffns.moe.gated | false | paperpaper§4: GLU (with GELU) replaces the first projection in the non-MoE feed-forward sub-layers only; experts taken as two-matrix MLPs |
| layout | 1× full/dense · 1× full/moe · 1× full/dense · 1× full/moe · 1× full/dense · 1× full/moe · 1× full/dense · 1× full/moe · 1× full/dense · 1× full/moe · 1× full/dense · 1× full/moe · 1× full/dense · 1× full/moe · 1× full/dense · 1× full/moe · 1× full/dense · 1× full/moe · 1× full/dense · 1× full/moe · 1× full/dense · 1× full/moe · 1× full/dense · 1× full/moe · 1× full/dense · 1× full/moe · 1× full/dense · 1× full/moe · 1× full/dense · 1× full/moe · 1× full/dense · 1× full/moe · 1× full/dense · 1× full/moe · 1× full/dense · 1× full/moe · 1× full/dense · 1× full/moe · 1× full/dense · 1× full/moe · 1× full/dense · 1× full/moe · 1× full/dense · 1× full/moe · 1× full/dense · 1× full/moe · 1× full/dense · 1× full/moe · 1× full/dense · 1× full/moe · 1× full/dense · 1× full/moe · 1× full/dense · 1× full/moe · 1× full/dense · 1× full/moe · 1× full/dense · 1× full/moe · 1× full/dense · 1× full/moe · 1× full/dense · 1× full/moe · 1× full/dense · 1× full/moe | paperpaperTable 4: L 64 |
Sources
- paper · GLaM: Efficient Scaling of Language Models with Mixture-of-Experts
- modelling code · transformers 5.18.0 modelling code, or the model repository's own modelling file at the pinned revision