llm-architectures-explained

/models

OLMoE-1B-7B

Ai2 · OLMo · open weights

Facts and where they come from

Released2024-07config.jsonconfig.jsonHugging Face repository creation date (api.createdAt)
Licenceapache-2.0config.jsonconfig.jsonREADME metadata: license
Total parameters7Blabmodel cardREADME: 1B active and 7B total parameters
Active parameters1Blabmodel cardREADME: 1B active
Context length4K tokensconfig.jsonconfig.jsonmax_position_embeddings
Norm placementprecodemodelling codetransformers 5.18.0 / repo modelling code for olmoe: input_layernorm before attention, post_attention_layernorm before the MLP
Norm typeRMSNormcodemodelling codetransformers 5.18.0 / repo modelling code for olmoe: input_layernorm before attention, post_attention_layernorm before the MLP
QK-normyescodemodelling codetransformers 5.18.0 olmoe: q_norm and k_norm
Positional encodingRoPEcodemodelling coderotary on the full head (default)
Parallel attention and MLPnocodemodelling codetransformers 5.18.0 / repo modelling code for olmoe: input_layernorm before attention, post_attention_layernorm before the MLP

Architecture, drawn from the data

MHA 16q/16kv. Each column is one layer: its token mixer above, its feed-forward block below. Paler columns reuse another layer’s keys and values.

OLMoE-1B-7B: layer stack and blockslayers (16)mixer / FFNlayer 0: MHA: 16 query / 16 KV heads · head 128layer 0: MoE: 64 experts, 8 active · expert 1024layer 1: MHA: 16 query / 16 KV heads · head 128layer 1: MoE: 64 experts, 8 active · expert 1024layer 2: MHA: 16 query / 16 KV heads · head 128layer 2: MoE: 64 experts, 8 active · expert 1024layer 3: MHA: 16 query / 16 KV heads · head 128layer 3: MoE: 64 experts, 8 active · expert 1024layer 4: MHA: 16 query / 16 KV heads · head 128layer 4: MoE: 64 experts, 8 active · expert 1024layer 5: MHA: 16 query / 16 KV heads · head 128layer 5: MoE: 64 experts, 8 active · expert 1024layer 6: MHA: 16 query / 16 KV heads · head 128layer 6: MoE: 64 experts, 8 active · expert 1024layer 7: MHA: 16 query / 16 KV heads · head 128layer 7: MoE: 64 experts, 8 active · expert 1024layer 8: MHA: 16 query / 16 KV heads · head 128layer 8: MoE: 64 experts, 8 active · expert 1024layer 9: MHA: 16 query / 16 KV heads · head 128layer 9: MoE: 64 experts, 8 active · expert 1024layer 10: MHA: 16 query / 16 KV heads · head 128layer 10: MoE: 64 experts, 8 active · expert 1024layer 11: MHA: 16 query / 16 KV heads · head 128layer 11: MoE: 64 experts, 8 active · expert 1024layer 12: MHA: 16 query / 16 KV heads · head 128layer 12: MoE: 64 experts, 8 active · expert 1024layer 13: MHA: 16 query / 16 KV heads · head 128layer 13: MoE: 64 experts, 8 active · expert 1024layer 14: MHA: 16 query / 16 KV heads · head 128layer 14: MoE: 64 experts, 8 active · expert 1024layer 15: MHA: 16 query / 16 KV heads · head 128layer 15: MoE: 64 experts, 8 active · expert 10240815× 16normMHA: 16 query / 16 KV heads · head 128+normMoE: 64 experts, 8 active · expert 1024+full attentionMoE FFN

Modelled costs

From the cost model, batch size 1. Totals the lab states are in the table above; differences come from rounding, from what a lab counts, or from parts the model does not describe (listed on the about page).

Parameters (modelled)6.92B
Active per token (modelled)1.28B
Without embeddings and output head6.71B total, 1.08B active
Published weights (Hugging Face count)6.92B
KV cache per token, BF16 (layers that grow with context)128 KiB
KV cache + state at 4K tokens, BF16512 MiB
Decode FLOPs per token at 4K context2.89 GFLOP
Prefill FLOPs for a 4K prompt9.91 TFLOP

KV cache against context

OLMoE-1B-7B: KV cache bytes against context length101001,000980 KiB9.5 MiB95 MiBcontext (tokens)KV cache + state (BF16)OLMoE-1B-7B

Compare with other models →

Every architecture field

FieldValueSource
d_model2,048config.jsonconfig.jsonhidden_size
vocab50,304config.jsonconfig.jsonvocab_size
tied_embeddingsfalseconfig.jsonconfig.jsontie_word_embeddings
mixers.full.typeattncodemodelling codeattention block
mixers.full.heads16config.jsonconfig.jsonnum_attention_heads
mixers.full.kv_heads16config.jsonconfig.jsonnum_key_value_heads
mixers.full.head_dim128codemodelling codetransformers 5.18.0: head_dim = hidden_size / num_attention_heads
mixers.full.qk_normtruecodemodelling codetransformers 5.18.0 olmoe: q_norm and k_norm
ffns.moe.typemoecodemodelling codeMoE block
ffns.moe.experts64config.jsonconfig.jsonnum_experts
ffns.moe.active8config.jsonconfig.jsonnum_experts_per_tok
ffns.moe.d_expert1,024config.jsonconfig.jsonintermediate_size
ffns.moe.gatedtruecodemodelling codeexperts are gated MLPs
layout16× full/moeconfig.jsonconfig.jsonnum_hidden_layers

Sources