llm-architectures-explained

/models

Phi-3-mini

Microsoft · Phi · open weights

Facts and where they come from

Released2024-04config.jsonconfig.jsonHugging Face repository creation date (api.createdAt)
Licencemitconfig.jsonconfig.jsonREADME metadata: license
Total parameters3.8Blabmodel cardREADME: has 3.8B parameters
Active parametersnot disclosednot disclosed
Context length4K tokenslabmodel cardREADME: Context length: 4K tokens
Norm placementprecodemodelling codetransformers 5.18.0 / repo modelling code for phi3: input_layernorm before attention, post_attention_layernorm before the MLP
Norm typeRMSNormcodemodelling codetransformers 5.18.0 / repo modelling code for phi3: input_layernorm before attention, post_attention_layernorm before the MLP
QK-normnocodemodelling codeno q/k normalisation in the attention block
Positional encodingRoPEcodemodelling coderotary on the full head (default)
Parallel attention and MLPnocodemodelling codetransformers 5.18.0 / repo modelling code for phi3: input_layernorm before attention, post_attention_layernorm before the MLP

Architecture, drawn from the data

MHA 32q/32kv. Each column is one layer: its token mixer above, its feed-forward block below. Paler columns reuse another layer’s keys and values.

Phi-3-mini: layer stack and blockslayers (32)mixer / FFNlayer 0: MHA: 32 query / 32 KV heads · head 96layer 0: Gated MLP: 8192layer 1: MHA: 32 query / 32 KV heads · head 96layer 1: Gated MLP: 8192layer 2: MHA: 32 query / 32 KV heads · head 96layer 2: Gated MLP: 8192layer 3: MHA: 32 query / 32 KV heads · head 96layer 3: Gated MLP: 8192layer 4: MHA: 32 query / 32 KV heads · head 96layer 4: Gated MLP: 8192layer 5: MHA: 32 query / 32 KV heads · head 96layer 5: Gated MLP: 8192layer 6: MHA: 32 query / 32 KV heads · head 96layer 6: Gated MLP: 8192layer 7: MHA: 32 query / 32 KV heads · head 96layer 7: Gated MLP: 8192layer 8: MHA: 32 query / 32 KV heads · head 96layer 8: Gated MLP: 8192layer 9: MHA: 32 query / 32 KV heads · head 96layer 9: Gated MLP: 8192layer 10: MHA: 32 query / 32 KV heads · head 96layer 10: Gated MLP: 8192layer 11: MHA: 32 query / 32 KV heads · head 96layer 11: Gated MLP: 8192layer 12: MHA: 32 query / 32 KV heads · head 96layer 12: Gated MLP: 8192layer 13: MHA: 32 query / 32 KV heads · head 96layer 13: Gated MLP: 8192layer 14: MHA: 32 query / 32 KV heads · head 96layer 14: Gated MLP: 8192layer 15: MHA: 32 query / 32 KV heads · head 96layer 15: Gated MLP: 8192layer 16: MHA: 32 query / 32 KV heads · head 96layer 16: Gated MLP: 8192layer 17: MHA: 32 query / 32 KV heads · head 96layer 17: Gated MLP: 8192layer 18: MHA: 32 query / 32 KV heads · head 96layer 18: Gated MLP: 8192layer 19: MHA: 32 query / 32 KV heads · head 96layer 19: Gated MLP: 8192layer 20: MHA: 32 query / 32 KV heads · head 96layer 20: Gated MLP: 8192layer 21: MHA: 32 query / 32 KV heads · head 96layer 21: Gated MLP: 8192layer 22: MHA: 32 query / 32 KV heads · head 96layer 22: Gated MLP: 8192layer 23: MHA: 32 query / 32 KV heads · head 96layer 23: Gated MLP: 8192layer 24: MHA: 32 query / 32 KV heads · head 96layer 24: Gated MLP: 8192layer 25: MHA: 32 query / 32 KV heads · head 96layer 25: Gated MLP: 8192layer 26: MHA: 32 query / 32 KV heads · head 96layer 26: Gated MLP: 8192layer 27: MHA: 32 query / 32 KV heads · head 96layer 27: Gated MLP: 8192layer 28: MHA: 32 query / 32 KV heads · head 96layer 28: Gated MLP: 8192layer 29: MHA: 32 query / 32 KV heads · head 96layer 29: Gated MLP: 8192layer 30: MHA: 32 query / 32 KV heads · head 96layer 30: Gated MLP: 8192layer 31: MHA: 32 query / 32 KV heads · head 96layer 31: Gated MLP: 819201631× 32normMHA: 32 query / 32 KV heads · head 96+normGated MLP: 8192+full attentiondense FFN

Modelled costs

From the cost model, batch size 1. Totals the lab states are in the table above; differences come from rounding, from what a lab counts, or from parts the model does not describe (listed on the about page).

Parameters (modelled)3.82B
Active per token (modelled)3.82B
Without embeddings and output head3.62B total, 3.62B active
Published weights (Hugging Face count)3.82B
KV cache per token, BF16 (layers that grow with context)384 KiB
KV cache + state at 4K tokens, BF161.5 GiB
Decode FLOPs per token at 4K context9.06 GFLOP
Prefill FLOPs for a 4K prompt33 TFLOP

KV cache against context

Phi-3-mini: KV cache bytes against context length101001,000980 KiB9.5 MiB95 MiB950 MiBcontext (tokens)KV cache + state (BF16)Phi-3-mini

Compare with other models →

Every architecture field

FieldValueSource
d_model3,072config.jsonconfig.jsonhidden_size
vocab32,064config.jsonconfig.jsonvocab_size
tied_embeddingsfalseconfig.jsonconfig.jsontie_word_embeddings
mixers.full.typeattncodemodelling codeattention block
mixers.full.heads32config.jsonconfig.jsonnum_attention_heads
mixers.full.kv_heads32config.jsonconfig.jsonnum_key_value_heads
mixers.full.head_dim96codemodelling codetransformers 5.18.0: head_dim = hidden_size / num_attention_heads
ffns.dense.typedensecodemodelling codeMLP block
ffns.dense.d_ff8,192config.jsonconfig.jsonintermediate_size
ffns.dense.gatedtruecodemodelling codeMLP: gated (SwiGLU/GeGLU)
layout32× full/denseconfig.jsonconfig.jsonnum_hidden_layers

Sources