Module spec 

A decoder-only language model’s architecture, as data.

ModelSpec says what a checkpoint computes — the width, the vocabulary, and for every layer its token mixer (attention or a gated-delta recurrence), its feed-forward block (dense or a mixture of experts) and how its norms are arranged. It is read from the checkpoint’s own metadata (ModelSpec::from_gguf), so bringing up a model is reading its file: the per-architecture knowledge lives in one match over general.architecture, and everything downstream (the authored .lg, the reference forward, the weight binding) is generic over the spec.

The meaning of each field follows llama.cpp’s graph for that architecture (src/models/<arch>.cpp at the pinned commit), which is the reference the conformance locks hold the model to.

Re-exports

pub use crate::tensor_recognize::qwen::Window;

Structs

Attention

Grouped-query attention, with the variants checkpoints use.

GatedDelta

The Gated DeltaNet recurrence (Qwen3.5/3.8 linear-attention layers).

HyperConnections

Hyper-connections: the residual is lanes copies of the hidden state. Before each sublayer the lanes are normed each on its own (one weight across all of them), weighted by σ(up·silu(down·x̂ / lanes)) through a bottleneck of rank and averaged into the sublayer’s input; the sublayer’s output joins every lane scaled by 2σ(inject·x̂ / lanes). A last mix with no injection feeds the head in place of an output norm.

Indexer

Qwen’s sparse attention selection: heads index queries of head_dim over one index key a position, the keys of each complete block of block positions averaged, normed and rotated at the block’s first position, each block scored by Σ_h relu(q_h·k̄)/√head_dim; a query attends the best budget / block blocks and the positions after the last complete block, every visible position while the complete blocks are no more than that. The attention layers in layers select; any other attends every position.

Layer

One block: its token mixer, its feed-forward block, and its norms.

Mla

Multi-head latent attention (DeepSeek V2/V3): the keys and values of every head are expanded from one shared low-rank latent per position (kv_lora_rank wide, RMS-normed), and each head’s query and key carry a further qk_rope dimensions that rotate — the key’s shared by every head. The query is a full projection, or itself low-rank (q_lora_rank, normed between its two halves).

ModelSpec

What a checkpoint computes.

Moe

A mixture of experts.

Ple

The per-layer n-gram embedding: each position’s n-grams of 2 to ngram tokens, read back no further than the last end of document, are hashed heads_per_ngram ways each (t₀·m₀ ⊕ t₁·m₁ ⊕ … modulo a head’s prime, from the head’s offset in one table); the heads’ rows, side by side, project to a value and to a key for each hyper-connection lane, the lane’s normed state gating the value by σ(sign(g)·√|g|) of its agreement g with the key, and a dilated depthwise convolution over the normed gated values adds local context. Its sum joins the layer’s residual input.

Rope

The rotary position embedding.

Temperature

Llama 4’s attention temperature: the query at position p (0-based) scaled by 1 + scale·ln(⌊(p + 1)/floor⌋ + 1), so short contexts are untouched and long ones sharpen (attn_temperature_tuning).

Enums

Act

A gated activation act(gate) · up.

Ffn

A block’s feed-forward network.

GateActivation

A gate’s activation: SiLU (Qwen3.5’s and Qwen3-Next’s Gated DeltaNet) or the sigmoid (Qwen3.8-Flash-Next’s).

Gating

How a router turns logits into expert weights, and where a weight applies: to its expert’s output, unless the gating says otherwise.

Mixer

How a block mixes information across positions.

RopeScaling

Context extension applied to the rotary frequencies.

SpecError

Why a checkpoint is not a model this crate describes.

Traits

SpecSource

What a checkpoint’s architecture is read from: its architecture’s name, its keys and its tensor directory — a GGUF checkpoint, or anything describing one.