Module qwen 

The Qwen-family recognizers — the ops beyond matmul / elementwise / fused-row / swiglu that an authored transformer forward is made of, each lifted from the EXACT statement shape its parser sugar desugars to (parser/mod.rs, the desugar_tensor_* family) into a typed spec.

Shared the way super is: the CUDA orchestrator’s stage walk adapts these specs into its tuples, and the spatial front end’s whole-program walk composes them. Recognition is structural — no markers, no naming conventions — and every recognizer here is keyed on its own sugar’s body, so a composite block is never claimed as one of its parts and a neighbouring sugar with the same skeleton (silu / gelu, rope / transpose) is told apart by the sub-expression that differs. Each answers None for anything else.

Structs

ArgMaxSpec

t = argmax of X — the 1-based index of the first maximal element of X’s n elements, an Int.

AttentionSpec

ctx = attention(q over kc, vc) — single-query multi-head GQA attention: q is 1 × (h·hd), kc is l × (kv·hd) and vc l × (kv·vd) (the cached keys/values), query head g attends to kv head g / (h/kv) and writes vd values. pos is the position bound of the masked up to P form (the KV cache as carried state); None is the unmasked form that attends to every cached row.

CausalConvSpec

Y = causal conv(X, WC, CS) — the depthwise causal conv1d over chan channels with a taps-deep decode ring CS (shifted up, appended with X, reduced against WC). CS is read AND written: carried state.

DecaySpec

A = decay(Ar, Dt, Alog) — Gated DeltaNet’s per-head forget gate (1 + e^(Ar+Dt))^Alog over n heads.

FullAttentionSpec

C = full attention of Q over K and V with H heads — bidirectional multi-head self-attention over t positions; Q/K/V/C are t × (h·hd). Every query row attends to every position, unmasked.

GatedDeltaSpec

O = gated delta(Q, K, V, Z, B, A, S, GN) — the Gated DeltaNet step over hv value heads reading hk key heads at head dim d: u = S·k, S ← α·S + β·(v − α·u)·kᵀ, o = S·q, RMSNorm’d, scaled by GN, gated by silu(z), the norm taking the program’s epsilon. S (st) is read AND written: the carried memory.

GeluSpec

Y = gelu(X) — ggml’s tanh approximation in its one-exponential form x / (1 + e^-2z), z = √(2/π)·(x + 0.044715·x³), over n entries.

HeadSliceSpec

P = the first K of every N of X / the last K of every N of X — one part of every group of every consecutive elements of X’s n: the width elements from offset from of each group, in order (P[g·width + i] = X[g·every + from + i]). A head’s part of a row of heads, or a part of every cached row.

IntClampSpec

Y = X clamped between LO and HI — saturation, min(max(x, LO), HI) over n Int entries, LO ≤ HI.

IntRequantSpec

Y = X times M shifted right by S — an int8 model’s requant (x · M) >> S over n Int entries, an arithmetic shift of the exact product.

JoinSpec

J = A joined with B [across H heads] — rows rows of heads heads, each head’s part of A then its part of B: A is wa wide and B wb wide per row, a head’s parts wa / heads and wb / heads wide. One head joins the rows.

L2NormSpec

Y = l2norm(X) — per-row X / sqrt(Σ X² + eps) over a rows × cols view (the across H heads form is rows == H, cols == |X| / H), with the program’s epsilon.

LutAttentionSpec

Let C be lut attention of Q over K and V with H heads KV groups [up to P] with table E and scale M shifted right by S — single-query attention over Int tensors whose softmax is an exp table: per head, scores q · k_j, their maximum, weights E[clamp(((max − s_j) · M) >> S, 0, table_len − 1)] and the context ⌊Σ w_j · v_j / Σ w_j⌋. The shapes and pos are AttentionSpec’s.

MoeGroups

A mixture’s experts in count groups of ne / count, each scored by the sum of its two best selection scores (its one where a group holds one expert), the experts chosen only among the top kept groups.

MoeSpec

A recognized Mixture-of-Experts FFN — Let out be moe of x with router wr gate wg up wu down wd choosing K of E. Lifted from the desugared block the SAME way recognize_matmul/recognize_rope lift theirs — purely structural, no markers, no naming conventions: the OPERAND tensors are the indexed reads NOT Let-bound inside the block (the temps are), and the dims fall straight out of the weight index arithmetic (wr[(t-1)·E+e] yields E; wg[((e-1)·dff+(o-1))·d+t] yields d and dff); out is the tensor the weighted accumulation writes.

ReluSpec

Y = relu of X — max(x, 0) over n entries, in X’s element kind (int).

RopeSpec

Qr = rope(Q, cos, sin) — RoPE-NEOX over h heads of hd dims, rotating each head’s leading 2·half dims as (i, i+half) pairs and passing the remaining hd − 2·half through (Qwen3.8’s partial rotary; hd == 2·half is the classic full form). hd is read off the pass-through loop’s bound.

RowGatherSpec

x = row i of T — the embedding lookup: row i (1-based) of a rows × cols table, as 1 × cols.

RowsGatherSpec

X = rows I of T — the block gather: row k of X is row I[k] (1-based) of a rows × cols table, for each of I’s count indices.

ScaleSpec

Y = X scaled by C — x·C over n entries.

SetRowSpec

Set row i of T to S — the KV-cache append: row i (1-based) of a rows × cols table overwritten by the cols elements of S. T is read AND written: carried state.

SigmoidSpec

Y = sigmoid(X) — 1 / (1 + e^-x) over n entries.

SiluSpec

Y = silu(X) — x / (1 + e^-x) over n entries.

SoftcapSpec

Y = X capped at C — C·tanh(x/C) over n entries (Gemma 2’s logits).

TransposeSpec

Kt = transpose(X) — an rows × cols tensor becomes cols × rows (Kt[j·rows + i] = X[i·cols + j]).

UnaryMapSpec

The one-loop unary map Y[i] = f(X[i]) over n entries, with f classified exactly.

Enums

Activation

The activation a unary elementwise map applies: the three sugars that share the one-loop Push f(X[i]) skeleton and differ only in f.

GateAct

The activation of a Gated DeltaNet’s output gate.

MoeAct

A mixture’s gated activation: SwiGLU, or gpt-oss’s SwiGLU with the gate clamped to at most limit, the up to [−limit, limit], and the sigmoid sharpened by slope (both kept as their bits, so the spec stays Eq).

MoeScore

How a mixture scores its experts.

RowIndex

Where a row sugar’s row comes from: a NAMED scalar (the data-indexed case — a token id, a cache position, an arg-max) or a literal. An arbitrary expression is not lifted; the walk lists the statement instead.

Window

The positions of the n an attention query attends that it keeps.

Functions

recognize_argmax

Recognize Let t be argmax of X.

recognize_attention

Recognize the single-query GQA attention block. The desugar is a Splice that: Lets the output Seq (first New), zero-fills it, then a head-loop (While ≤ H whose body Lets a scores New(Seq)) computes scores = Q·K / √HD, a softmax (Pow(E, …)), and the weighted sum of V, SetIndexing the output. It does NOT mask with the -1e30 sentinel (that’s moe). Dims come from the loop bounds (H = head-loop bound, L = score-loop bound, HD = score-inner bound) and the kvh divisor (group = H/KV); the three externally-indexed operands, in program order, are Q, K, V. The masked up to P form sets pos.

recognize_causal_attention

Recognize Let C be causal attention of Q over K and V with H heads KV groups from P — a block of T query rows: the Let from be P and its block check from ≥ 1 and from + (T − 1) ≤ L at the top, the output’s zero fill, then a row loop (to T) whose body binds the row’s length from + row − 1 and runs the single-query head loop over it.

recognize_causal_conv

Recognize Let Y be causal conv of X with weight WC state CS over T taps — the three loops: the ring shift CS[(t−1)·chan+c] ← CS[t·chan+c], the append of X, and the tap reduce against WC.

recognize_decay

Recognize Let A be decay of Ar with bias Dt rate Alog — Push (1 + E^(Ar[i]+Dt[i]))^Alog[i].

recognize_full_attention

Recognize Let C be full attention of Q over K and V with H heads.

recognize_gated_delta

Recognize Let O be gated delta of Q over K and V with gate Z beta B decay A state S norm GN with HV heads HK groups — the head loop opened by the (head−1) mod HK key-head binding, the in-place S update, and the gated norm that writes the head.

recognize_gelu

Recognize Let Y be gelu of X — Push X[i] / (1 + E^(0 − 2z)) with the 0.044715 cubic in z.

recognize_head_slice

Recognize Let P be the first K of every N of X / the last K of every N of X — a loop over the groups, each pushing its K elements X[(g·N + i) + from].

recognize_int_clamp

Recognize Let Y be X clamped between LO and HI — Push min(max(X[p], LO), HI) in one counted loop.

recognize_int_requant

Recognize Let Y be X times M shifted right by S — Push (X[p] * M) shifted right by S in one counted loop.

recognize_join

Recognize Let J be A joined with B [across H heads] — a loop over the rows and in it a loop over the heads, each head pushing its part of A (A[(r·wa + h·ha) + i]) then its part of B.

recognize_l2norm

Recognize Let Y be l2norm of X [across H heads] — the per-row sqrt denominator bound in a Let and the X[base+q] / denom normalize loop.

recognize_lut_attention

Recognize the lut attention block: a Splice holding the LUT weight read and a floor-division write, its head loop and cache bound the single-query attention’s (attention_heads, masked_length). Its operands, the tensors it reads by index and does not bind, are Q, K, E, V in program order; the table is the one the LUT read names.

recognize_moe

Recognize a MoE block anywhere in the statement list (through Splice). None if not the moe shape.

recognize_relu

Recognize Let Y be relu of X — Push max(X[p], 0) in one counted loop, over Int or Real.

recognize_rope

Recognize Let Qr be rope of Q with cos RC and sin RS [from position P] [across H heads] — per head, the two rotation loops over half and the pass-through loop up to hd, after the position’s check when there is one.

recognize_row_gather

Recognize Let x be row i of T.

recognize_rows_gather

Recognize Let X be rows I of T.

recognize_scale

Recognize Let Y be X scaled by C anywhere in the statement list.

recognize_set_row

Recognize Set row i of T to S.

recognize_sigmoid

Recognize Let Y be sigmoid of X — Push 1 / (1 + E^(0 − X[i])) in one counted loop.

recognize_silu

Recognize Let Y be silu of X — Push X[i] / (1 + E^(0 − X[i])) in one counted loop.

recognize_softcap

Recognize Let Y be X capped at C anywhere in the statement list.

recognize_transpose

Recognize Let Kt be transpose of X — the cols-outer / rows-inner loop nest pushing X[ti·cols+tj+1].

recognize_unary_map

Recognize the one-loop unary map Let Y be silu/sigmoid/gelu of X anywhere in the statement list.