Module numerics 

The decode engine’s arithmetic, bit for bit.

Every number the engine produces is defined here as a sequence of exact integer operations and f32 operations, so the hardware is held to these functions exactly and these functions are held to the model’s own arithmetic within a measured bar.

★ THE ENGINE NEVER ROUNDS A WEIGHT. A block format stores integer codes and a few scales; every weight is a scale times an integer. Activations are quantized under scales m·2^e, the codes the exact nearest integers of x / scale, in one of two ways (ActScaling): • per block of 32 under a 4-bit significand (m in 8..15): a matrix row times an activation row is, within a group of 256 elements, a sum of products of a few small integers and a binary16 significand with integer dot products, taken EXACTLY, aligned to the group’s least exponent in a wide integer, the group rounded to f32 once; • per weight block (the matrix format’s own block, 32 or 256 elements) under an 8-bit significand (m in 128..255), llama.cpp’s own blocking: each weight block’s two terms — its scale times the block’s integer sum, its second scale times the offset terms’ — summed exactly and rounded to f32 once, which is what the array’s block-scale lanes compute at the array’s rate. Either way the rounded pieces of a row are added in order.

f32 arithmetic is IEEE binary32, round to nearest even, with denormal operands and results flushed to zero keeping their sign (ftz); multiply and add round separately. Dot products and sums over a vector run in LANES lanes (element i in lane i mod LANES, in index order) folded by a balanced pairwise tree. The transcendentals are fixed sequences of those operations over fixed tables.

Structs

ActRow

An activation row as the array reads it: a whole-number code per element, worth code·m·2^e of its block of block elements.

ActScale

An activation block’s scale m·2^e: m in 8..15 per ActScaling::Sub32 block, in 128..255 per ActScaling::PerBlock block.

CodedRow

A row of a block-format matrix as the engine reads it: every weight is a·coef + b·offs of its segment.

Logged

One logged token as a lazily flushed head reads it: its key group’s key, the head’s values, the exponent of its decay and its update gate, as the log stores them.

Scale

A scale stored in a block: sig·2^exp, exactly.

Segment

A run of a row’s elements under one pair of scales: the elements’ coefficients under the first scale (codes times their sub-block scales) and under the second (the offset terms), as stored, exactly.

Unreadable

A block a format stores that the engine cannot read: an infinite or NaN scale.

Enums

ActScaling

How an activation row is scaled, and so how a matrix row’s dot with it rounds.

FpUnit

The multiplier an f32 product is computed by. Its add is the engine’s either way: the card’s block adds as fadd does.

Words

Words an activation crosses the array in.

Constants

ACT_BLOCK

Elements that share one activation scale: a ggml sub-block, so a block’s scale meets a weight sub-block’s.

COEF_BOUND

The largest magnitude a coded row’s coefficient or offset takes in any format the array reads: Q6_K’s −128·−32.

GROUP

Elements whose products round to f32 together.

INV_127

1/127, the int8 cache’s scale per unit of its row’s largest magnitude.

LANES

Lanes a dot product or a sum accumulates in before its tree.

NAN

The one NaN the engine produces: a quiet NaN of no payload and no sign.

SUM_WINDOW

Binary places an exact sum keeps below its largest term’s least place.

VALUE_BLOCK

Positions a block of the array’s weighted sum spans.

WIDE_BITS

Significand bits of a per-weight-block activation scale.

Functions

absmax

The largest magnitude in x, a denormal read as a zero, folded from +0; a NaN is passed over.

argmax

The index of x’s largest element, the first of equals (the two zeros equal), a NaN passed over; 0 when there is none. Lanes each keeping their first largest and folded to the larger, the lower index of equals, agree with it in any order.

attend_int8

One query head’s attention over its cached int8 rows on the array: the query quantized as words and scaling name, each position’s int8_score times scale, the softmax, the weighted_int8 sum.

attend_vector

One query head’s attention over its cached rows on the vector units: each position’s score the head’s dot with the key times scale, the softmax of the scores, and each value element the probabilities’ dot (fdot, lanes by position) with that element across the positions.

block_scale

The least scale m·2^e (m in 8..15) with 127·m·2^e ≥ amax, for a finite positive amax.

cache_row

A cache row stored as int8 codes under one f16 scale amax/127 (absmax): the scale’s bits and the codes under its reciprocal (int8_codes), all zero when the scale rounds to zero.

conv

A Gated-DeltaNet block’s causal convolution through SiLU, channel by channel: the ring’s rows, oldest first, against the channel’s first taps, then the token’s row x against its last, the products summed in tap order from +0. kernel holds each channel’s taps weights together.

dot

The f32 the engine produces for row · act.

exact_sum

Σ v·2^e over terms, rounded once to f32: exact while every term lies within SUM_WINDOW places of the largest term’s least place, a term past the window floored at it, however far past; no terms is +0.

exp

e^x: x = n·ln2/64 + r with n the nearest multiple, e^r by its cubic, times 2^((n mod 64)/64) from the table, times 2^(n div 64).

f32_to_f16

x rounded to binary16, to nearest with ties to even, subnormals kept, past the top infinite.

fadd

fdot

Σ a_i·b_i in LANES lanes, then the tree.

fmax

The largest of x, folded from −∞; a NaN is passed over, and a largest value that is a zero is +0 whichever zeros it was folded from, so lanes folding in any order agree.

fmul

fsub

fsum

Σ a_i in LANES lanes, then the tree.

ftz

x with a denormal flushed to a zero of its sign.

gdn_flush

A lazily flushed state row’s flush: every logged update replayed over the row in order (gdn_update with value r of each token), exactly as the token-by-token memory makes them, and the row stored once through store.

gdn_fold

A query folded back through a lazily flushed head’s log, newest token first: per token, its gate times the key’s dot with the folded query; that coefficient times the token’s values collected into width sums; the query decayed past the token’s update, y ← decay·(y − coef·k). Returns the folded query and the collected values.

gdn_fold_trailing

gdn_fold over a log whose oldest older tokens a trailing flush is replaying, in one walk: first through the newer tokens alone, the fold a row the flush has replayed reads, then on through the older ones, the fold every other row reads. The walk is newest first, so the first is where the second has reached after the newer tokens: each is gdn_fold over its own tokens, bit for bit.

gdn_read

A lazily flushed state row’s read-out through its head’s folded query: (row·y + carried)·scale.

gdn_row

One value row of a Gated-DeltaNet memory through one token, in one pass over the row: its dots with the token’s key and query, the rank-one update (gdn_update), and the row’s read-out (decay·(row·q) + coef·(k·q))·scale of the row as it was. kq is the head’s k·q.

gdn_update

One value row of a Gated-DeltaNet memory through one token’s rank-one update: coef = beta·(v − decay·(row·k)), each element decay·m + coef·k written back through store; returns coef.

hadamard

The Walsh–Hadamard transform of x in place, unnormalized: for each stride h from 1 doubling to len/2, every pair (x[j], x[j+h]) of each 2h-block becomes (x[j] + x[j+h], x[j] − x[j+h]), in index order. Applied twice it is len times the identity, so len a power of two makes the inverse exact: the transform, then 2^-log2 len.

int8_codes

The int8 codes of x under the inverse scale inv: each x·inv to the nearest integer, ties to even, within ±127, a NaN product zero; every code zero when inv is infinite, the reciprocal of a scale that rounded to zero.

int8_score

A score as the array computes it against a cached int8 key row (its f16 scale’s bits and its codes): per activation block of the query, its scale’s m times the integer dot of its codes and the key’s codes, times the key’s scale, summed exactly and rounded once.

l2norm

x / √(Σx² + ε).

ldexp

x·2^n, exact while the result is a normal f32: infinite past the top, a zero of its sign below the bottom.

ln

ln x for x > 0: x = m·2^E with m in [3/4, 3/2), so a value near 1 keeps E = 0; m against the nearest table point c_j, and ln(m/c_j) by its quartic.

ln1p

ln(1 + t) for t ≥ 0: ln u of u = 1 + t rounded, corrected by the rounding’s own error (t − (u − 1))/u, which is exact.

quantize

x (a whole number of groups) quantized per block of ACT_BLOCK.

quantize_blocks

x (a whole number of blocks) quantized per weight block of block elements.

quantize_head

A query head as the array reads it: per 32 elements (the head padded with zeros to a whole group), or the whole head one block.

reads

Whether the engine’s array reads matrices of ty.

recip

1/x: the seed from the top seven mantissa bits, then two Newton steps y·(2 − x·y).

rmsnorm

RMSNorm: x_i · rsqrt(Σx²/n + ε) · w_i, the product in that order.

rne

x rounded to the nearest whole number, ties to even, as an integer.

rne_div

The nearest integer to x / (m·2^e), ties to even, exactly.

rope

A head turned by a rotary table: pair i of the first 2·cos.len() elements — (i, i + cos.len()) under NEOX, (2i, 2i + 1) otherwise — becomes (x₀·cos_i − x₁·sin_i, x₀·sin_i + x₁·cos_i); the rest pass through.

rsqrt

1/√x for x ≥ 0: the seed from the exponent’s parity and the top six mantissa bits, then two Newton steps y·(1.5 − ½·((x·y)·y)), every intermediate near 1 so none flushes at the ends of the range.

scaled_to_f32

v·2^exp rounded once to f32, to nearest with ties to even, a denormal flushed to a zero of its sign.

sigmoid

silu

softmax

The softmax: the maximum (fmax), each exp(s − max), their sum in lanes (fsum), one reciprocal, the products.

softplus

ln(1 + e^u): u past 16 adds e^-u.

tree

Folds lane partials by a balanced pairwise tree: adjacent pairs, then pairs of pairs.

weighted_int8

The weighted sum on the array over cached int8 value rows (each its f16 scale’s bits and its codes): each position’s probability times its row’s scale, quantized per block of VALUE_BLOCK positions to 8-bit codes at a power-of-two scale; per value element the block’s exact integer sum against the value codes, rounded once a block; the blocks added in order.

wide_scale

The least scale m·2^e (m in 128..255) with 127·m·2^e ≥ amax, for a finite positive amax.