Module from_q8 

Q8_0 weights on the chip — a product y = x · W of up to a tile’s rows over a weight the deployment stores Q8_0.

★ A DECODE STEP IS WEIGHT-BOUND, SO A WEIGHT MOVES AS IT IS STORED. Q8_0 keeps each column’s rows in blocks of 32: one scale d and 32 int8 values q, w = q·d. The weight is two tensors on the chip — W__q8, the int8 values (K × N), and W__q8_scale, one fp32 scale per block (K/32 × N) — about 1.13 bytes a weight where a bf16 copy moves 2. The product FOLDS its K blocks of 1,024 rows — 32 Q8_0 blocks, one tile row of scales — into an fp32 carry: per block, ONE stage a row, y[n] += Σ_b s[b, n]·Σ_i x[32b + i]·q[32b + i, n] — the lanes contracted over the int8 tile rows, the block’s 32 Q8_0 blocks folded down each column with their scales. An int8 times a bf16 activation is exact in fp32, and the scale is fp32.

x is read as the K/32 × 32 matrix of its 32-element blocks — one tile per K block — a view named for the value it views (q8_rows_name), made once and read by every product of that value. A BATCH of m rows (up to a tile’s) is m such views interleaved by K block, and the fold runs each row’s block dots against the SAME int8 and scale pages: the weight moves once for all the rows.

Enums

Q8Part

Which part of a Q8_0 weight a stored tensor holds — what the host repacks from the checkpoint’s blocks into it.

Constants

Q8_BLOCK

Rows of a Q8_0 block.

Functions

q8_gemv

y = x · W for an m-row x and a Q8_0 W: the reshape of x into its blocks, then the fold — or None when the fold would move more than reading the weight dense.

q8_gemv_applies

Whether a product over a Q8_0 weight folds on target: up to a tile’s rows, whole Q8_0 blocks of K, on a square tile as tall as a Q8_0 block. Its columns are the weight’s column tiles, the last padded with zero columns as every tensor’s last tile is — a tensor-parallel chip’s 24 of a gate’s 48 heads, a width of 40 — so the pad’s values and scales are zero and its sums land only in the result’s own pad.

q8_rows_name

The K/32 × 32 view of the one-row activation x a Q8_0 product reads — x’s 32-element blocks as the rows of a matrix, one tile per K block. It is named for the VALUE it views: x after the program has written it writes times. Every product of one value reads one view, and a value written again is viewed again under a new name (an activation has one writer).

q8_rows_name_on

q8_rows_name for a product that runs on chip of a tensor-parallel deployment: a chip reads only what lives on it, so each chip makes its own view of a replicated value, shared by that chip’s products.

stored_name

The tensor holding part of the Q8_0 weight weight.

stored_part

The Q8_0 weight a tensor named by stored_name is part of, and the part.