Module numerics 

Quantize insertion: every product that reads its activation quantized, as the chip’s shim asks, gets that activation’s quantized form — from its producer’s epilogue where the chip runs that whole (a *_quant class: the producer and the quantize as one op), or from a quantize op of its own right after the producer.

A product reading a row of a tensor in place at a whole block reads the tensor’s quantized form in place too, so the tensor is quantized once, at the root of its views, for every row a product reads of it. A tensor no op computes whole — a table assembled a row at a time — is quantized right before the first product that reads it, once its last row has landed; or, where each of its rows is set, once and in whole blocks, from a result its producer can quantize in its epilogue — a batch’s sequences’ rows — a row at a time by those producers, each set carrying its row’s quantized form into the table’s (PARITY-BATCH-ROW-QUANT).

Structs

AttentionNumerics

How an attention a chip computes on narrower units than its values takes its operands: its queries, its keys and values, and its weights — each weight times its value row’s scale where the values are quantized.

Bound

How far a value a chip keeps may lie from the value the meaning computes: of_block of the largest magnitude in its block, plus of_value of its own magnitude, plus floor.

Contract

What a chip computes at one knob point, and how far it may lie from the meaning: the scheme a product by a matrix in a block format reads its activation in (None, its values), how the chip keeps a cache’s rows and a recurrent memory, how an attention takes its operands where it narrows them (None, it attends on its values as the chip keeps them), how far a fused op may lie from its parts run apart (Bound::EXACT, bit for bit), and the widest integer a product or a sum of integers accumulates in.

Scheme

A quantized activation’s scheme: its blocks, how each block’s scale is chosen, and the int8 words each value takes — one, the nearest whole number of the scale; two, a high word and a low word whose code is the nearest whole number of a finer step (the high word’s rest at a 128th of the scale, or a 16-bit code at a 256th); three, a third word at a 128th of the second’s step.

Enums

Block

The values of a quantized activation that share one scale: a fixed count, the block of the matrix format that reads it, or the row the op takes it in — a head’s queries, a cached key or value row, a query’s weights over a tile of keys.

Keep

How a chip keeps the values it stores between steps — a cache’s rows, a recurrent memory: as the meaning computes them, rounded to the nearest of a narrower float, or a row at a time as int8 codes under one binary16 scale (max / 127), the row first turned by a Walsh–Hadamard transform where it is Int8RowsRotated.

Scale

How a block’s scale is chosen from its largest magnitude: that magnitude over 127 in f32; the least m·2^e with a significand_bits-bit m at least that; or, for values never negative, the least 2^e with the largest at most 255·2^e, its codes unsigned eight bits.

Taken

How an op takes a value: kept in a float format, or quantized in a scheme.

Constants

BLOCK

The values of an activation’s quantized form that share one scale.

Functions

holds_accumulator

Whether a chip whose integers accumulate in bits holds every partial result op accumulates: Ok, or why not — the bits its operands’ proven ranges need, or an operand no range bounds on a chip narrower than the language’s own 64-bit integers.

quantize

Give every product that reads its activation quantized that activation’s quantized form: in its producer’s epilogue where the chip runs the two as one op and fold allows it, otherwise as a quantize of its own.