Module serving 

The cost of a pass: batched, speculative decode on one card that holds its tokens and streams its weights.

★ A CARD THAT HOLDS ITS TOKEN-ROWS AND STREAMS ITS WEIGHTS PAYS FOR EACH WEIGHT ONCE A PASS. Every weight crosses the card once and multiplies every token-row the card holds: each sequence’s verified token and the tokens drafted after it. What does not amortize is per sequence: the recurrent state it reads and writes back, its KV cache, its conv ring. A pass therefore costs the weights once plus each sequence’s own bytes, against the arithmetic of every held token-row, and the batch grows until the card’s memory is full.

The pass is read off the same DecodeStep the batch-one price uses — the same stages and bytes, from the checkpoint’s header — plus what serving adds: how many sequences ride a pass (Serving::batch), how many drafted tokens each verifies and how often they are accepted (Draft), and how the recurrent state is carried between tokens (StateFlush). The card is facts with sources (Card): its memory, the shares of that memory’s bandwidth its stream engines sustain on each way a pass moves its bytes (Streams), the multiply-adds its integer array and its f32 vector units retire.

The array multiplies 4-bit weight codes by 8-bit activation codes. A wider integer code is split into 4-bit digits and a two-word activation into two words, each digit and word one pass, recombined exactly (array_passes), so every block format the card decodes rides one datapath. A float matrix rides the vector units. A format the card does not decode is refused by name, never priced as a guess.

Every weight block’s integer sums leave the array for a block-scale lane: its sub-block scales and mins multiplied in, the block’s two scales, the exact sum and its one rounding, the f32 accumulate (lane_cycles). The lanes are a fifth rate beside the memories and the two kinds of arithmetic: a format of small blocks, or activations scaled per 32 elements against large ones, asks more of them than of the array.

Structs

Card

A card that streams its weights, as a pass sees it.

ContextClass

A class of the batch’s sequences by context: its share of the batch, the mean position its sequences have reached, and the context each of them is allowed.

ContextMix

The contexts of a batch’s sequences. Every per-position cost is linear in position, so a pass reads, and a paged cache holds, what the mean position says; a contiguous cache holds what the mean limit says.

Draft

The drafted tokens each sequence verifies in a pass.

HostLink

The host memory a card reads and writes over its host link.

PassShape

A pass as a function of its batch. Everything a pass costs is shared by the batch or paid by each sequence, so the cost is affine in the batch and in how the batch splits between the card’s memory and the host’s.

Serving

One serving configuration of a checkpoint on a card.

ServingCost

One pass of a serving on a card.

Streams

The share of a card memory’s bandwidth its stream engines sustain on each way a pass moves its bytes. The ways differ in how often the memory’s bus turns between reading and writing and how often a row is opened for few bytes.

Enums

AttentionMath

Where attention’s two products over the cache run.

ServingBind

The term a pass is bound by.

ServingRefusal

Why a serving could not be priced.

StateFlush

How the recurrent state is carried from one token to the next.

Constants

LANE_PRODUCTS

Sub-block multipliers a block-scale lane applies in one cycle.

SOFTMAX_MACS_PER_SCORE

Vector multiply-add slots per score a softmax takes: the engine’s exponential is twelve f32 operations (decode_engine::numerics::exp: the scaling and its rounding, two for the range reduction, six for the cubic, one for the table product, one for the power of two), and the running maximum, the sum and the rescale three more; fifteen operations, two to a slot.

VALUE_BLOCK

Positions one block of attention’s weighted sum on the array spans: its probabilities share a scale.

Functions

array_passes

Passes of the 4-bit array one weight of ty takes, or None for a type the array does not read.

best_serving

The batch at which s emits the most tokens per second on card, and its pass.

emitted_per_pass

Tokens a sequence emits per pass: its verified token, plus each draft accepted after every draft before it.

is_float

Whether the vector units read ty as a float matrix.

lane_cycles

Cycles a block-scale lane spends on one weight block of ty read against one activation word scaled as scaling, or None for a type the array does not read.

max_batch

The most sequences s can carry on card: every larger batch is refused for capacity.

price_serving

Prices one pass of s over the checkpoint h describes, on card.