Module decode 

The cost of one token: batch-1 decode, priced on a machine from the checkpoint itself.

★ AT BATCH ONE, EVERY TOKEN REREADS EVERY WEIGHT. A decode step multiplies one activation vector through every projection of the model, so its arithmetic intensity is one multiply-add per weight byte read and the matrix engines idle: the step is bounded by where the weights live and how fast that memory streams, and by how long its chain of dependent stages takes to walk. That is the regime a GPU is worst at and an SRAM-resident dataflow machine is built for, so it is the regime this module prices for both, by one rule.

The step is read off a GGUF header (GgufHeader): every tensor’s bytes from ggml’s block arithmetic (GgmlType), every tensor’s role from llama.cpp’s tensor names, the KV cache and the recurrent state from the header’s metadata. Nothing is per model: a block’s mixer is attention when it carries attention projections and a recurrence when it carries ssm_* tensors; a tensor no role names is refused by name.

The machine is facts with sources (Chip), by the rule the roofs keep: a GPU’s rates are its roofs preset’s, its capacities and links its vendor’s table’s. What the record does not give is absent, never guessed — an undeclared per-stage overhead or link latency leaves the FLOOR priced (it is a true lower bound with the term at zero) and the PREDICTION refused.

price lays the step’s stages onto the chips Megatron’s way — projections split, attention split by heads, norms replicated, an all-reduce after each block’s attention output and FFN down projection, an arg-max across vocabulary shards at the head — and returns both numbers: the floor no implementation can beat (the largest of the memory, compute and collective sums), and, where every term is declared, the prediction (each stage’s roofline plus its collective and overhead, summed along the chain).

Structs

Chip

A chip as a decode sees it.

DecodeCost

One token of a step on a machine.

DecodeStep

What one token of a deployment reads, writes and computes, as the chain of its dependent stages.

Deployment

What a deployment of a checkpoint holds per token.

GgufHeader

A GGUF header: the metadata and the whole tensor table, and where in the file the tensor data lives.

HeaderSource

A header ledger read as a checkpoint’s description: its metadata as GGUF values — an integer unsigned, or signed when negative; a decimal that is exactly a binary32 one, as the converters store metadata decimals, any other binary64; an array of its first item’s type; a value the ledger summarized (a vocabulary as <n strings>) the summary’s string — and its tensor directory.

Held

A class of resident bytes and the bytes of it one token moves: what decides which memory tier keeps it.

Link

The chip-to-chip network. Either rate may be undeclared: the floor then prices it at zero, which is a true lower bound, and the prediction refuses.

Machine

A number of identical chips running one step, tensor-parallel.

Stage

One dependent stage of the decode chain.

StageCost

One stage’s terms, in seconds, on the machine it was priced for.

TensorRow

One row of a GGUF tensor table.

Tier

One memory tier of a chip: how much it holds and how fast it streams.

Enums

Bind

The term that binds.

DecodeRefusal

Why a step or a price could not be computed.

GgmlType

A ggml tensor storage type: the id GGUF stores and the block it packs elements into.

HeldClass

Split

How far a stage’s work divides across chips.

StageKind

What a stage of the chain does.

TierKind

A kind of memory a chip holds bytes in.

WeightBytes

How the matrix weights are stored for the run.

Constants

GGML_TYPES_SOURCE

Where ggml’s block arithmetic is read from.

Functions

price

Prices step on m, with matrix arithmetic at precision.