Module residence 

Where a lane’s arrays live. Every array the program allocates goes on the device while they fit the memory it is given; past it, constant matrices move to pinned host memory the device maps, in the order that costs a step least: a table only ever gathered from (a row a token), then each expert stack from the last layer’s back (a step reads its chosen experts’ rows), then the other matrices from the last read back — until the rest fits. A kernel reads a mapped matrix through the same pointer it reads one in device memory, so where a matrix lives changes no value. Matrices that read the same checkpoint rows (a tied head and the embedding table it is) share one upload.

The compiler fixes the order (plan); the program decides how far down it to go when it starts, from the device memory free once its context is up (or the --device-bytes it is given), less its Plan::spare — place is that decision, as the program makes it. The arena is an allocation of its own; every other array the program keeps on the device is carved from one pool, and every matrix it moves from one mapped host pool, each at POOL_ALIGN, so the bytes counted are the bytes the device gives up.

Structs

Plan

What a lane’s program allocates, before it decides where its matrices live.

Residence

Where the program’s matrices live on a budget.

Constants

CONTEXT_RESERVE

The device memory a chip loses before the program allocates: what the driver holds back from every program (379 MiB of the RTX 3060’s 12 GiB under driver 580) and the program’s context with its modules loaded (113 MiB).

POOL_ALIGN

The alignment of every array carved from a pool: cudaMalloc’s, the arena’s.

SPARE

What the program’s graphs, streams and events, and the rounding of its three allocations to the driver’s 2 MiB granule, take of the device beside its arrays, past SPARE_A_LAUNCH for each launch it captures.

SPARE_A_LAUNCH

The device memory each captured launch’s graph node takes, at twice the 8 KiB a node of a 1000-launch graph takes on the RTX 3060.

Functions

matrix_bytes

The bytes a matrix kept as rows takes of its pool: a Q8_0 matrix’s levels and its half scales, each an array; any other format’s blocks as the checkpoint lays them out and the 32 bytes a staged read runs past them.

place

Where plan’s matrices live on budget bytes of device memory: as many of its order as must move to mapped host memory for the rest to fit, and the bytes each side takes — the decision the program makes when it starts. Refused when the arrays but the matrices outrun the budget.

plan

What the program of forms allocates, its caches ctx positions long: the arena, the arrays it keeps on the device, its matrices in the order they leave it, and the spare its graphs take.

pooled

The bytes an array of n bytes takes of its pool.

shared_names

The constant buffers every form shares, allocated once: the constants, and the carried state the decode step holds.