Module residency 

Residency — which tensors the SRAM holds, and what leaves the chip.

★ THE SRAM IS THE PMUS’ MEMORY, AND IT IS ALLOCATED. A spatial chip’s SRAM tier is the sum of its sites’ local memories: every buffer a program lays out takes a share of it, and a tensor held there between sections — an intermediate one section writes and a later one reads, a state one step leaves for the next — takes more. It is only a tier a tensor can LIVE in where any site reaches any other’s memory (a crossbar, the RDU); a target whose sites share data through a global memory (Tensix, a GPU) hands intermediates through that memory.

Two decisions, at the two scopes that can make them: • a PROGRAM places its intermediates (place_activations): each lives from the phase that writes it to the last phase that reads it, and the live set of every phase must fit the SRAM its buffers leave — past it, the intermediate whose last use is furthest away spills first (linear-scan allocation over live ranges); • a DEPLOYMENT places the carried state (plan_states): a step’s state outlives the program, every layer holds its own, and only the whole model knows what else is resident. The state that moves the most bytes per byte held — a recurrent memory read and written every step before a cache read once — is kept first, within each chip’s SRAM less the largest program’s own footprint. What leaves the SRAM goes to the next tier down (plan_off_chip): an RDU’s HBM while it holds it, densest first, then the DDR behind it (Dram); a chip with no HBM keeps it all in its DRAM.

Structs

ResidencyPlan

Where a deployment keeps its carried state: each chip’s SRAM, the reserve its largest program takes while it runs, and every state class with the tier it was given.

StateClass

A carried state of a deployment: tensor of the program program, held on chip chip once for each of the instances layers that run the program, each bytes in memory and moving moved bytes a step.

Functions

buffer_footprint

The SRAM each chip’s buffers take: over every site of the chip, the highest byte its local memory map reaches.

holds_tensors_on_chip

Whether target can hold a tensor in its SRAM between sections: an SRAM tier any site reaches.

moved_bytes

Bytes each tensor of p moves a step, every loop at its bound.

place_activations

★ PLACE THE INTERMEDIATES. Where target holds tensors on chip, each chip’s intermediates are allocated over their live ranges in the SRAM its buffers leave; where it does not, or past what fits, they stay in the tier they were declared in. A live set past the SRAM spills, at the first phase it overflows, the intermediate live there whose last use is furthest away — then the largest, then the lowest id — until every phase fits.

plan_deployment

A deployment’s carried state planned and placed: every CARRIED State tensor of every program becomes a StateClass, once per layer that runs the program; the reserve is each chip’s largest footprint while a program runs — sram_footprint and the program’s own tables, kept on chip when the program has room for them — and every carried state takes the tier plan_states gives it.

plan_off_chip

★ THE NEXT TIER DOWN. Each class the SRAM does not hold goes, in the order plan_states ranks by, to its chip’s HBM while hbm_bytes holds it whole, and to the DDR (Dram) past it; with no HBM, all of it to the DRAM.

plan_states

★ KEEP WHAT MOVES MOST PER BYTE HELD. In order of bytes moved a step per byte held — then the smaller class, then by name — each class goes on chip whole if its chip has room left beyond reserve, and stays in HBM/DDR (Dram) otherwise; a class that does not fit leaves the room to the classes after it. Every byte a class keeps on chip is a byte a step does not move off it, moved / bytes times over, so filling the SRAM densest-first is the most traffic whole classes can keep.

sram_footprint

The SRAM each chip of p holds while it runs: its buffers and the peak of its on-chip intermediates — the running sum of the live bytes over the phases, a range added at its first phase and taken away after its last.

tensor_bytes

Bytes t takes in memory: its tiles, whole.

Type Aliases

DeployedProgram

One program of a deployment: its name, the layers that run it, the program, and — when the lowering knows it — which of its State tensors a step carries to the next. A state the step assembles whole each time is the step’s own table: it lives while the program runs, like an intermediate, and is not the deployment’s to keep.