Module lane 

The compiled generation lane: a model’s decode step, written in LOGOS from its ModelSpec and compiled to one CUDA program that loads the checkpoint, runs the prompt and generates.

The path is the compiler’s own, end to end, and nothing on it is written for one model:

1. author::decode_step writes the step as a LOGOS program — the token and position in, every layer’s KV cache, convolution ring and delta-rule memory carried, the next token out — a cut of the model’s one program (program), whose identity the emitted program prints above the text it compiled;
2. the shared tensor front end lifts it into a StepGraph: every statement an op, or the compile refused by name;
3. weights places every constant — quantized projections as rows read in place from the checkpoint, the rest computed once into a sidecar;
4. plan gives each op a kernel from the kernels library, applies the fusion rules and lays out the buffers;
5. arena places every array a step computes and reads within the step in one arena, by its lifetime;
6. emit prints the program, each step captured as one CUDA graph and replayed per token or per chunk.

A lane has several forms of the step, compiled the same way: the decode step, and author::prefill’s chunks of 2, 4, … rows — the same layers over a block of prompt tokens, every projection reading its weights once for the block. They share the model’s constants and its carried state; the prompt runs as the largest chunks that fit.

The lane’s knobs are CudaKnobs’: activations picks the projections’ numerics — f32 activations with f64 sums, a two-word ~14-bit int8 activation (the default), or llama.cpp’s one int8 word — cuda_graphs captures each step, fuse_quant quantizes a projection’s activation in its producer.

Modules

arena

The step-local memory of a lane’s forms. Every array a step computes and reads within the step — a tensor’s f32 (or i32) buffer, or its quantized form — lives in one arena the program allocates once, placed by its lifetime: from the launch that first writes it whole to the last launch that touches it, through any view of it. A launch’s scratch lives there while the launch runs. The forms share the arena, one form running at a time.

branches

A form’s launches as a graph over several streams. A launch runs after every earlier launch that writes memory it reads or writes, or reads memory it writes — memory by where it lives: a step-local array or a launch’s scratch by its bytes in the arena (which several tensors share in turn), anything else by its buffer, an alias — a view — where its array lives (arena::home). Each launch takes a stream: the stream whose last launch it follows, or one whose work it already follows, or the least recently used; it waits on another stream’s launch by an event only where nothing it already follows orders the two. The launches compute what they compute in order, every dependency kept.

emit

The lane’s forms printed as one CUDA program, and the sidecar of values it reads.

graph

The step graph: an authored step (a decode step, a prefill chunk) as the ops it runs and the tensors they touch, each tensor given the role a compiled lane needs to place it.

kernels

The lane’s kernel library: one CUDA kernel per kind of op a step runs, each templated over the forms the fusion rules ask of it (a residual added before a norm, an activation quantized in a producer’s epilogue, a result written into the row of a cache a position names). The planner instantiates them; no kernel here knows a model.

plan

A step graph lowered to launches of the kernel library. The lane decides no fusion of its own: the step, lifted into the one IR, is optimized by the shared table under the lane’s shim (LaneShim), which answers whether a fused class has a launch; each op the optimizer leaves takes the launches its link names (link), a class the lane runs whole one launch, a class it runs as its parts each part’s. Then the lane’s schedule sets how the launches run — four rows a warp for a wide product (rows_a_warp), the shared-row attention, the split counts, a batch’s sequences as one launch (sequences) — and every tensor is given its buffers.

residence

Where a lane’s arrays live. Every array the program allocates goes on the device while they fit the memory it is given; past it, constant matrices move to pinned host memory the device maps, in the order that costs a step least: a table only ever gathered from (a row a token), then each expert stack from the last layer’s back (a step reads its chosen experts’ rows), then the other matrices from the last read back — until the rest fits. A kernel reads a mapped matrix through the same pointer it reads one in device memory, so where a matrix lives changes no value. Matrices that read the same checkpoint rows (a tied head and the embedding table it is) share one upload.

weights

Where each constant of a compiled step comes from.

Structs

Form

One form of a lane: the step of rows rows (1 for the decode step; a batch’s, one a sequence), its LOGOS source, its graph, its schedule and where its step-local arrays live in the program’s arena.

Lane

A compiled lane: its forms, the decode step first, and the one program that runs them.

LaneOptions

How a lane is compiled: its caches’ length in positions, its knobs, and its largest prefill chunk (a power of two up to plan::MAX_CHUNK; 1 runs the prompt a decode step at a time).

Enums

FormKind

What a form of a lane computes: the decode step, a prefill chunk of one sequence’s tokens, or a batch’s step of one token for each of its sequences.

LaneError

Why a model’s step does not compile to the lane, by name.

Functions

batch_io

The names an authored batch of rows sequences reads and writes across steps: its tokens, one a sequence, and sequence b’s position pos_s<b>, logits logits_s<b> and next token next_s<b>.

compile

Compile the lane of spec against the checkpoint at checkpoint: its decode step and its prefill chunks.

compile_batch

compile with a batch step beside the decode step and the chunks: the model’s batch cut (Cut::Batch) of batch sequences, one token each a step, every weight read once for all of them, each sequence its own position, caches and next token. The program decodes up to batch prompts together, each prefilled into its own sequence’s state.

compile_batch_holding

compile_batch with the fusion rules held held back.

compile_holding

compile with the fusion rules held (of plan::RULES) held back, their launches kept apart: the program each fusion is checked against, computing what the fused one computes.

decode_io

The names the authored decode step reads and writes across steps.

prefill_io

The names an authored prefill chunk reads and writes across steps.