Skip to main content

from_gated_delta

Function from_gated_delta 

Source
pub fn from_gated_delta(
    spec: &GatedDeltaSpec,
    names: &StageNames,
    dense: &BTreeSet<String>,
    target: &SpatialTarget,
) -> Result<Vec<SpatialProgram>, SpatialReject>
Expand description

The Gated DeltaNet step over hv value heads reading hk key heads at head dim d: u = S·k, S ← α·S + β·(v − α·u)·kᵀ, o = S·q / √d, RMSNorm’d, scaled by GN, gated by silu(z).

★ THREE SECTIONS, BECAUSE THE MEMORY IS READ ONCE AND WRITTEN ONCE. Section 1 reads S to form the prediction u; section 2 writes the new S from the old one and u; section 3 reads the new S to form the output. Each boundary is a barrier, so “read the old memory, then write, then read the new one” is enforced rather than hoped for, and StaleStateRead catches any scheduler that tries otherwise. Fusing them would make a wrong answer indistinguishable from a right one.

★ A BLOCK OF ROWS IS THE STEP, ROW AFTER ROW. The delta rule is a recurrence — row r reads the memory row r − 1 wrote — so a block of tokens is its rows’ three sections in turn against the one memory, each reading its row of the operands in place and writing its row of the output. dense names the block’s tables the program lays out as matrices; the output is written in its layout.