Skip to main content

fold

Function fold 

Source
pub fn fold(jobs: &[Job], layout: &Layout) -> Vec<Job>
Expand description

The step jobs with each matrix whose sums only a residual add or a gated FFN’s product takes folding them into that job’s output slot (Fold): a matrix job and the add of its output into another slot become one job adding its sums into that slot; a gate’s and an up projection’s matrix jobs and their product become the gate’s sums into the product’s slot and the up projection’s folded into them through the gate’s SiLU. Jobs fold only where they stand next to each other and the slots the step no longer writes are read by no job after (StoreLayout::live_out), so every value the step computes is the one it computed, and those slots leave the store.