Skip to main content

prefill

Function prefill 

Source
pub fn prefill(
    spec: &ModelSpec,
    dir: &str,
    ctx: usize,
    toks: &[usize],
    pos0: usize,
) -> Authored
Expand description

A PREFILL CHUNK: the tokens toks of ONE sequence at positions pos0, pos0 + 1, … (1-based) as one step, the next token after the last of them out. The hidden states are stacked into a T × d matrix, so every projection, norm, FFN and residual reads its weights once for the chunk; the chunk’s keys and values append to the cache as one block from pos0, and the rows attend as one causal block. The state is the decode step’s own, so a decode step continues where the chunk stops; the LM head reads the last row only.