Skip to main content

decode_step

Function decode_step 

Source
pub fn decode_step(
    spec: &ModelSpec,
    dir: &str,
    ctx: usize,
    tok: usize,
    pos: usize,
) -> Authored
Expand description

ONE DECODE STEP: token tok at position pos (1-based) in, the next token out. Every attention layer appends this token’s key and value to its KV cache (ctx positions, the step’s carried state alongside any recurrent memories and conv rings) and attends over the first pos rows; the step ends with the arg-max of the logits.