Skip to main content

lower_prefill_streamed

Function lower_prefill_streamed 

Source
pub fn lower_prefill_streamed(
    spec: &ModelSpec,
    storage: Storage<'_>,
    ctx: usize,
    chunk: usize,
    target: &SpatialTarget,
    fill: impl Fn(&str) -> String,
) -> Result<StreamedModel, StreamedReject>
Expand description

Lower spec’s PREFILL as an SN50 deployment: chunk tokens of one sequence per step, the cuts of the model’s Cut::Prefill — the chunk’s embedding lookups stacked, one program per layer kind taking the rows in order over the sequence’s single state (the decode step’s own, so a decode bundle continues from it), and a head reading the last row. Each row’s token and position are runtime inputs.