Skip to main content

lower_tp_decode_streamed

Function lower_tp_decode_streamed 

Source
pub fn lower_tp_decode_streamed(
    spec: &ModelSpec,
    storage: Storage<'_>,
    ctx: usize,
    tp: usize,
    target: &SpatialTarget,
    fill: impl Fn(&str) -> String,
) -> Result<StreamedModel, StreamedReject>
Expand description

Lower spec as an SN50 TENSOR-PARALLEL decode deployment over tp chips: the embedding lookup and the head on the replicated spine, and each layer kind one program over the chips — every chip its own heads and FFN columns, the partials summed by an all-reduce twice a layer, the chips’ slices of each op in one phase (see spatial::tensor_parallel). tp must be a width the model admits (fit::admits_tp); each kind program carries its manifest’s slices, so a host knows which rows or columns of which checkpoint tensor each chip loads.