Enum Cut 

pub enum Cut {
    Decode {
        ctx: usize,
    },
    Prefill {
        rows: usize,
        ctx: usize,
    },
    Batch {
        rows: usize,
        ctx: usize,
    },
    Forward,
    ForwardLayer {
        layer: usize,
    },
    ForwardHead,
    DecodeEmbed,
    DecodeLayer {
        layer: usize,
        ctx: usize,
    },
    DecodeHead,
    BatchEmbed {
        rows: usize,
    },
    BatchLayer {
        layer: usize,
        rows: usize,
        ctx: usize,
    },
    BatchHead {
        rows: usize,
    },
    PrefillLayer {
        layer: usize,
        rows: usize,
        ctx: usize,
    },
    PrefillHead {
        rows: usize,
    },
}

Which cut of a model’s program a backend lowers.

Variants

Decode

One decode step over a cache of ctx positions (author::decode_step).

Fields

ctx: usize

Prefill

A prefill chunk of rows tokens of one sequence over a cache of ctx positions (author::prefill).

Fields

rows: usize

ctx: usize

Batch

One decode step of rows sequences at once, each with its own state (author::decode_batch).

Fields

rows: usize

ctx: usize

Forward

The single token’s forward with no history (author::forward).

ForwardLayer

Layer layer of the forward alone (author::layer).

Fields

layer: usize

ForwardHead

The forward’s final norm and head alone (author::head).

DecodeEmbed

A decode step’s embedding lookup alone (author::decode_embed).

DecodeLayer

Layer layer of a decode step alone (author::decode_layer).

Fields

layer: usize

ctx: usize

DecodeHead

A decode step’s final norm, head and arg-max alone (author::decode_head).

BatchEmbed

The embedding lookups of rows stacked rows alone (author::decode_batch_embed).

Fields

rows: usize

BatchLayer

Layer layer of a batch of rows sequences alone (author::stacked_layer, each row its own sequence).

Fields

layer: usize

rows: usize

ctx: usize

BatchHead

A batch’s head alone, every sequence’s next token (author::decode_batch_head).

Fields

rows: usize

PrefillLayer

Layer layer of a prefill chunk of rows tokens alone (author::stacked_layer, one sequence’s rows).

Fields

layer: usize

rows: usize

ctx: usize

PrefillHead

A prefill chunk’s head alone, read off its last row (author::prefill_head).

Fields

rows: usize

Implementations

impl Cut

pub fn write(self, spec: &ModelSpec, dir: &str) -> Authored

The program author writes for this cut of spec’s computation, its weight manifest naming files under dir (the text does not depend on dir).

pub fn write_split( self, spec: &ModelSpec, dir: &str, tp: usize, ) -> Result<TpAuthored, NotYet>

The program author writes for this cut of spec’s computation split over tp chips, its manifest under dir: a decode step, a batch, or one layer of either. Any other cut, or a width the model does not split over, is refused.

Trait Implementations

impl Clone for Cut

fn clone(&self) -> Cut

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Cut

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Display for Cut

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Hash for Cut

fn hash<__H: Hasher>(&self, state: &mut __H)

Feeds this value into the given Hasher.

fn hash_slice<H>(data: &[Self], state: &mut H)

where H: Hasher, Self: Sized,

Feeds a slice of this type into the given Hasher.

impl PartialEq for Cut

fn eq(&self, other: &Cut) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Copy for Cut

impl Eq for Cut

impl StructuralPartialEq for Cut