Struct PassShape 

pub struct PassShape {

    pub rows_per_sequence: u32,
    pub emitted_per_sequence: f64,
    pub weight_bytes: f64,
    pub embed_bytes: f64,
    pub read_bytes: f64,
    pub write_bytes: f64,
    pub rewrite_bytes: f64,
    pub array_macs: f64,
    pub vector_macs: f64,
    pub scale_cycles: f64,
    pub held_shared: f64,
    pub held_per_sequence: f64,
    pub on_chip_per_sequence: f64,
    pub provenance: Provenance,
}

A pass as a function of its batch. Everything a pass costs is shared by the batch or paid by each sequence, so the cost is affine in the batch and in how the batch splits between the card’s memory and the host’s.

Fields

rows_per_sequence: u32

Token-rows each sequence carries through a pass: its verified token and its drafts.

emitted_per_sequence: f64

Tokens each sequence emits per pass, expected.

weight_bytes: f64

Weight bytes a pass streams, whatever its batch.

embed_bytes: f64

Embedding-table bytes a pass reads for each sequence: a row per verified row and per drafting step.

read_bytes: f64

State, cache, log and ring bytes a pass reads and writes for each sequence.

write_bytes: f64

rewrite_bytes: f64

Of those, the bytes of each sequence’s recurrent states read and written back in place; the rest are read in long runs with the pass’s rows appended after them.

array_macs: f64

vector_macs: f64

scale_cycles: f64

Block-scale lane cycles a pass spends for each sequence.

held_shared: f64

Bytes every serving of the checkpoint holds: the weights and the embedding table.

held_per_sequence: f64

Bytes each sequence holds: its states, its log, its ring, its caches.

on_chip_per_sequence: f64

Bytes each sequence’s token-rows take on chip through a layer.

provenance: Provenance

The weakest acceptance the shape rests on.

Implementations

impl PassShape

pub fn of(h: &GgufHeader, s: &Serving) -> Result<PassShape, ServingRefusal>

The shape of a pass of s over the checkpoint h describes. The batch and the host split are the cost’s arguments; everything else in s is read here.

pub fn cost( &self, card: &Card, batch: u32, host_sequences: u32, ) -> Result<ServingCost, ServingRefusal>

The pass of batch sequences, host_sequences of them held in host memory, on card.

pub fn cost_split( &self, card: &Card, batch: u32, host_memory: bool, ) -> Result<ServingCost, ServingRefusal>

The pass of batch sequences split between the card’s memory and the host’s as fast as the split allows: the card’s memory streams far faster, so it holds sequences until its own traffic would outlast the host link’s.

pub fn best( &self, card: &Card, host_memory: bool, ) -> Result<ServingCost, ServingRefusal>

The batch that makes card emit the most tokens per second, and its pass. Ties go to the smaller batch.

Trait Implementations

impl Clone for PassShape

fn clone(&self) -> PassShape

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for PassShape

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for PassShape

fn eq(&self, other: &PassShape) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Serialize for PassShape

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl StructuralPartialEq for PassShape