Struct Reference 

pub struct Reference {
    pub spec: ModelSpec,
    /* private fields */
}

The reference model over a checkpoint.

Fields

spec: ModelSpec

Implementations

impl Reference

pub fn new( spec: ModelSpec, g: impl WeightSource + 'static, ) -> Result<Reference, ReferenceError>

pub fn with_activations(self, mode: Activations) -> Reference

The same reference with its quantized projections reading mode’s activations.

pub fn with_kv_storage(self, kv: KvStorage) -> Reference

The same reference keeping its attention caches’ rows as kv stores them.

pub fn with_state_storage(self, state: StateStorage) -> Reference

The same reference keeping its gated-delta memories as state stores them.

pub fn with_cache_budget(self, bytes: usize) -> Reference

The same reference keeping at most bytes of tensor data in memory.

pub fn position(&self) -> usize

The number of positions run so far.

pub fn cached(&self, l: usize) -> (&[Vec<f64>], &[Vec<f64>])

Layer l’s cached keys and values, one row per position (empty for a layer that is not attention).

pub fn memory(&self, l: usize) -> &[f64]

Layer l’s gated-delta memory (empty for a layer that is not gated delta).

pub fn conv(&self, l: usize) -> &[Vec<f64>]

Layer l’s convolution ring: its last taps − 1 inputs, oldest first (empty for a layer that is not gated delta).

pub fn reset(&mut self)

Forget every position: empty caches, zero recurrent state, no routing or near calls. The calls decided the other way stay decided so.

pub fn near_calls(&self) -> &[NearCall]

Every rounding and routing call since the last reset made near its other outcome, in order.

pub fn flip(&mut self, call: NearCall)

Decide a near call the other way at every run past it: a rounding takes the other level, a routing call keeps the first expert it would pass over in place of the last it would keep.

pub fn closest_routing(&self) -> Option<RoutingCall>

The closest routing call since the last reset — None for a model without experts.

pub fn flip_routing(&mut self, call: RoutingCall)

Decide one routing call the other way at every run past it: at call’s layer and position the router keeps the first expert it would pass over in place of the last it would keep — how a run in another precision that resolves the call the other way routes.

pub fn step(&mut self, tokens: &[u32]) -> Result<Vec<f64>, ReferenceError>

Run tokens at the next positions, appending them to the state; returns the logits at the last one.

pub fn step_every( &mut self, tokens: &[u32], ) -> Result<Vec<Vec<f64>>, ReferenceError>

Run tokens at the next positions, appending them to the state; returns the logits after every one of them — the distributions a perplexity or a divergence over a text is taken from.

pub fn generate( &mut self, prompt: &[u32], n: usize, ) -> Result<Vec<u32>, ReferenceError>

Greedy continuation: the prompt in one step, then n tokens one step each.