Struct CudaKnobs 

pub struct CudaKnobs {
    pub gemv_load: LoadWidth,
    pub precision: MatmulPrecision,
    pub kv_cache: bool,
    pub cuda_graphs: bool,
    pub fuse_quant: bool,
    pub activations: Q8MatKind,
    pub chip: CudaChip,
    pub recurrent_state: StatePrecision,
    pub cache_precision: CachePrecision,
}

The CUDA backend’s optimization knobs — one point in the lattice. Construct with CudaKnobs::default for the best-general point and flip individual knobs to trade off. Every field is a trade-off; there are deliberately no fields for strictly-better transforms (those are always applied, see STRICTLY_BETTER).

Fields

gemv_load: LoadWidth

Weight-load width for GEMV/GEMM. Default LoadWidth::Int4 (measured best-general on the projections).

precision: MatmulPrecision

Contraction precision. Default MatmulPrecision::Auto (⇒ always-correct int64).

kv_cache: bool

KV-cache for autoregressive generation — skip recomputing past K/V. A large win for generation, but costs VRAM and only applies to attention over a growing sequence. Default true.

cuda_graphs: bool

Capture the steady-state (T=1) step as a CUDA graph and replay it — collapses ~540 launches/token to one. Wins when many tiny kernels make launch latency dominate; neutral-to-negative for one big kernel. Default true.

fuse_quant: bool

Fuse activation-quantization into the preceding norm/elementwise kernel — saves a global round-trip at the cost of a slightly larger, less-reusable kernel. Default true.

activations: Q8MatKind

The activation precision of a projection by Q8_0 weights in the compiled generation lane: f32 activations with f64 sums (Dequant), a high and a low int8 word per value (TwoPassInt8, ~14 bits), or one int8 word (Int8Act, llama.cpp’s precision). Default Q8MatKind::TwoPassInt8.

chip: CudaChip

The GPU part this kernel targets — its SM arch, shared-memory budget, and realized precisions. Default CudaChip::rtx3060 (the local dev GPU). Flipping it to CudaChip::b200 retargets the SAME recognized op to Blackwell. A trade-off (portability vs hardware exploitation), hence a knob.

recurrent_state: StatePrecision

The precision the compiled generation lane stores a gated-delta layer’s recurrent memory in between steps: f32, or bf16 — computed in f32, rounded to bf16 after every row the delta rule walks. Default StatePrecision::F32.

cache_precision: CachePrecision

The precision the compiled generation lane stores an attention’s key and value cache in: f32, or binary16 — each row computed in f32 and rounded to the nearest as it is written, every read of the cache taking the rounded row. Default CachePrecision::F32.

Implementations

impl CudaKnobs

pub fn fabric_precision(&self) -> Precision

Map a MatmulPrecision to the fabric-level Precision descriptor (for FabricTarget gating).

pub fn tradeoffs() -> &'static [KnobTradeoff]

The trade-off table — one row per knob, each naming the regime where the aggressive setting wins and the regime where it loses (the reason it can’t be baked in unconditionally). This is the lattice’s map; the knob-lattice lock asserts it stays complete and honest.

Trait Implementations

impl Clone for CudaKnobs

fn clone(&self) -> CudaKnobs

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for CudaKnobs

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for CudaKnobs

fn default() -> Self

The best general point of the lattice: wide loads, always-correct precision, and the generation-time wins (KV-cache, graph replay, quant fusion) enabled — each the right call for the common case, each overridable.

impl PartialEq for CudaKnobs

fn eq(&self, other: &CudaKnobs) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Copy for CudaKnobs

impl Eq for CudaKnobs

impl StructuralPartialEq for CudaKnobs