Struct Attention 

pub struct Attention {

    pub heads: usize,
    pub kv_heads: usize,
    pub head_dim: usize,
    pub value_dim: usize,
    pub rope: Option<Rope>,
    pub qk_norm: bool,
    pub output_gate: bool,
    pub fused_qkv: bool,
    pub bias: bool,
    pub output_bias: bool,
    pub window: Option<Window>,
    pub sinks: bool,
    pub softcap: Option<f64>,
    pub scale: f64,
    pub temperature: Option<Temperature>,
    pub qk_unit_norm: bool,
    pub qk_norm_whole: bool,
}

Grouped-query attention, with the variants checkpoints use.

Fields

heads: usize

kv_heads: usize

head_dim: usize

Query/key head width.

value_dim: usize

Value head width.

rope: Option<Rope>

The rotary embedding, or None for a layer without positions (NoPE).

qk_norm: bool

Per-head RMSNorm of the queries and keys before the rotation.

output_gate: bool

The query projection also produces a per-element gate, and the attention output is multiplied by its sigmoid (Qwen3.5/3.8).

fused_qkv: bool

The projections are one fused tensor (attn_qkv) rather than three.

bias: bool

Biases on the query, key and value projections.

output_bias: bool

A bias on the output projection (attn_output.bias): gpt-oss’s; Qwen2’s attention has none.

window: Option<Window>

The positions each query keeps: the last W (Mistral, Gemma 2 and 3, gpt-oss), or its own chunk of C (Llama 4’s rotated layers).

sinks: bool

A learned per-head logit joins each softmax’s denominator (gpt-oss).

softcap: Option<f64>

Soft-capping of the attention scores.

scale: f64

The score scale (1/√head_dim unless the checkpoint says otherwise).

temperature: Option<Temperature>

Each query scaled by a factor of its position before it scores (Llama 4’s rotation-free layers).

qk_unit_norm: bool

A per-head RMSNorm of the queries and keys with no weight (Llama 4’s rotated layers, after the rotation: the same as before it, a rotation keeping each head’s norm).

qk_norm_whole: bool

With qk_norm, the queries and the keys are each normed across every head at once, their weights as wide as the projections (MiniMax-M2’s qk_norm_type “per_layer”), rather than head by head.

Trait Implementations

impl Clone for Attention

fn clone(&self) -> Attention

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Attention

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for Attention

fn eq(&self, other: &Attention) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl StructuralPartialEq for Attention