Skip to main content

emit_qwen30b_generate

Function emit_qwen30b_generate 

Source
pub fn emit_qwen30b_generate(
    cfg: &Qwen30bCfg,
    wdir: &str,
    prompt: &[usize],
    n_gen: usize,
    precise: bool,
    flash: bool,
) -> String
Expand description

Emit a self-contained CUDA program that AUTOREGRESSIVELY GENERATES from the real Qwen3-30B-A3B weights: the per-token math is the recognized LOGOS forward (RMSNorm → Q/K/V → per-head QK-norm + RoPE → causal attention over a KV cache → O-proj → residual → RMSNorm → 8-of-128 MoE → residual → LM head), wrapped in the compiler’s generation runtime — a prompt prefill, then a T=1 decode step captured as ONE CUDA graph and replayed per token (device-side position + token). This is the “llama.cpp, but authored in LOGOS” whole demo: it prints the generated token stream and the fair tg-style tok/s (a growing KV cache, exactly like llama-bench tg128).