Module qwen38_batched 

════════════════════════════════════════════════════════════════════════════════════════════════ ★ BATCHED Qwen3.8-27B DECODE ★ — where the bandwidth wall stops being the wall.

Batch-1 decode reads all 27.3 GB of weights to produce ONE token. That is the entire reason it is bandwidth-bound, and the reason a 4% throughput difference is a hard-won result there. At batch B the same weight pass serves B sequences, so the per-token weight traffic falls by B and aggregate throughput scales close to linearly until the arithmetic takes over.

The whole trick lives in one kernel: load each weight int4 ONCE into registers and reuse it across the sequences in a batch tile, instead of re-reading the entire matrix per sequence. Everything else here is the single-sequence program with a sequence index threaded through it — the recurrence state, the KV cache and the conv ring become per-sequence, and the norms become one block per sequence. ════════════════════════════════════════════════════════════════════════════════════════════════

Constants

BTG

Sequences whose accumulators a single warp carries in registers. Above this the register file, not the bandwidth, decides the shape — so a larger batch is tiled over grid.y rather than over registers.

QWEN38_BATCHED_KERNELS

The batched device code. Every kernel is the single-sequence form with a sequence index added; the GEMV is the one that changes shape, because it is the one that decides the token time.

Functions

emit_qwen38_generate_batched

Emit the BATCHED decode program. batch sequences advance in lockstep, each with its own recurrence memory, conv ring and KV cache, sharing one pass over the weights.

emit_qwen38_generate_batched_mtp

emit_qwen38_generate_batched_pt

The same emitter with the prefill chunk width passed in rather than read from the environment.

emit_qwen38_generate_batched_q4

Every lane choice explicit — the form the compile gate and the equivalence locks drive.