arXiv
The Correctness Illusion in LLM-Generated GPU Kernels
Dipankar Sarkar
Abstract
Fixed-shape allclose checks used by LLM-kernel benchmarks certify buggy kernels as correct. Under op-schema-aware seeded fuzzing with an fp64 reference and per-(op, dtype) tolerances, a 26-op corpus shows 10 of 10 seeded illusions caught and 16 of 16 controls clean on five GPU classes.
Benchmarks for LLM-generated GPU kernels (KernelBench, TritonBench, GEAK) score correctness through fixed-shape, small-sample allclose-style checks. The number of inputs varies between benchmarks. The shape, dtype, and tolerance are fixed for each kernel. We test that oracle empirically. We construct a controlled corpus of 24 Triton and CPU stand-in kernels (15 correct controls and 9 LLM-style buggy variants seeded with documented transcription errors) and re-evaluate it under op-schema-aware seeded fuzzing with a high-precision (fp64) CPU reference and per-(op, dtype) absolute tolerances. The seeded oracle flags 9 of 9 buggy kernels and passes 15 of 15 correct controls, at zero precision cost on controls. We extend the corpus to 26 ops (adding a flash-attention pair) and re-run the same protocol on five GPU classes (RTX 3060, A10, L40S, A100 SXM4, H100 NVL). The verdicts are identical across all five GPUs: 10 of 10 illusions caught and 16 of 16 controls clean. The corpus result is about LLM-style transcription bugs that the allclose-on-one-shape oracle certifies as correct, not about the bug rate of any specific deployed LLM. Every flagged failure replays byte-for-byte from a stored seed.
For more on why fixed-shape checks miss these bugs, see Why GPU Kernel Benchmarks Miss Real Bugs.
Frequently Asked Questions
What is the Correctness Illusion?
Benchmarks for LLM-generated GPU kernels (KernelBench, TritonBench, GEAK) score correctness with fixed-shape, small-sample allclose-style checks. The paper shows that this oracle certifies LLM-style transcription bugs as correct. Re-evaluated under op-schema-aware seeded fuzzing with a high-precision (fp64) CPU reference and per-(op, dtype) absolute tolerances, the seeded oracle flags every buggy kernel while passing every correct control.
What corpus and hardware were used?
A controlled corpus that grew to 26 ops (16 correct controls and 10 LLM-style buggy variants seeded with documented transcription errors, including a flash-attention pair), re-run on five GPU classes: RTX 3060, A10, L40S, A100 SXM4 and H100 NVL. Verdicts were identical across all five.
What does the result not claim?
It is about LLM-style transcription bugs that an allclose-on-one-shape oracle certifies as correct, not about the bug rate of any specific deployed LLM.
Where are the corpus and harness?
The corpus is on Hugging Face (huggingface.co/datasets/dipankarsarkar/gpuemu-corpus, MIT) and GitHub (github.com/sarkar-dipankar/gpuemu-corpus, MIT or Apache-2.0). Every flagged failure replays byte-for-byte from a stored seed.