arXiv
Test-Input Generation for Tensor Programs: What Actually Finds Kernel Bugs
Dipankar Sarkar
Abstract
Seven test-input generation strategies evaluated on the 26-op gpuemu corpus, reporting bug recall and control false-positive rate. Boundary-only shape sampling reaches 78% recall with 0% false positives; adversarial values reach 99% recall but 94% control false positives.
Test-input generation for tensor kernels is folkloric. Most projects pick a representative shape and dtype, run a fixed-shape allclose-style check, and ship. We make the choices explicit and measure them. Using the gpuemu op-schema-aware seeded fuzzer (arXiv:2606.20128), we evaluate seven test-generation strategies across a 26-op corpus (16 correct controls and 10 LLM-style buggy variants seeded with documented transcription patterns) on an RTX 3060 GPU instance. Strategies vary the shape candidate set, the dtype mix, and the input value distribution. We report each strategy on two axes: bug recall and control false-positive (FP) rate. Boundary-only shape sampling is the operationally safe winner: 78% recall on the 10 buggy kernels with 0% FP on the 16 controls. Adversarial value sampling reaches higher recall (99%) but inflates control FP to 94% because the strategy injects NaN and Inf inputs and the validator’s NaN check fires on every kernel that propagates them, not only on buggy kernels. On the two softmax tail-mask bugs the “regular” strategy (no boundary shapes) catches 0%, while boundary raises recall to 100% and 62% respectively. That gap is the clearest single signal in the data. The corpus result is about which seeded bug patterns each strategy catches, not about the bug rate of any specific deployed LLM.
arXiv comments: 8 pages, 1 figure, LNCS format. Companion paper: arXiv:2606.20128
Frequently Asked Questions
Which test-generation strategy worked best?
Boundary-only shape sampling was the operationally safe winner: 78% recall on the 10 buggy kernels with 0% false positives on the 16 controls. Adversarial value sampling reached 99% recall but inflated control false positives to 94%, because injected NaN and Inf inputs trip the validator on every kernel that propagates them.
Why report false positives alongside recall?
A strategy that flags correct kernels is not usable as a gate. Reporting both axes shows the trade-off a recall-only number hides.
How does it relate to the Correctness Illusion paper?
It is a companion paper (arXiv:2606.20128) and uses the same op-schema-aware seeded fuzzer and corpus. Its result concerns which seeded bug patterns each strategy catches, not the bug rate of a deployed LLM.