Commissioned research · Technical validation
Test the technical claim before your decision depends on it
When a purchase, architecture or research decision rests on a technical claim — a vendor's benchmark, a paper's headline table, your own team's prototype — a validation study tests it under a protocol fixed before any results exist, and reports the outcome whichever way it falls.
The situation
Our next decision depends on a vendor's technical claim. Who can design a transparent test and report an unfavourable result as readily as a favourable one?
What you leave with
A pre-agreed study protocol, raw results and a report stating what the evidence supports for your decision — including when the answer is no.
For: R&D director; technical diligence lead; platform researcher
Five questions, five kinds of study
Each route answers a different question. The cards further down describe each in full; this table is for choosing between them.
| Study | Question it answers | Typical trigger | Main output |
|---|---|---|---|
| Replication | Does a published result hold, as published and under our conditions? | A decision rests on a leaderboard, a paper’s table or a vendor benchmark | Deviation report with raw per-run results |
| Benchmark design | What test would measure the decision we actually face? | Existing benchmarks measure something adjacent, or flatter one system | Pre-specified tasks, baselines, sampling plan and failure conditions |
| Model and retrieval comparison | Which candidate is better here, by how much, and at what cost? | A shortlist of models, embedders, rerankers or retrieval pipelines | Common-workload results with uncertainty and quality–resource trade-offs |
| Generated-kernel correctness | Are generated GPU kernels correct across the inputs that matter? | Kernels compile, pass a benchmark check and look fast | Operator-aware oracle plan, adversarial inputs, reproducible failure report |
| Feasibility | Is the approach viable at all before we build it? | A proposal untested in your setting | Bounded protocol, stop criteria and a proceed-or-stop report |
Choosing between them
- A number exists and you depend on it → replication. If the number exists but measures the wrong thing, benchmark design — often after a short replication shows why.
- You already have candidates and must pick one → comparison. You have one candidate and the question is whether it works at all → feasibility.
- The claim is about generated GPU code → kernel correctness. It sits under generated-code research because its method comes from that work, but it is commissioned the same way as the others.
What every study shares
- A pre-specified protocol. Comparisons, metrics, repeats and the decision rule are fixed before results exist; amendments are logged with reasons. See pre-specified evaluation plans.
- A result-independent fee. Quoted after scoping, fixed for the agreed scope, unchanged by the outcome.
- A conflict check before scoping. Disclosed in writing; some work is declined on that basis.
- Raw results, not only summaries. Per-run outputs, seeds and environment details, so a third party can recompute the tables. See reproducibility packages.
- Uncertainty against run-to-run variation, not against zero. See uncertainty and repeated trials.
Why the protocol has to come first
My own audits keep finding that a single number, or a test chosen after looking at the results, can point the wrong way. A self-audit of an LLM evaluation across eight open model variants found that a small-sample ranking identified the worst model reliably but not the best (How Reproducible Are Evaluation Conclusions?, preprint). An audit of RAG conflict detectors found that which detector looks better depends on the input distribution (Input-Regime Audit, manuscript under submission). And fixed-shape correctness checks used by LLM-kernel benchmarks certify buggy kernels as correct (The Correctness Illusion, preprint).
None of these is exotic. They are what happens when the test is narrower than the decision. Two further method pages cover the commonest traps: benchmark leakage and contamination and evaluation dataset design.
Starting
If the question is still loose, the evaluation brief builder turns it into the claim, the decision, the conditions and the access you have — which is what the first conversation needs.
Useful if
- A decision depends on a technical claim you cannot verify from the published material alone.
- You want the test designed by someone with no stake in which system wins, and the conflicts that do exist written down before work starts.
- You can name the result that would change the decision, and you are prepared to act on it if it arrives.
Not the right fit if
- You need existing evidence synthesised into a recommendation for an investment or board decision, without a new experiment. That is decision advisory, not a study.
- You want an evaluation system built, run on every release and maintained in production. That is delivery engineering.
- The conclusion is fixed in advance and the study is meant to decorate it.
Limits and unfavourable results
- A study answers the question in its protocol, under the stated conditions. It is not a certification, an audit opinion or a guarantee of production behaviour.
- I design and run studies personally, so scope is bounded by what one researcher can run and check carefully. Larger questions are staged, and the report says what was left out.
- Results on hosted models carry a measurement date. Providers change models behind stable names, so a re-run later may not match.
Every route on this page can end in 'the claim does not hold here' or 'the approach does not work for this decision'. The fee is fixed at scoping, and the report is written the same way whichever way the result falls.
Evidence behind this page
- How Reproducible Are Evaluation Conclusions? A Self-Audit of LLM-Inferred Prompt Structure
2026 · Preprint (not peer reviewed)
- An Input-Regime Audit of Conflict Detection for Retrieval-Augmented Generation
2026 · Manuscript under submission
- The Correctness Illusion in LLM-Generated GPU Kernels
2026 · Preprint (not peer reviewed) · cited by 6 (Google Scholar, October 2026)
- A Four-Axis Trustworthiness Benchmark for LLM-as-Judge in Principle-Based Regulation
2026 · Peer-reviewed workshop paper
- gpuemu-corpus (GitHub) — What a study's raw-results package looks like: per-paper drivers, stored seeds and replay scripts. MIT or Apache-2.0; no tagged releases.
- ebrag-vecdb-2026-paper (GitHub) — One JSON artefact per experiment plus a runnable Python slice. MIT.
Narrower routes from here
Commissioned study
AI Benchmark Replication Studies
Re-run a published benchmark or paper result under your workload, with raw results and a deviation report.
Commissioned study
AI Benchmark and Evaluation Study Design
Build a pre-specified test for a decision existing benchmarks do not measure, with baselines and failure conditions.
Commissioned study
Independent Model and Retrieval Comparison Studies
Run shortlisted models or retrieval methods on one workload; report uncertainty, input-regime effects and costs.
Free resource
Evaluation Brief Builder
Turn an uncertain technical claim into a structured experiment brief in your browser, before any conversation.
Commissioned study
Empirical AI Feasibility Studies
A bounded experiment with stop criteria fixed in advance, ending in a proceed-or-stop recommendation.
Questions
What does an independent AI technical validation actually test?
A specific claim under specified conditions — for example, that a model beats an incumbent on your query mix, that generated kernels are correct for your operators, or that an approach meets a threshold on your data. It does not test whether a system is good in general. The claim, conditions and decision rule are written down before any run.
Which kind of study do I need?
Use the comparison table above as a first cut, or the evaluation brief builder to write the question down. Many enquiries mix two routes — a replication that turns into benchmark design, a comparison that needs a feasibility check first. The scoping conversation settles which is primary; the other becomes a stated limitation or a second phase.
Who sets the protocol, and can it change after results come in?
The protocol is agreed with you before results exist. If something has to change — a model is withdrawn, a dataset turns out to be contaminated — the amendment is logged with its reason and date and reported alongside the results, so a reader can see what was decided after the fact.
How independent are you from the systems being tested?
I founded Neul Labs, which builds AI agent infrastructure in Rust, and my research affiliation is Skelf Research. Any relationship with a vendor or competitor in scope is disclosed before scoping and may mean I decline the work. My other sites (dipankar.co, dipankar.org) are the same person; a different domain name does not make the reviewer independent.
Is this the same as technical due diligence?
Not quite. Diligence weighs existing evidence for a decision; a validation study generates new evidence by running a test. A study can feed a diligence process, but the synthesis and recommendation to an investment committee is advisory work on dipankar.org.
See also
Commissioned study
Generated GPU Kernel Correctness Studies
Test generated GPU kernels with operator-aware oracles, calibrated tolerances and adversarial inputs.
Method
Pre-Specified Evaluation Plans
Versioned research question, primary outcomes, exclusion and analysis rules, and an amendment log, fixed before results.
Method
Uncertainty, Repeated Trials and Evaluation Claims
Repeated-trial design, cluster bootstrap over items, rank stability and sensitivity comparisons for small performance gaps.
Method
Benchmark Leakage and Contamination Checks
Overlap audit, written contamination hypotheses and a sensitivity analysis that tests whether the conclusion survives.
Method
Evaluation Dataset Design and Provenance
Dataset specification, sampling design with controls, per-item provenance ledger, exclusion log and access restrictions.
Free resource
Evaluation Brief Builder
Turn an uncertain technical claim into a structured experiment brief in your browser, before any conversation.
Scope a validation question
Send the claim, the decision it informs and what access exists. I will say whether a study can answer it, and which kind.
Last reviewed 2026-10-07.