Commission
Research
All research areas →

Agent Evaluation and Coordination

Generated-Code and GPU-Kernel Correctness

Federated Learning and Privacy

Decentralised Systems and Protocol

Methods
Papers
Collaborate
About Contact Bring a technical claim

Commissioned research · Technical validation

Test the technical claim before your decision depends on it

When a purchase, architecture or research decision rests on a technical claim — a vendor's benchmark, a paper's headline table, your own team's prototype — a validation study tests it under a protocol fixed before any results exist, and reports the outcome whichever way it falls.

The situation

Our next decision depends on a vendor's technical claim. Who can design a transparent test and report an unfavourable result as readily as a favourable one?

What you leave with

A pre-agreed study protocol, raw results and a report stating what the evidence supports for your decision — including when the answer is no.

For: R&D director; technical diligence lead; platform researcher

Five questions, five kinds of study

Each route answers a different question. The cards further down describe each in full; this table is for choosing between them.

StudyQuestion it answersTypical triggerMain output
ReplicationDoes a published result hold, as published and under our conditions?A decision rests on a leaderboard, a paper’s table or a vendor benchmarkDeviation report with raw per-run results
Benchmark designWhat test would measure the decision we actually face?Existing benchmarks measure something adjacent, or flatter one systemPre-specified tasks, baselines, sampling plan and failure conditions
Model and retrieval comparisonWhich candidate is better here, by how much, and at what cost?A shortlist of models, embedders, rerankers or retrieval pipelinesCommon-workload results with uncertainty and quality–resource trade-offs
Generated-kernel correctnessAre generated GPU kernels correct across the inputs that matter?Kernels compile, pass a benchmark check and look fastOperator-aware oracle plan, adversarial inputs, reproducible failure report
FeasibilityIs the approach viable at all before we build it?A proposal untested in your settingBounded protocol, stop criteria and a proceed-or-stop report

Choosing between them

  • A number exists and you depend on it → replication. If the number exists but measures the wrong thing, benchmark design — often after a short replication shows why.
  • You already have candidates and must pick one → comparison. You have one candidate and the question is whether it works at all → feasibility.
  • The claim is about generated GPU code → kernel correctness. It sits under generated-code research because its method comes from that work, but it is commissioned the same way as the others.

What every study shares

  1. A pre-specified protocol. Comparisons, metrics, repeats and the decision rule are fixed before results exist; amendments are logged with reasons. See pre-specified evaluation plans.
  2. A result-independent fee. Quoted after scoping, fixed for the agreed scope, unchanged by the outcome.
  3. A conflict check before scoping. Disclosed in writing; some work is declined on that basis.
  4. Raw results, not only summaries. Per-run outputs, seeds and environment details, so a third party can recompute the tables. See reproducibility packages.
  5. Uncertainty against run-to-run variation, not against zero. See uncertainty and repeated trials.

Why the protocol has to come first

My own audits keep finding that a single number, or a test chosen after looking at the results, can point the wrong way. A self-audit of an LLM evaluation across eight open model variants found that a small-sample ranking identified the worst model reliably but not the best (How Reproducible Are Evaluation Conclusions?, preprint). An audit of RAG conflict detectors found that which detector looks better depends on the input distribution (Input-Regime Audit, manuscript under submission). And fixed-shape correctness checks used by LLM-kernel benchmarks certify buggy kernels as correct (The Correctness Illusion, preprint).

None of these is exotic. They are what happens when the test is narrower than the decision. Two further method pages cover the commonest traps: benchmark leakage and contamination and evaluation dataset design.

Starting

If the question is still loose, the evaluation brief builder turns it into the claim, the decision, the conditions and the access you have — which is what the first conversation needs.

Useful if

  • A decision depends on a technical claim you cannot verify from the published material alone.
  • You want the test designed by someone with no stake in which system wins, and the conflicts that do exist written down before work starts.
  • You can name the result that would change the decision, and you are prepared to act on it if it arrives.

Not the right fit if

  • You need existing evidence synthesised into a recommendation for an investment or board decision, without a new experiment. That is decision advisory, not a study.
  • You want an evaluation system built, run on every release and maintained in production. That is delivery engineering.
  • The conclusion is fixed in advance and the study is meant to decorate it.

Limits and unfavourable results

  • A study answers the question in its protocol, under the stated conditions. It is not a certification, an audit opinion or a guarantee of production behaviour.
  • I design and run studies personally, so scope is bounded by what one researcher can run and check carefully. Larger questions are staged, and the report says what was left out.
  • Results on hosted models carry a measurement date. Providers change models behind stable names, so a re-run later may not match.

Every route on this page can end in 'the claim does not hold here' or 'the approach does not work for this decision'. The fee is fixed at scoping, and the report is written the same way whichever way the result falls.

Evidence behind this page

  • gpuemu-corpus (GitHub) — What a study's raw-results package looks like: per-paper drivers, stored seeds and replay scripts. MIT or Apache-2.0; no tagged releases.
  • ebrag-vecdb-2026-paper (GitHub) — One JSON artefact per experiment plus a runnable Python slice. MIT.

Narrower routes from here

Questions

What does an independent AI technical validation actually test?

A specific claim under specified conditions — for example, that a model beats an incumbent on your query mix, that generated kernels are correct for your operators, or that an approach meets a threshold on your data. It does not test whether a system is good in general. The claim, conditions and decision rule are written down before any run.

Which kind of study do I need?

Use the comparison table above as a first cut, or the evaluation brief builder to write the question down. Many enquiries mix two routes — a replication that turns into benchmark design, a comparison that needs a feasibility check first. The scoping conversation settles which is primary; the other becomes a stated limitation or a second phase.

Who sets the protocol, and can it change after results come in?

The protocol is agreed with you before results exist. If something has to change — a model is withdrawn, a dataset turns out to be contaminated — the amendment is logged with its reason and date and reported alongside the results, so a reader can see what was decided after the fact.

How independent are you from the systems being tested?

I founded Neul Labs, which builds AI agent infrastructure in Rust, and my research affiliation is Skelf Research. Any relationship with a vendor or competitor in scope is disclosed before scoping and may mean I decline the work. My other sites (dipankar.co, dipankar.org) are the same person; a different domain name does not make the reviewer independent.

Is this the same as technical due diligence?

Not quite. Diligence weighs existing evidence for a decision; a validation study generates new evidence by running a test. A study can feed a diligence process, but the synthesis and recommendation to an investment committee is advisory work on dipankar.org.

See also

Scope a validation question

Send the claim, the decision it informs and what access exists. I will say whether a study can answer it, and which kind.

Last reviewed 2026-10-07.