Method · Reproducibility
Ship the result with everything needed to check it
A reproducibility package lets someone else recompute every table from raw outputs, rerun the experiments that can still be rerun, and see exactly which parts cannot be — and why. It is designed alongside the study, not assembled after the paper is accepted.
The situation
What must accompany a paper or technical report so another team can inspect and rerun the result?
What you leave with
Environment manifest, inputs, scripts, raw outputs and known reproduction limits, with a claim-to-artefact map tying every reported number to the files that produce it.
For: Academic collaborator; industrial research team
Inspect, rerun, replicate
Three levels are worth separating, because a package should say which it supports for each result:
- Inspect and recompute — a reader recalculates every table from shipped raw outputs.
- Rerun — a reader regenerates raw outputs from inputs, scripts and the pinned environment.
- Replicate — an independent team tests the claim under its own conditions (see replication studies).
Recomputation is the floor. My self-audit of an LLM evaluation persisted 293 raw outputs; four of the eight model endpoints were withdrawn within ten weeks, so the study as specified can no longer be rerun, and the stored outputs are what remains checkable. The paper recommends reporting rank stability, per-cell provenance, executed sensitivity comparisons, raw per-run outputs and a measurement date alongside any ranking — this protocol builds those in.
Worked examples from my own packages
- gpuemu-corpus supports rerunning. It holds the 26-op corpus behind four papers, including The Correctness Illusion, with drivers per paper and stored seeds; every flagged failure replays byte-for-byte from its seed. Its gap: no tagged releases, so a reader cannot yet pin the exact repository state a paper used.
- ebrag-vecdb-2026-paper supports recomputation and a partial rerun. It ships the Input-Regime Audit paper’s source, one JSON artefact per experiment and a minimal runnable Python slice. Rerunning the experiments needs an API key for an OpenAI-compatible hosted endpoint (the paper run used Ollama Cloud), so hosted-model results can drift; the stored JSON outputs are what make them recomputable. It is pinned by the tag
v1.0-vecdb2026. - on-device-auction-audit shows partial release done openly. For the on-device auction paper, results, calibration data and analysis scripts are public, while the simulator source is withheld with hashes recorded, so the analysis is checkable but the simulation is not independently rerunnable.
None of these is a complete package by the standard on this page; each states what it supports. That is the point.
Assumptions
The protocol assumes raw outputs are captured during the study, that input rights are known well enough to decide what ships, and that someone outside the project can run the cold-start test.
Illustrative example — not a client engagement. An industrial research team is releasing a technical report comparing three hosted models on a document-extraction task. The claim-to-artefact map finds two figure values computed in a notebook that no longer runs; they are regenerated from stored outputs and the notebook replaced by a script. The limits file states that hosted-model outputs are recompute-only and dated, while the scoring code reruns exactly. A colleague’s cold-start run takes forty minutes and surfaces one missing environment variable.
Using the checklist
When the package exists and you need a release-readiness pass, use the reproducibility checklist. It is a worksheet, not a substitute for designing the package while the study runs, and passing it does not show that a result is correct.
Useful if
- You are publishing a paper or technical report whose numbers others will cite, compare against or try to beat.
- Your results depend on hosted model endpoints, GPUs or data that will not stay available.
- A collaborator or reviewer has tried to rerun your work and could not.
Not the right fit if
- You want a quick readiness check on a package that already exists. Use the reproducibility checklist instead.
- You need someone to independently reproduce a third party's result. That is a replication study.
- You need a continuously maintained evaluation pipeline. Packaging freezes a result; operating a pipeline is delivery work.
What this produces
- Claim-to-artefact map. Every table, figure and quoted number linked to the script that produces it and the input and output files it reads.
- Environment manifest. Lockfile or container, hardware and driver versions, model and API identifiers, and the measurement date for every hosted call.
- Inputs with provenance. Datasets, prompts and configurations with source, licence and hash; anything withheld is replaced by its hash and the reason.
- Raw per-run outputs and seeds. Every run's output, not only aggregates, with the seed or call parameters that produced it.
- Scripts with one entry point per result. Scripts that regenerate each table from raw outputs, and, where possible, the raw outputs from inputs.
- Known reproduction limits. What cannot be rerun, which sources of nondeterminism remain, and how close a rerun is expected to land.
What you need before starting
- The study's code and outputs. Ideally from the start of the work, so raw outputs are persisted as they are produced rather than recovered later.
- Rights over inputs. Knowledge of what each input's licence or agreement allows to be redistributed.
- A clean machine or a second person. Someone or something that has not seen the project, to run the cold-start test.
Protocol
- 1 Declare reproduction levels and map every claim. For each claim, say whether a reader can recompute it from shipped outputs, rerun it from inputs, or neither — and why. Hosted-endpoint results are often recompute-only after a few months. Then list every number in the paper and point it to a script, an input and an output file. Numbers without a path are either given one or removed.
- 2 Pin the environment and persist raw outputs. Record dependencies, container or lockfile, hardware, drivers, model versions and measurement dates. Store per-run outputs with seeds and call parameters in a stable, documented format — one file per experiment is a good default.
- 3 Package inputs and handle what cannot ship. Include inputs with licences and hashes. For withheld data or code, record hashes and the reason, so a party with access can confirm they hold the same material.
- 4 Write the limits file and run a cold-start test. Name each source of nondeterminism, each withdrawn or changing dependency, and the tolerance within which a rerun should match. Then follow the README from scratch on a clean machine or with a colleague; fix every step that needed undocumented knowledge.
- 5 License, version and archive. Apply licences to code and data separately, tag a release that matches the paper version, and archive it where it will outlive the repository.
Limits and unfavourable results
- A package makes a result checkable; it does not make it correct. A bug faithfully packaged reproduces perfectly.
- Hosted models change and are withdrawn. For those results the package supports recomputation from stored outputs, not regeneration.
- GPU and library nondeterminism can prevent bit-identical reruns; the limits file states the expected tolerance instead of promising exact equality.
- Withheld inputs or code reduce what a third party can verify, even with hashes recorded.
If the cold-start test fails or a number cannot be traced to an artefact, the package says so. A known gap stated plainly is more useful to a reader than a package that silently fails.
Evidence behind this page
- How Reproducible Are Evaluation Conclusions? A Self-Audit of LLM-Inferred Prompt Structure
2026 · Preprint (not peer reviewed)
- The Correctness Illusion in LLM-Generated GPU Kernels
2026 · Preprint (not peer reviewed) · cited by 6 (Google Scholar, October 2026)
- An Input-Regime Audit of Conflict Detection for Retrieval-Augmented Generation
2026 · Manuscript under submission
- When Privacy Moves ML-Mediated Decisions On Device: Information and Incentive Misalignment in Auctions
2026 · Preprint (not peer reviewed)
- gpuemu-corpus (GitHub) — 26 ops, drivers for four papers, stored seeds and scripts that replay flagged failures byte-for-byte. MIT or Apache-2.0. No tagged releases yet.
- gpuemu-corpus (Hugging Face) — The same 26 rows with fp64 references, as a dataset. MIT.
- ebrag-vecdb-2026-paper (GitHub) — Paper PDF and source, one JSON artefact per experiment and a runnable Python slice. MIT. Tagged v1.0-vecdb2026.
- on-device-auction-audit (GitHub) — Results, calibration data and analysis scripts; code MIT, data CC BY-NC 4.0; simulator source withheld with hashes recorded.
Questions
What should an AI research reproducibility package contain?
An environment manifest with measurement dates, inputs with provenance and licences, raw per-run outputs and seeds, one script per reported result, a map from every number to its files, and a statement of what cannot be reproduced and why.
How is this different from the reproducibility checklist?
This page is the method and the worked examples — how to design a package while the study runs. The reproducibility checklist is a short worksheet for checking whether an existing package is ready to release.
How can a study using hosted LLM APIs be reproduced?
Often it cannot be fully rerun once endpoints change, so persist every raw output with its endpoint identifier and date. A reader can then recompute every table and audit every judgement, even when regeneration is impossible.
What if some data or code cannot be released?
Release what you can, record hashes for what you cannot, and state the reason. Someone with legitimate access can then confirm they hold the same material and rerun that part.
See also
Papers & Artefacts
Every paper with its publication status, co-authors, code and data.
Free resource
Reproducibility Readiness Checklist
Check which missing artefacts would stop another team rerunning your study, and decide what to fix before release.
Commissioned study
AI Benchmark Replication Studies
Re-run a published benchmark or paper result under your workload, with raw results and a deviation report.
Method
Uncertainty, Repeated Trials and Evaluation Claims
Repeated-trial design, cluster bootstrap over items, rank stability and sensitivity comparisons for small performance gaps.
Prepare an evaluation brief
Describe the paper or report, what it depends on (endpoints, GPUs, data) and who needs to rerun it. I will say what level of reproduction is realistic and what the package needs.
Last reviewed 2026-10-07.