Commission
Research
All research areas →

Agent Evaluation and Coordination

Generated-Code and GPU-Kernel Correctness

Federated Learning and Privacy

Decentralised Systems and Protocol

Methods
Papers
Collaborate
About Contact Bring a technical claim

Method · Dataset design

Build an evaluation set whose every item can be accounted for

An evaluation set is a sample drawn for a purpose. Designing one means saying which inputs it stands for, how items were chosen and excluded, where each came from and under what rights — and including items that should pass, so false alarms can be measured as well as hits.

The situation

We need a representative evaluation set with clear rights and provenance. How should examples and exclusions be documented?

What you leave with

Dataset specification, provenance ledger, sampling design and access restrictions, versioned together so any reported number can be traced to the items behind it.

For: Research manager; dataset owner

Why design matters more than size

Two of my preprints turn on dataset design rather than model quality.

  • The gpuemu corpus contains 26 ops: 16 correct controls and 10 buggy variants seeded with documented transcription patterns. Because controls are in the set, the test-input generation study could report recall and control false-positive rate: boundary-only shape sampling reached 78% recall with 0% false positives, while adversarial value sampling reached 99% recall with 94% false positives. A recall-only dataset would have ranked these the other way round.
  • Principle-Bench (workshop poster) maps 168 scenarios to two UK FCA principles, with paraphrase, adversarial keyword-stuffing and boundary perturbations authored under a pre-registered rubric. Fixing the rubric first is what lets the perturbations count as a test rather than a search for failures.

A third result shows why the population definition matters. In the input-regime audit (a manuscript under submission), a cross-encoder conflict detector scored F1 0.85 on short claim/evidence pairs but 0.18 on long multi-claim passages. Which detector looked better depended on which inputs were in the set.

These corpora are deliberately constructed; their results concern the seeded patterns and probes they contain, not the rate of those failures in deployed systems. That is the general trade-off of a designed set, and the specification states it.

Assumptions

The method assumes the population of interest can be described in a few explicit dimensions, that the origin and terms of source material can be established or honestly marked unknown, and that labels can be defined well enough for two people to apply them consistently.

Illustrative example — not a client engagement. A research manager needs an evaluation set for a contract-clause classifier. The specification stratifies by clause type, document length and drafting jurisdiction, with a quota for rare indemnity clauses. Twenty per cent of items are clauses that should receive no flag. The ledger records that 140 candidate documents came from a licensed database whose terms forbid redistribution; they stay in the evaluation but are tiered as withheld, with hashes published so a third party holding the same licence can verify them.

What this can and cannot establish

It can make every reported number traceable to identified items, make false alarms measurable, show which strata are thin, and document what may be shared. It cannot guarantee representativeness beyond the stated strata, clear rights on your behalf, or rule out contamination — overlap with training data is checked separately (see leakage and contamination checks).

Useful if

  • You are building a first-party evaluation set from logs, documents or expert-written cases and need it to survive external review.
  • An existing benchmark does not match your inputs and you need to decide what a representative replacement looks like.
  • You plan to share or publish an evaluation set and need to know which items you are allowed to release.

Not the right fit if

  • You need a legal opinion on data rights. The ledger records the rights basis you supply; it is not legal advice.
  • You need a large training corpus. The method is built for evaluation sets, where provenance per item is affordable.
  • The evaluation target is still undefined. Agree the decision and the population first, or the set will measure something else.

What this produces

  • Dataset specification. The decision the set informs, the population it represents, strata and quotas, item format, label definitions and the version number.
  • Sampling design. How items are drawn per stratum — random from logs, curated, expert-authored or perturbed — with the reason for each, and the known-good controls included.
  • Provenance ledger. One row per item: source, collection date, rights basis, transformations, author or labeller, guideline version and content hash.
  • Exclusion log. Exclusion rules fixed before labelling, and every excluded item with a reason code, so counts per reason can be reported.
  • Access tiers and datasheet. Which items are public, which are available on request and which are withheld with hashes recorded, plus a short datasheet for users.

What you need before starting

  • The decision and the population. What the set should tell you and which inputs it must stand for — users, document types, languages, difficulty.
  • Candidate sources. Logs, documents or expert time, with whatever is known about their origin and the terms under which they can be used.
  • Labelling capacity. People who can label to a written guideline, with time for a double-labelled subset.

Protocol

  1. 1
    Specify the population and strata. List the dimensions the result must hold across — for example input length, topic, language or difficulty — and set quotas per stratum, including rare strata the decision depends on.
  2. 2
    Design the sample, controls and perturbations. Choose a sampling method per stratum and include items whose correct outcome is known to be negative (correct code, compliant text, non-conflicting passages) so the false-positive rate can be measured, not only recall. If robustness matters, write the perturbation rubric — paraphrase, adversarial, boundary — before generating variants or scoring any system on them.
  3. 3
    Record provenance per item. Fill the ledger as items are created, not reconstructed afterwards. Hash each item so later changes are detectable.
  4. 4
    Apply exclusions by rule. Exclude only by pre-written rules (duplicates, unusable formats, rights not cleared), log each exclusion with a reason code, and report counts per reason.
  5. 5
    Label and measure agreement. Label to a versioned guideline, double-label a subset, report agreement and adjudicate disagreements with reasons kept.
  6. 6
    Seal, check and version. Hold back a sealed test split, check overlap with any development or training data, assign access tiers and release a versioned datasheet.

Limits and unfavourable results

  • Representativeness is only defined relative to the stated strata. A set can be balanced on every listed dimension and still miss one that matters.
  • Curated and perturbed items measure the behaviours they were designed to probe; they do not estimate how often those behaviours occur in production.
  • A provenance ledger is only as reliable as the source information given to it. Unknown origins are recorded as unknown.
  • Results on one input distribution may not carry over to another, even within the same task.

If the available sources cannot fill a stratum the decision depends on, the specification says so and the set is released with that gap stated, rather than padded with items that do not belong.

Evidence behind this page

Questions

How do you design an AI evaluation dataset?

Start from the decision and the population of inputs it concerns, stratify on the dimensions the result must hold across, include known-good controls alongside cases that should fail, write perturbation and exclusion rules before scoring, and keep a per-item provenance ledger from the first item.

What belongs in a benchmark dataset provenance record?

For each item: source, collection date, rights basis, any transformation applied, who authored or labelled it under which guideline version, and a content hash. For the set: version, exclusion counts by reason, and access tier per item.

Why include items that should pass?

Without them you can measure recall but not false alarms. In my test-input generation study, adversarial value sampling reached 99% bug recall but flagged 94% of correct controls, which would have looked like a strong result without the controls.

How should exclusions be documented?

As rules written before labelling, applied mechanically, with each excluded item logged under a reason code. The report gives counts per reason so a reader can judge whether exclusions shaped the result.

See also

Prepare an evaluation brief

Describe the decision the evaluation set must inform, the inputs it should represent and the sources you have. Rights and data access are agreed after a first conversation.

Last reviewed 2026-10-07.