Write the brief before you meet the evaluator

Why the decision comes first

An evaluation is only useful if it can change something. Starting from the decision — what you will do if the claim holds, what you will do if it fails, and who decides — fixes what the test has to distinguish. It also exposes the case where no plausible result would change anything, in which case the evaluation is not worth commissioning.

Why conditions and baseline are separate fields

Published results are measured somewhere else. My RAG conflict-detection audit found that which detector is preferable depends on the input distribution, and The Correctness Illusion found that fixed-shape checks used by kernel benchmarks certify buggy kernels as correct. In both cases, how and where the test was run decided the answer. Writing your conditions down — and naming what the claim must beat — is how the brief keeps the test about your situation rather than the original author’s.

Why the criteria are written before any result

Results usually allow more than one reasonable reading. In How Reproducible Are Evaluation Conclusions?, a ranking of eight open model variants identified the worst model reliably but not the best. Had the criterion been chosen after seeing the table, either story could have been told. The builder asks for the difference you would care about and the rule for calling it while nobody yet knows which way it goes.

Why the cost of being wrong matters

The two errors are rarely equal. Adopting a model that turns out worse may be cheap to reverse; missing a regression in a regulated workflow may not be. The relative cost sets how strict the criterion should be, how many repeats are worth paying for, and whether a cheaper, smaller study is enough.

Illustrative example — not a client engagement. A team is offered a cheaper hosted model. Their brief states the decision (“switch if quality holds”), the conditions (their own 400 historical tickets, current latency limit), the baseline (the current model on the same tickets), the criterion (no more than a two-point drop in graded accuracy, judged on the bootstrap interval) and the error costs (a false switch means customer-facing regressions; a missed switch means one more quarter at the higher price). The asymmetry tells them to set the bar strictly and accept a slower decision.

Using the builder

Fill in the fields below in any order; the brief assembles as you type and is saved in this browser only. When you are done you can copy it, download it as markdown, or choose Send this brief, which opens the contact form with the brief pre-filled. Nothing is sent until you submit that form.