Commissioned study · Model and retrieval comparison
Compare candidates on the same workload, with the uncertainty shown
A comparison study runs your shortlisted models or retrieval methods under one protocol — same inputs, same constraints, same scoring — and reports which differences exceed run-to-run noise, where the ranking changes with the kind of input, and what each option costs.
The situation
We need to compare candidate models or retrieval methods under the same constraints. What makes the comparison defensible?
What you leave with
Common workload protocol, uncertainty reporting and quality–resource trade-off results, broken down by input regime.
For: R&D sponsor; technical decision owner
What makes a comparison defensible
Four conditions, each of which published comparisons routinely miss:
- The same workload. Every candidate sees the same inputs, under the same constraints, scored by the same rule. Vendor numbers measured on different data are not a comparison.
- The same effort. If one candidate’s prompt was tuned for a week and another’s for an hour, the study measures tuning effort. The budget is agreed in advance and stated.
- Uncertainty, not just means. A difference matters only if it exceeds run-to-run and item-sampling variation. See uncertainty and repeated trials.
- Regimes declared before running. Slicing results after seeing them lets any candidate win somewhere. Slices are fixed in the protocol.
What my own comparisons showed
The Input-Regime Audit (manuscript submitted to the VecDB workshop at VLDB 2026) compared three conflict detectors for vector-database-backed RAG — a DeBERTa-MNLI cross-encoder, an LLM-as-judge baseline and the same judge conditioned on the question — across curated probes with bootstrap intervals, a 10-model LLM sweep, a prompt-robustness ablation, long-passage tests and external validation on 100 human-labelled FEVER pairs. The detector trade-off turned out to depend on the input distribution, so the result is a five-row input-regime ladder rather than a single winner. That is the shape most honest comparisons end up in: which option, for which inputs.
The reproducibility self-audit (preprint) kept 293 raw outputs from eight open model variants and applied a cluster bootstrap. The ranking resolved the bottom of the table and not the top — a reminder that “best” is the claim a small comparison is least able to support.
Quality and resources together
A candidate that is marginally better and several times more expensive is often the wrong choice, but only the decision owner can say how much quality a unit of latency or cost is worth. The study reports both axes at the measured settings and identifies options that are worse on both; it does not collapse them into one score unless you supply the exchange rate.
Illustrative example — not a client engagement. A team choosing between three embedding models for internal search defines two regimes before running: short keyword-style queries and long natural-language questions. On short queries the three are inside each other’s intervals; on long questions one model leads clearly but doubles index size. The memo recommends the cheapest model unless long questions are a priority, and states the query mix at which the recommendation flips.
Commission this if
- You have a shortlist of models, embedding models, rerankers or retrieval pipelines and must choose one, or decide when to use which.
- The comparisons you have use different prompts, versions, hardware or datasets for each candidate.
- Your inputs are heterogeneous and you suspect the best option depends on the kind of input.
Not the right fit if
- You only need one published comparison re-checked under your conditions. That is a replication.
- No workload exists yet that reflects the decision. Start with benchmark design.
- The choice is made and you need the stack integrated and operated. That is delivery work.
What you receive
- Common workload protocol. Inputs, regime slices, prompt templates, decoding and retrieval settings, and version pins — identical for every candidate, or with each difference declared.
- Raw per-item outputs. Every candidate's output on every item and repeat, so the tables can be recomputed or re-scored.
- Results with uncertainty. Intervals from a bootstrap that resamples items (and repeats where they exist), pairwise differences, and which rank positions are actually resolved.
- Regime breakdown. Results per input slice, showing where the ordering holds and where it reverses.
- Quality–resource trade-off. Quality plotted against latency, memory or token cost at the measured settings, with dominated options identified.
What the study needs from you
- Candidate list and access. API keys, weights or containers for each candidate, and any vendor-recommended settings.
- Representative inputs. A sample of real queries and documents, or enough description to construct a faithful proxy.
- Constraints. Latency budget, hardware, cost ceiling, data residency — the limits a winning candidate must live within.
- A decision owner. Someone who can say how large a quality difference has to be to justify a cost difference.
How the work runs
- 1 Scope and fairness rules. Agree the candidates, the tuning budget each one gets, who supplies prompts and configurations, and any conflicts with vendors in scope.
- 2 Regime map. Define the input slices before running — for example short versus long passages, or queries with and without an answer in the corpus.
- 3 Pre-specify. Fix metrics, repeats, the comparison rule and how multiple comparisons are handled, before any candidate is scored.
- 4 Run under one harness. Interleave candidates, pin versions, record measurement dates and resource use alongside quality.
- 5 Report and review. Deliver results, raw outputs and a decision memo; walk through which differences are resolved and which are not.
Limits and unfavourable results
- Results apply to the versions measured on the stated dates. Hosted models can change behind the same name.
- Equal tuning effort is a convention. A candidate might do better with more tuning; the budget each received is stated.
- Cost and latency figures depend on pricing and hardware at measurement time and are reported with those conditions.
A common outcome is that the candidates cannot be separated on your workload at the sample size you can afford. That is reported plainly, with an estimate of what sample would be needed, rather than dressed up as a ranking. The fee does not change.
Engagement terms
- Model
- Fixed-scope study with an agreed protocol and acceptance criteria for the deliverables (not for which candidate wins).
- Who does the work
- Dipankar Sarkar personally designs and runs the study. Any specialist help is disclosed and agreed in advance.
- Commercial basis
- Fixed research scope, quoted after a scoping conversation. The fee does not depend on the result.
- Availability
- Checked per enquiry.
Conflicts are checked before scoping. See publication, IP and independence.
Evidence behind this page
- An Input-Regime Audit of Conflict Detection for Retrieval-Augmented Generation
2026 · Manuscript under submission
- How Reproducible Are Evaluation Conclusions? A Self-Audit of LLM-Inferred Prompt Structure
2026 · Preprint (not peer reviewed)
- A Four-Axis Trustworthiness Benchmark for LLM-as-Judge in Principle-Based Regulation
2026 · Peer-reviewed workshop paper
- ebrag-vecdb-2026-paper (GitHub) — Detector comparison package: paper source, one JSON artefact per experiment and a runnable Python slice. MIT; tagged v1.0-vecdb2026.
Questions
How do you keep a comparison from favouring the preferred option?
The workload and its slices are fixed before any candidate runs; every candidate gets the same tuning budget; prompts and configurations supplied by a vendor or by your team are declared; and every slice is reported, not only those where a preferred option wins. Any relationship I have with a vendor in scope is disclosed before scoping.
Why report uncertainty in a model comparison?
Because a ranking can look decisive and not be. In my own self-audit, a small-sample ranking of eight open model variants identified the worst model reliably but not the best. Intervals and rank-stability tell you which positions in the table you can act on.
Can you compare retrieval pipelines as well as LLMs?
Yes — embedding models, rerankers, hybrid retrieval, and components such as conflict detectors in a RAG pipeline. The Input-Regime Audit compared three conflict detectors and found the trade-off depends on the input distribution, which is why regime slices are part of every comparison protocol.
Do you compare hosted APIs or only open-weight models?
Both. Open weights are pinned by version or hash and can be re-run. Hosted results carry a measurement date and may not be reproducible later; the report says which results are which.
See also
Commissioned study
AI Benchmark Replication Studies
Re-run a published benchmark or paper result under your workload, with raw results and a deviation report.
Commissioned study
AI Benchmark and Evaluation Study Design
Build a pre-specified test for a decision existing benchmarks do not measure, with baselines and failure conditions.
Method
Uncertainty, Repeated Trials and Evaluation Claims
Repeated-trial design, cluster bootstrap over items, rank stability and sensitivity comparisons for small performance gaps.
Research topic
RAG Knowledge Conflicts and Input Regimes
How conflict detectors for RAG behave across input regimes: five findings from an audit, an input-regime ladder and open questions.
Method
Benchmark Leakage and Contamination Checks
Overlap audit, written contamination hypotheses and a sensitivity analysis that tests whether the conclusion survives.
Commission a scoped study
Send the claim, the decision it informs and what access exists. A short, non-confidential description is enough to start; data and code access are agreed afterwards.
Last reviewed 2026-10-07.