Research area · Agent evaluation
Evaluating agents beyond task completion
A completed benchmark task says little about whether an agent stayed within its authority, declined to invent an answer, avoided colliding with other agents or would rank the same way on a re-run. This area collects my published evidence on those questions and the open problems around them.
The situation
What evidence distinguishes an agent that completes a benchmark from one that respects authority and recovers from failure?
What you leave with
A research collaboration or evaluation study built around a measurable definition of the agent behaviour you care about, with its uncertainty reported.
For: Agent researcher; evaluation lead
What task completion leaves out
A benchmark success rate answers one question: did the agent reach an accepted end state? The failures that matter in practice are often orthogonal to it. My published work on agents addresses five of them, each with a different measurement problem.
- Acting within authority. The bounded-autonomy harness tracks six indicators, including escalation correctness and unsafe-action rate, and separates signals by their source of supervision. Its main result is a caution: at smoke scale, configuration variance was large enough to mask tuning effects.
- Refusing to fabricate. When a generated constraint program is unsatisfiable, handing the model a minimal unsatisfiable core instead of an error message cut fabricated solutions from 79% to 7%. A strong chain-of-thought baseline matched it on accuracy, so the gain is in recovery behaviour, not in the success rate.
- Coordinating with other agents. With a shared append-only log, the share of work that re-did a teammate’s task fell from 78% to 0% (Before the Pull Request).
- Trusting the judge. LLM judges for principle-based rules need four axes, not one: accuracy, paraphrase robustness, adversarial robustness and calibration. Retrieval-side detectors show the same pattern: the input-regime audit finds the best detector depends on the input distribution.
- Trusting the ranking. A self-audit of an LLM evaluation found the small-sample ranking identified the worst model reliably but not the best.
Choosing a route
The narrower pages below this one differ by the question you bring, not by technique.
| If your question is… | Start with |
|---|---|
| Do our agents duplicate or overwrite each other before review? | Multi-agent coordination |
| Does our conflict or grounding check survive a different kind of input? | RAG knowledge conflicts |
| How do we classify agent failures so they can be counted? | Agent failure taxonomies |
| Does a published agent result hold under our conditions? | Replication studies |
| No existing benchmark measures the behaviour we care about | Benchmark design |
If none of these fits, the question is probably still open, which is the most useful kind to discuss as collaboration.
What I do not claim
None of this work shows that any agent is safe, compliant or production-ready. The bounded-autonomy pilots were small (n=8 in the paper); the coordination results come from one substrate built by a company I founded; the judge benchmark covers two UK FCA principles in one domain. The contribution is measurement design and bounded evidence, stated with its limits. A collaboration starts from those limits rather than from the headline numbers.
Useful if
- Your agent benchmark reports a success rate, but the failures that worry you are overreach, fabrication or silent conflict.
- You are designing an agent evaluation and want indicators that a third party could recompute.
- You use an LLM as a judge and need to know how far its verdicts survive paraphrase or adversarial input.
- You have an unresolved evaluation question and want to work on it as research, with results that may be negative.
Not the right fit if
- You need an agent system built, integrated or monitored in production. That is delivery work on dipankar.co.
- You want a certificate that an agent is safe or compliant. Nothing on this page can establish that.
- You need a leaderboard position confirmed for publicity.
Limits and unfavourable results
- Most of the cited work is preprint or workshop-poster status, and several experiments are deliberately small (smoke-scale pilots, curated probes).
- Each result is bounded by its own models, tasks and measurement date. Hosted endpoints have already been withdrawn for one of the studies.
- The coordination evidence was produced with grite, an open-source tool from Neul Labs, a company I founded. It has not been independently replicated.
Evidence behind this page
- Evaluating Bounded Autonomy in Regulated Agentic AI: A Diagnostic Harness with Constitutional Rewards, Escalation Labels, and Runtime Governance
2026 · Preprint (not peer reviewed)
- A Four-Axis Trustworthiness Benchmark for LLM-as-Judge in Principle-Based Regulation
2026 · Peer-reviewed workshop paper
- Before the Pull Request: Mining Multi-Agent Coordination
2026 · Preprint (not peer reviewed)
- An Input-Regime Audit of Conflict Detection for Retrieval-Augmented Generation
2026 · Manuscript under submission
- How Reproducible Are Evaluation Conclusions? A Self-Audit of LLM-Inferred Prompt Structure
2026 · Preprint (not peer reviewed)
- From Errors to Proofs: Minimal-Core-Guided Repair for Neuro-Symbolic Constraint Solving
2026 · Peer-reviewed workshop paper
- grite (GitHub, Neul Labs) — Git-native append-only coordination log for agents, released with the coordination paper's dataset, harness and mining toolkit. MIT. Neul Labs is a company I founded.
- ebrag-vecdb-2026-paper (GitHub) — Input-Regime Audit paper, source, one JSON artefact per experiment and a runnable Python slice. MIT.
Narrower routes from here
Research topic
Multi-Agent Coordination and Conflicting Work
Duplicate effort, conflicting edits and lost writes between coding agents, measured from a coordination log rather than from pull requests.
Research topic
RAG Knowledge Conflicts and Input Regimes
How conflict detectors for RAG behave across input regimes: five findings from an audit, an input-regime ladder and open questions.
Questions
How do you evaluate an AI agent beyond task completion?
By defining the behaviours task success hides and measuring them separately: whether the agent escalates when it should, whether it refuses to fabricate when a problem is infeasible, whether concurrent agents duplicate or overwrite each other, and whether any LLM judge in the loop holds up under paraphrase and adversarial input. Each indicator needs its own source of supervision and its own uncertainty.
How should tool-using agent performance be measured?
Separate outcome metrics from behaviour metrics, log every action with provenance so failures can be recovered after the fact, and repeat runs. In the bounded-autonomy pilots, nominally identical configurations gave task success of 0.25 versus 0.12, so a single run is not enough to compare configurations.
Is an LLM judge good enough to score agents?
Only if it has been tested on the inputs it will actually see. In the four-axis benchmark, no method dominated accuracy, paraphrase robustness, adversarial robustness and calibration together, and the judge strongest on benign inputs lost 47 accuracy points on keyword-stuffed inputs.
Are these results peer-reviewed?
Two are workshop posters (KDD 2026 SeT-LLM and IJCAI-ECAI 2026 LogiSymb), the input-regime audit is a manuscript under submission, and the rest are arXiv preprints. Each publication page states its status.
See also
Method
Agent Failure Taxonomies and Incident Coding
Versioned codebook, ambiguity rules, double coding with agreement statistics and annotated examples for agent traces.
Commissioned study
AI Benchmark and Evaluation Study Design
Build a pre-specified test for a decision existing benchmarks do not measure, with baselines and failure conditions.
Method
Uncertainty, Repeated Trials and Evaluation Claims
Repeated-trial design, cluster bootstrap over items, rank stability and sensitivity comparisons for small performance gaps.
Overview
Generated-Code and GPU-Kernel Correctness Research
Correctness oracles, test-input generation, calibrated tolerances and a 26-op failure corpus for LLM-generated kernels.
Technical guides
Guide
Governed AI Agents: Authority, Approval, and Evidence
A practical architecture for governing tool-using AI agents with identity, purpose, action tiers, approvals, replayable evidence, and human stop conditions.
Guide
Production AI Agent Reliability: State, Side Effects, and Recovery
How to design reliable AI agents using explicit state, durable checkpoints, idempotent effects, replay, evaluation, observability, and operator-owned recovery.
Discuss the unresolved evaluation question
Send the claim, the decision it informs and what access exists. A short, non-confidential description is enough to start; data and code access are agreed afterwards.
Last reviewed 2026-10-07.