Commission
Research
All research areas →

Agent Evaluation and Coordination

Generated-Code and GPU-Kernel Correctness

Federated Learning and Privacy

Decentralised Systems and Protocol

Methods
Papers
Collaborate
About Contact Bring a technical claim

Research topic · RAG knowledge conflicts

A conflict detector is only as good as the inputs it was tested on

Retrieval-augmented generation pipelines increasingly check retrieved passages for contradictions before generating. Which detector to use depends less on the detector than on the kind of evidence it sees: short claim pairs, long passages, same-name entities or multi-hop fragments.

The situation

Does a conflict detector still work when the input regime changes, and which failure modes should an audit expose?

What you leave with

Input-regime matrix, conflict-labelled cases and regime-specific results, so a detector choice is made per input type rather than from one aggregate score.

For: Retrieval researcher; evaluation team

Background

A conflict-detection stage flags contradictions among retrieved passages before generation. Two approaches dominate practice: a pairwise natural-language-inference cross-encoder run over retrieved passages, and a query-conditioned formulation in which the judgement depends on the question being answered. Practitioners choose between them with little empirical guidance. My earlier preprint, Navigating the Knowledge Sea (2024), looked at LLM-based answer retrieval over large knowledge bases; it is context for this work, not evidence about conflict detection.

Findings from the input-regime audit

An Input-Regime Audit of Conflict Detection for Retrieval-Augmented Generation compares three detectors — a DeBERTa-MNLI cross-encoder, an LLM-as-judge baseline, and the same judge conditioned on the question — across curated probes with bootstrap intervals, a 10-model LLM sweep, a prompt-robustness ablation, long-passage realism, an answer-bearing scoping study and 100 human-labelled FEVER pairs. It is a manuscript under submission. Its five findings:

  1. LLM judges keep perfect precision on labelled FEVER pairs and flag far fewer top-k retrieved non-gold passages than the cross-encoder (0.09 vs 0.43 flag rate).
  2. A same-name “sense trap” false positive fools both the cross-encoder and pairwise LLM judging. Only question-conditioning closes the gap (precision 0.68 to 1.00 on curated probes), replicated for 8 of 10 LLM families.
  3. The cross-encoder versus LLM trade-off depends on the input distribution: FEVER F1 0.85 on short claim/evidence pairs but 0.18 on long multi-claim passages.
  4. The small-model regression under question-conditioning is a capability limit, not a format limit.
  5. Pairwise conflict detection is ill-posed on multi-hop evidence fragments and must be scoped to answer-bearing units.

The paper synthesises these into a five-row input-regime ladder recommending a detector per retrieval context.

What an audit should expose

The practical lesson is that “our detector scores well” needs a qualifier: on which inputs? An audit for your pipeline should at least separate short from long evidence, include ambiguous-entity cases, and identify where the retrieved units do not bear on the answer at all, because there the pairwise question has no right answer.

Illustrative example — not a client engagement. A research team reports strong contradiction detection on a claim-verification benchmark and plans to apply it to retrieved documentation pages. An audit samples their real retrievals, finds most are long multi-claim passages, and shows the detector’s precision falls sharply there. The recommendation is to segment to answer-bearing units before detection and to report results for that regime separately.

Open questions

The audit’s regimes are the ones its probes and datasets define. Questions it leaves open include how detectors behave on multilingual evidence, on temporal conflicts (a fact that changed), on deliberately injected contradictions, and how detector choice affects final answer quality. Each would need its own probes and labels, and any of them is a reasonable scope for a study.

Useful if

  • Your RAG method or conflict check performs well on one benchmark and you need to know whether it survives different evidence.
  • You are choosing between an NLI cross-encoder and an LLM judge for contradiction detection.
  • You want regime-specific failure cases you can inspect, not just an F1 score.

Not the right fit if

  • Your RAG application gives wrong answers and needs fixing and running. That is delivery work on dipankar.co.
  • You want a general RAG quality score. Conflict detection is one stage, and this work does not measure end-to-end answer quality.
  • You need results in languages or domains you cannot supply labelled cases for.

What this produces

  • Input-regime matrix. The input types your pipeline actually retrieves — short pairs, long passages, ambiguous entities, multi-hop fragments — each with a target sample size.
  • Conflict-labelled cases. Curated probes and, where possible, an external human-labelled set, with labelling rules written down before labelling.
  • Regime-specific results. Precision, recall and flag rate per detector per regime, with bootstrap confidence intervals, and the prompts and model versions used.
  • Detector recommendation per regime. A ladder in the style of the paper, bounded explicitly by the probes, models and data used.

What you need before starting

  • Representative retrievals. Samples of what your retriever returns, including the passage lengths and formats seen in practice.
  • Candidate detectors. The detectors under consideration and any that are already deployed.
  • Labelling capacity. Access to someone who can adjudicate disputed conflict labels in your domain.

Protocol

  1. 1
    Map the regimes. Characterise retrieved inputs by length, claim count, entity ambiguity and whether each unit actually bears on the answer.
  2. 2
    Build probes per regime. Curated cases with known labels, including sense traps (same name, different referent) and long multi-claim passages.
  3. 3
    Run detectors with and without the question. Pairwise and question-conditioned variants, across more than one model family, with a prompt-robustness check.
  4. 4
    Validate externally. Score against an independent human-labelled set so the curated probes are not the only evidence.
  5. 5
    Report per regime. Results with intervals for each regime, and the regimes where the question is ill-posed rather than merely hard.

Limits and unfavourable results

  • The paper is a manuscript submitted to a workshop (VecDB @ VLDB 2026), not yet a peer-reviewed publication, and it is not on arXiv.
  • The curated probe set is small (n=63, with bootstrap intervals) and external validation uses 100 FEVER pairs.
  • The input-regime ladder is bounded by the probes, models and datasets in the paper; it is a starting hypothesis for your inputs, not a rule.

Evidence behind this page

  • ebrag-vecdb-2026-paper (GitHub) — Paper PDF and source, one JSON artefact per experiment and a minimal runnable Python slice. MIT; tagged v1.0-vecdb2026.

Questions

How do you evaluate knowledge-conflict detection in RAG?

Per input regime, not in aggregate. Build labelled cases for each kind of evidence the retriever returns, run each detector with and without the question, report intervals per regime, and validate on an independent human-labelled set.

What is an input-regime audit?

A study that tests whether a method's result survives changes in the kind of input it receives. In the RAG conflict audit, the cross-encoder's FEVER F1 was 0.85 on short claim/evidence pairs but 0.18 on long multi-claim passages.

Should I use an NLI cross-encoder or an LLM judge?

It depends on the regime. On top-k retrieved non-gold passages the LLM judges flagged far fewer (0.09 vs 0.43 flag rate), but the cross-encoder was competitive on its native short-pair distribution. Same-name sense traps needed question-conditioning to resolve.

Is there a benchmark for RAG conflict detection?

The paper uses curated probes, a 10-model LLM sweep and 100 human-labelled FEVER pairs, and releases its artefacts. It is an audit with a small probe set rather than a large benchmark, and it says so.

See also

Commission a scoped study

Send the claim, the decision it informs and what access exists. A short, non-confidential description is enough to start; data and code access are agreed afterwards.

Last reviewed 2026-10-07.