Commission
Research
All research areas →

Agent Evaluation and Coordination

Generated-Code and GPU-Kernel Correctness

Federated Learning and Privacy

Decentralised Systems and Protocol

Methods
Papers
Collaborate
About Contact Bring a technical claim
Back to publications

SeT-LLM @ KDD 2026

A Four-Axis Trustworthiness Benchmark for LLM-as-Judge in Principle-Based Regulation

Dipankar Sarkar

KDD 2026 Workshop on Secure and Trustworthy Large Language Models (SeT-LLM), poster Peer-reviewed workshop paper

Abstract

Principle-Bench: 168 scenarios mapped to two UK FCA principles with paraphrase, adversarial and boundary perturbations, evaluating LLM judges on accuracy, paraphrase robustness, adversarial robustness and calibration.

Principle-based regulation, with evaluative standards such as “fair, clear, and not misleading” or “deliver good outcomes”, cannot be reduced to binary predicates, and LLM-as-judge is increasingly used as the substitute. Our position is that any such judge must be evaluated on four axes: accuracy, paraphrase robustness, adversarial robustness, and calibration. We release Principle-Bench, 168 cryptoasset financial-promotion scenarios mapped to two UK FCA principles, with paraphrase, adversarial keyword-stuffing, and boundary perturbations authored under a pre-registered rubric; the first benchmark covering all four axes for principle-based regulation. We also introduce Ceca (Calibrated Exemplar-Cluster Assessment): a calibrated, auditable assessor that emits exact per-exemplar counterfactual attributions. Across keyword counting, three sentence-transformer embedders, an open-weight LLM-judge, and a calibrated cascade, no method dominates all four axes. A 120B LLM-judge, strongest on benign inputs, loses 47 accuracy points (0.74 to 0.27) on keyword-stuffed Consumer Duty inputs: “compliance theatre.” A second judge from a different model family agrees only at Cohen’s kappa = 0.16 on that split, localising the failure to the model rather than the corpus. Any deployment-grade LLM-judge for principle-based regulation must report per-principle adversarial deception and post-hoc calibration alongside aggregate accuracy.

arXiv comments: 7 pages, 3 figures. Accepted at the KDD 2026 Workshop on Secure and Trustworthy Large Language Models (SeT-LLM), poster

Frequently Asked Questions

What are the four axes?

Accuracy, paraphrase robustness, adversarial robustness and calibration. The paper argues any LLM judge used for principle-based regulation must be evaluated on all four.

What did the benchmark show?

No method dominated all four axes. A 120B LLM judge that was strongest on benign inputs lost 47 accuracy points (0.74 to 0.27) on keyword-stuffed inputs, and a second judge from a different model family agreed only at Cohen's kappa = 0.16 on that split.

What was released?

Principle-Bench (168 cryptoasset financial-promotion scenarios with perturbations authored under a pre-registered rubric) and Ceca, a calibrated exemplar-cluster assessor with per-exemplar counterfactual attributions.

LLM-as-JudgeBenchmarkingRegulationAdversarial RobustnessCalibration

Related Content