SeT-LLM @ KDD 2026
A Four-Axis Trustworthiness Benchmark for LLM-as-Judge in Principle-Based Regulation
Dipankar Sarkar
Abstract
Principle-Bench: 168 scenarios mapped to two UK FCA principles with paraphrase, adversarial and boundary perturbations, evaluating LLM judges on accuracy, paraphrase robustness, adversarial robustness and calibration.
Principle-based regulation, with evaluative standards such as “fair, clear, and not misleading” or “deliver good outcomes”, cannot be reduced to binary predicates, and LLM-as-judge is increasingly used as the substitute. Our position is that any such judge must be evaluated on four axes: accuracy, paraphrase robustness, adversarial robustness, and calibration. We release Principle-Bench, 168 cryptoasset financial-promotion scenarios mapped to two UK FCA principles, with paraphrase, adversarial keyword-stuffing, and boundary perturbations authored under a pre-registered rubric; the first benchmark covering all four axes for principle-based regulation. We also introduce Ceca (Calibrated Exemplar-Cluster Assessment): a calibrated, auditable assessor that emits exact per-exemplar counterfactual attributions. Across keyword counting, three sentence-transformer embedders, an open-weight LLM-judge, and a calibrated cascade, no method dominates all four axes. A 120B LLM-judge, strongest on benign inputs, loses 47 accuracy points (0.74 to 0.27) on keyword-stuffed Consumer Duty inputs: “compliance theatre.” A second judge from a different model family agrees only at Cohen’s kappa = 0.16 on that split, localising the failure to the model rather than the corpus. Any deployment-grade LLM-judge for principle-based regulation must report per-principle adversarial deception and post-hoc calibration alongside aggregate accuracy.
arXiv comments: 7 pages, 3 figures. Accepted at the KDD 2026 Workshop on Secure and Trustworthy Large Language Models (SeT-LLM), poster
Frequently Asked Questions
What are the four axes?
Accuracy, paraphrase robustness, adversarial robustness and calibration. The paper argues any LLM judge used for principle-based regulation must be evaluated on all four.
What did the benchmark show?
No method dominated all four axes. A 120B LLM judge that was strongest on benign inputs lost 47 accuracy points (0.74 to 0.27) on keyword-stuffed inputs, and a second judge from a different model family agreed only at Cohen's kappa = 0.16 on that split.
What was released?
Principle-Bench (168 cryptoasset financial-promotion scenarios with perturbations authored under a pre-registered rubric) and Ceca, a calibrated exemplar-cluster assessor with per-exemplar counterfactual attributions.