Where AI judges disagree is where evaluation can be fixed.

Abstract in brief

LLMs are widely used to judge AI-generated content, but a single LLM judge is prone to position, verbosity and self-enhancement biases. A panel of different models reduces those biases, yet when the judges disagree, majority voting discards the conflict instead of resolving it.

JuryFlow treats that disagreement as a precise, claim-level signal of where an evaluation is uncertain. Instead of re-scoring the whole response, a person acts as a structural guide who picks which conflict to resolve, and the system handles the rest.

To allow large-scale, reproducible comparison, the paper’s experiments evaluate an automatic configuration in which the focal claim is chosen by verdict-entropy ranking rather than by a person.

Method: a five-stage pipeline

By breaking a response into verifiable claims and treating disagreement as a graph, JuryFlow spends extra inference only where it can change the outcome.

1. 1. Multi-agent structured judging: Five LLMs from different providers (GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, Llama-3.1-70B-Instruct and Qwen2.5-72B-Instruct) decompose the response into atomic claims and give each an accept, reject or uncertain verdict with dimension tags. No aggregation or voting happens here.
2. 2. Disagreement graph: Each claim becomes a node scored by the entropy of its verdict distribution. Claims are connected by edges weighted by a mix of tag similarity and embedding similarity.
3. 3. Focal selection (the only human input): The system shows the most contested claims, and a person picks one to resolve first. No verdict, label or justification is needed. In the experiments, entropy ranking makes this selection.
4. 4. Propagated re-evaluation: A reviewer agent re-judges the focal claim against evidence, then propagates the correction to strongly connected claims in the graph and to similar historical cases found by embedding search.
5. 5. Incremental rubric update: The correction becomes a reusable rubric entry with a criterion, a positive example and a negative example, and every judge inherits it in later evaluations. The evaluator improves as it is used.

Main results: agreement with gold labels

Key findings

• JuryFlow beat the best single judge by 5.3 accuracy points on MT-Bench (0.804 → 0.857) and 10.5 points on LLMBar (0.679 → 0.784), and beat the majority-vote panel by 2.9 and 6.6 points.
• The gain was larger on LLMBar, which contains adversarial cases prone to judge disagreement.
• On LLMBar, re-evaluating only the contested claims lifted the majority-vote baseline by 3.0 points; propagation added 2.8 points and rubric induction 1.2 points.
• The propagation threshold showed an inverted-U relationship. Propagating too widely (8.7 calls per instance) spread errors across weakly related claims and dropped accuracy to 0.761; the selected setting reached 0.784 at 3.6 calls per instance.
• A panel of one model with different personas reached only 0.726 on LLMBar, barely above majority vote (0.718). Judges that share a model share its blind spots, so the gains need genuinely different models.

What it means for judging AI answers and measuring GEO

When LLMs score AI search answers or brand mentions, the reliability of the scoring itself becomes the question. Here is how VeReach reads the paper for practice; these are hypotheses.

Limitations

• Every experiment uses automatic focal selection; the human-guided mode has not been validated.
• When all judges confidently agree on the same error, entropy is low and the error is not flagged.
• Results come from two English benchmarks. Generalization to other languages, long-form responses and other modalities is untested.
• Cross-instance propagation and a growing rubric raise inference cost and risk rubric drift over long runs.

Frequently asked questions

What is JuryFlow?

JuryFlow is a human-in-the-loop, multi-agent evaluation framework for LLM-as-a-Judge. It treats disagreement between judges as a claim-level uncertainty signal, re-evaluates only the contested claims, and propagates each correction to related cases and to the evaluation rubric.

How is it different from majority voting among LLM judges?

Majority voting throws disagreement away; JuryFlow locates the contested claims and re-evaluates them against evidence. In the paper’s experiments, with the same five-model panel, it beat majority vote by 2.9 accuracy points on MT-Bench and 6.6 points on LLMBar.

How much human involvement does it need?

By design, the only human action is choosing which disagreement to resolve first; no verdicts or labels are entered. In the paper’s experiments this choice is also automated, so the effect of human selection has not yet been validated.

Where was the paper published?

It appears in HAI ’26, the 14th ACM International Conference on Human-Agent Interaction (November 16–19, 2026, Osaka), and was posted to arXiv (2609.40103) on September 30, 2026. Its DOI is 10.1145/3841580.3841619.