Essay · Part 1 of 2

Are your eval scores measuring the model or the raters?

A psychometric teardown of an influential judging pattern in AI.

Shawn Fraine

· 17 min read

Part 1 of a two-part series. Part 2 runs the experiments.

Key points

  • MT-Bench's famous 85% describes GPT-4's agreement with individual human votes on first-turn questions, computed on decisive votes only. With ties and position-inconsistent verdicts included under a different scoring rule, agreement is 66%.
  • The paper provides useful evidence that an LLM judge can approximate human preference. Whether those judgments support a deployment decision or a claim about a specific capability requires additional evidence.
  • On comparisons designed to be hard, GPT-4 stayed consistent 65% of the time when answer order was swapped. The authors investigated this weakness and tested ways to reduce it.
  • The judging prompt names six qualities without defining their boundaries, anchoring performance, or specifying how conflicts should be resolved. That leaves important parts of the measurement procedure implicit.
  • Test whether your judge stays stable when irrelevant features change and responds appropriately when answer quality changes. Report uncertainty and unresolved cases, and validate the procedure for the decision you actually need to make.

I was trained to read validation studies for psychological tests, so that is how I read evaluation papers now. I want to know what construct the authors are claiming to measure, what evidence supports the scores, and where the error is coming from. When I read MT-Bench that way, I keep coming back to a practical question: what decisions can these scores actually support?

That concern reaches beyond one benchmark. A 2025 systematic review of 445 LLM benchmarks found construct-validity problems across the field, including benchmarks that never defined the phenomenon they claimed to measure (Bean et al., 2025).

Zheng et al. (2023) reported that GPT-4, used as a judge, agreed with human preferences about 85% of the time. Their paper helped establish a standard pattern in modern eval pipelines: a strong model weighs candidate answers against natural-language criteria, explains its choice, and produces a verdict.

The paper set out to build a cheap approximation of human preference for benchmarking models, and it provided substantial evidence for that use. The authors were also unusually honest about the limitations. I want to keep that distinction clear, because criticizing a paper for failing to validate a use it never claimed would be a pretty poor start to an article about measurement.

My concern is what happens when that preference result is asked to carry a larger claim. A team uses a similar judging procedure to say that a model reasons better, that a product is safer, or that a small score increase justifies a release. The original agreement result doesn't establish those interpretations. The team needs evidence connecting its own scores to its own decision. This teardown is about what that evidence would have to cover.

What the instrument is

The pairwise judge uses a prompt template called pair-v2 in the FastChat repository. It asks the model to act as an impartial judge, consider six qualities (helpfulness, relevance, accuracy, depth, creativity, and level of detail), write a short explanation, and output [[A]], [[B]], or [[C]] for a tie. It also tells the judge to disregard response order, response length, and assistant names.

There is a second template, and the difference matters. The agreement study examined both pairwise comparison and single-answer grading. Table 5 shows that single-answer grading matches pairwise GPT-4 and human preferences well, which supports using the more scalable 1-to-10 version for the leaderboard.

But evidence that the procedures produce similar preferences leaves questions about the scale itself. The ten points have no score-level anchors. The paper does not establish that a 7 means the same thing across questions, categories, or judge versions, or that moving from 6 to 7 represents the same improvement as moving from 8 to 9. The authors themselves warn that absolute scores are likely to fluctuate more than relative pairwise results if the judge model changes.

Back to pair-v2. None of the six qualities is defined. Nothing specifies how accuracy should trade off against creativity when they conflict, or whether "level of detail" means the right amount of detail or simply more of it.

That does not mean the model starts from nothing. A pretrained judge brings learned preferences and evaluative knowledge with it. The authors interpret the agreement between judging methods as evidence of a relatively stable internal rubric. That is a reasonable possibility. What remains undocumented is the policy itself: which considerations drive a verdict, whether those tradeoffs fit the intended use, and how stable they remain when the setup changes.

The explanation doesn't settle that question. It can read like a breakdown by criterion, but the procedure does not validate it as a faithful decomposition of the verdict. A plausible explanation gives you something to inspect. It doesn't, by itself, tell you how the judgment was produced.

Unpacking the 85%

The headline comes from Table 5, and to the authors' credit they reported it two ways. S2 includes only non-tie votes. S1 includes ties and recodes position-inconsistent verdicts as ties. GPT-4 pairwise agreement with humans on first-turn questions is 85% under S2 and 66% under S1. Human agreement with other humans follows a similar pattern: 81 to 82% on decisive votes and 63 to 67% with ties included. On the larger Chatbot Arena sample in Table 6, GPT-4 pairwise reached 87% on non-tie votes and 64% with ties.

Two details matter. First, agreement with humans means agreement with a randomly selected individual human vote on a randomly selected question. There is no adjudicated gold label behind that number. Second, S1 and S2 change both the included cases and the scoring setup. The paper gives uniform-random agreement baselines of 33% and 50%, respectively. You cannot read the difference between 85 and 66 as nineteen points of concealed error.

A chance correction can help interpret agreement, provided you state the chance model. Using the paper's uniform-random baselines, subtracting expected chance agreement and dividing by the remaining possible agreement gives about 0.70 for GPT-4's binary 85%, about 0.62 for human-versus-human 81%, and about 0.49 for the three-way 66%.

Those are illustrative rescalings under a stated assumption. They are not empirical Cohen's kappa estimates, and 0.70 is not the judge's hidden "true reliability." An estimate using the raters' observed label distributions would require the relevant distributions and a method suited to the design. I would report a suitable chance-corrected statistic alongside raw agreement, the included cases, and uncertainty. The smaller number alone is not a finding of worse performance.

The more consequential issue is what the headline leaves out. Excluding ties removes cases where a strict preference was not recorded, including genuine equality, uncertainty, and position inconsistency recoded as a tie. Those distinctions matter when a team is trying to choose between closely matched systems. I would want to see the unresolved cases, how often they occur, and what happens to the decision when they are included.

The strongest defense of the headline is straightforward. GPT-4's agreement with humans is numerically close to human agreement with other humans under both setups. The paper describes them as reaching the same level. That is useful evidence for a preference-matching task.

It still leaves the interpretation of those preferences open. Matching human votes does not demonstrate that each named criterion is being measured, or that a score supports a claim about reasoning, safety, or factual accuracy. Chen et al. (2026) formalize this gap between reliability and construct validity. In one audit, predictors using only surface features reproduced 67.4% of MT-Bench's human votes. That finding raises questions about the validation labels themselves; it does not show that every surface feature is irrelevant or that the entire benchmark is invalid.

Table 7 adds another qualification. GPT-4's agreement with humans is highest when the models being compared differ substantially in quality and lowest when they are close. That makes sense. It also matters for procurement and release decisions, where the choice may depend on narrow margins between strong systems.

The paper stress-tested its own judge

The authors ran attacks on their own procedure. I think this is some of the most useful evidence in the paper, because it shows where the judging setup holds up and where it needs help.

Measurement science calls score movement driven by features outside the intended construct construct-irrelevant variance. The prompt says order should not matter, so a verdict that changes with answer order is a clear candidate. Redundant padding, superficial formatting, or familiarity can create similar problems when they influence a score without improving the qualities being evaluated.

Some of that variation may be random. Some may be systematic bias. Repeated judgments can help estimate random variation; they do not guarantee that a systematic preference will average away. If the procedure consistently rewards the wrong feature, a larger sample can give you a more precise estimate of the wrong thing.

Position

The authors built comparisons designed to be hard, with answers that were very similar and sometimes indistinguishable even to humans, then swapped their order. GPT-4 stayed consistent 65% of the time, GPT-3.5 46%, and Claude-v1 24%. These are results from that deliberately difficult test set, not estimates of failure rates across all ordinary evaluations.

The instruction to ignore order did not eliminate the effect. A contemporaneous study showed how consequential ordering could be: reordering candidates let Vicuna-13B "beat" ChatGPT on 66 of 80 queries (Wang et al., 2023).

Running both orders and treating inconsistent results as ties is one mitigation, and S1 accounts for position inconsistency that way. It requires more judging calls and turns instability into an explicit unresolved outcome. That is useful bookkeeping. It also means the tie-excluding headline should be read alongside the inconsistency rate.

The authors tested a further repair that deserves attention. Providing a few judgment examples increased GPT-4's consistency from 65.0% to 77.5%. Examples helped. They did not establish order invariance, but they show why the whole judging procedure matters, including the support supplied to the model.

Verbosity

The authors took 23 answers containing numbered lists, padded them with redundant rephrasings that added no information, and tested whether the judge preferred the longer version. Claude-v1 and GPT-3.5 fell for it 21 times out of 23. GPT-4 failed twice.

GPT-4 resisted this attack much better than the weaker judges. I wouldn't turn that result into a blanket claim that every model rewards padding. The specification problem remains, though: "level of detail" is one of the criteria, and the prompt doesn't explain how to distinguish useful detail from redundancy. Whether a judge makes that distinction reliably needs to be tested.

Wu and Aji (2023) approached the issue from a different angle. In their main experiment, with GPT-4 judging GPT-4 outputs, responses constructed to contain several minor factual errors received an Elo of 1206, compared with 1096 for correct but short responses. That comparison raises a related question about how the judge trades off correctness and presentation. It is not the same controlled padding test, because the answers also differ in factual content.

Self-preference

GPT-4 awarded itself a win rate roughly 10 percentage points higher than humans awarded it, and Claude-v1's gap was about 25 points. The authors called this inconclusive on their data. That was the right call.

Later work tested self-preference under more controlled conditions. Pombal et al. (2026) found that judges could be more than 50% more likely to incorrectly mark a criterion as satisfied for their own outputs, among criteria those outputs actually failed. The effect persisted on programmatically verifiable criteria, and score distortions reached up to 10 points on HealthBench.

That supports concern about self-preference in objectively checkable settings. It does not retroactively explain the original MT-Bench gaps. Wataoka et al. (2024) also found evidence consistent with a familiarity account, in which judges score low-perplexity outputs more favorably regardless of who generated them. The practical question is whether the judge rewards a feature you intended to measure, and the answer requires controls for response quality.

Scaffolding

On the paper's small math diagnostic, GPT-4's judging failures went from 14 out of 20 with the default prompt to 6 out of 20 with chain-of-thought prompting and 3 out of 20 with a reference answer. Those twenty judgments came from ten questions evaluated in both answer orders.

Judge competence depended heavily on the setup. Much of the improvement appeared with prompting alone, and reference guidance helped further. The authors used reference-guided judging for math themselves. Their results give you a reason to validate the supported procedure you actually plan to run, rather than assuming the default prompt works equally well in every domain.

One more finding deserves a fair hearing. When humans disagreed with GPT-4 and then saw its reasoning, they called that reasoning reasonable 75% of the time and changed their own verdict 34% of the time. The explanations were persuasive to many readers. Whether those revised judgments were more accurate, and whether the explanations faithfully represented the judge's decision process, are separate questions.

What a conventional rating instrument would specify

A fair objection is that MT-Bench was designed for cheaper benchmarking, and not every benchmark needs a full psychometric dossier. I agree. The amount and kind of evidence should follow the interpretation and the stakes of the use.

The Standards for Educational and Psychological Testing (AERA, APA, & NCME, 2014) frame validity around the interpretation of scores for a proposed use. Zheng et al. provide evidence that their procedures approximate expert and crowdsourced human preferences in the settings studied. A closely related use can draw on that evidence. A claim that a model reasons better needs reasoning-specific evidence. A ship-or-no-ship threshold needs evidence about how consistently and appropriately the procedure makes that decision.

The paper already supplies more measurement evidence than a casual reading might suggest: stated criteria, output rules, human-machine agreement, human baselines, order testing, and bias analyses. The question is what else your intended use requires.

And nothing here depends on the rater being a model. Human annotation teams can interpret the same vague criteria differently. An LLM judge compared against an uncalibrated human team may reproduce that team's preferences without establishing the standard you actually need. Both sides of the comparison deserve scrutiny.

Behavioral definitions

"Depth" could mean conceptual sophistication, causal explanation, or completeness. "Creativity" might mean useful originality in one task and become a distraction in another. The quality domain needs enough definition that someone can explain what belongs in it and what would count as evidence of better performance.

The authors acknowledge that helpfulness, accuracy, relevance, and creativity are combined into one metric. A holistic score can be useful. But naming several considerations does not establish that the resulting verdict represents all of them adequately.

This creates a risk of construct underrepresentation: important parts of the intended quality domain may be neglected. Missing definitions do not prove that neglect occurred. They make coverage harder to evaluate, which is why I would test cases where improving each intended dimension should affect the judgment.

Anchors

The 1-to-10 scale has no score-level descriptors. Examples of poor, adequate, and excellent performance would make the intended distinctions more explicit and testable. They would also give a team something concrete to revisit when a judge version changes.

Husain (2024) describes the practical difficulty of uncalibrated multidimensional rating scales: evaluators can interpret the difference between adjacent scores differently. Anchors help specify that difference. They still need validation; writing a description beside each number does not automatically make the scale consistent or its intervals equal.

Tradeoffs

Consider an accurate but terse answer and a detailed answer with one factual error. Which should win? The answer may depend on the task, the seriousness of the error, and the user's needs.

You do not necessarily need numerical weights. You do need a defensible policy for resolving consequential conflicts, with examples that make the policy inspectable. The default prompt leaves those choices largely to the judge's learned preferences. I would want to know whether those preferences match the decision the organization intends to make.

Calibration

A useful calibration process starts with exemplars, documented adjudication, and a check that the rater can apply the intended standard. A pretrained model may already have substantial evaluative ability. The check establishes whether that ability carries over to this rubric and these tasks.

Adjudication also needs scrutiny. A resolved label is an agreed reference standard under a documented procedure; it is not automatically objective truth. Where expert disagreement remains defensible, preserve it and specify how the decision process handles it. Keep calibration examples separate from the held-out cases used to assess performance.

A reliability model

Percent agreement and order consistency are legitimate reliability evidence. They do not tell you how much score variation comes from the answers, the questions, the judge checkpoint, prompt wording, or repeated sampling.

Some variation across questions is exactly what you want to capture. Some becomes a problem when a score is generalized beyond the tested tasks. The design should let you estimate or control the sources that could materially change your conclusion, with uncertainty appropriate to the way observations were sampled and repeated.

Tie semantics

The same tie label can cover genuine equality, insufficient evidence, conflicting quality dimensions, or position inconsistency recoded during analysis. Those states may require different actions. A genuine tie might support choosing the cheaper model. An unstable judgment might require adjudication or a better test.

Keep enough information to tell them apart where possible. Dropping the entire bucket makes those decisions harder to inspect.

Validity scope

Evidence from a general preference benchmark does not automatically establish performance in medical QA, safety evaluation, or RAG faithfulness. Some evidence may transfer, but that transfer needs an argument and relevant checks. The authors' use of reference-guided judging for math is a concrete example of adapting the procedure to the task.

The follow-up literature kept testing the weaknesses

The three years since the paper read like a slow inventory of the problems above.

Length-controlled AlpacaEval addressed length statistically, estimating preferences at equal response length. Its correlation with human Arena rankings rose from 0.94 to 0.98 (Dubois et al., 2024). That is evidence for a particular adjustment in a particular setting. It also illustrates a useful habit: measure a suspected nuisance variable and test what happens when you account for it.

Arena-Hard's tooling added explicit Style Control for length and markdown features in late 2024. Its confidence-aware analysis found MT-Bench separating about 23% of model pairs among the top 20, using non-overlapping 95% confidence intervals as the separation criterion. A conventional ranking-agreement formulation had yielded 91% (Li et al., 2025).

Those are different statistics answering different questions. The 23% figure measures separability; the 91% figure measures agreement. It would be misleading to present the difference as one score collapsing after an error bar was added. The practical result is still relevant: under the separation criterion used in that analysis, many model pairs remained unresolved. A ranked list can suggest distinctions the underlying estimates do not clearly support.

Shi et al. (2025) found position bias varying systematically by judge, task, and quality gap across more than 150,000 evaluations. JudgeSense tested 880 items across 25 judges under two meaning-preserving wordings of the instructions. Rewording reduced agreement on all four tasks studied, with two clearing the authors' threshold for a practically meaningful effect (Bellibatlu et al., 2026).

JudgeSense also compared those changes with the judge's agreement with itself under identical prompts. That matters. If the same call already produces varying verdicts, you need that baseline before attributing every change to the rewording. Prompt wording is part of the instrument, and repeat-call variability is part of what you need to measure.

How far the pattern spread

The sources do not establish that tools such as RAGAS and DeepEval copied MT-Bench's prompt text. The shared pattern is broader: candidate answers, natural-language criteria, a model judge, and an evaluative output, often accompanied by a rationale.

The available repairs also matter. RAGAS offers anchored rating scales with behavioral descriptions, and DeepEval supports explicit rubric objects connecting score ranges to described performance levels. These features address part of the specification problem. Whether a particular implementation is calibrated for a particular use still has to be tested.

I cannot infer every team's validation practices from public benchmark papers or tool documentation. What the evidence does show is that agreement with human preference, even when reasonably high, leaves several measurement questions unresolved. Adopting the judging architecture carries those questions into the new use unless the team addresses them.

That is where I would focus an audit. What interpretation does the organization attach to the score? What evidence supports it? And which unresolved weakness could actually change the decision?

What to do Monday morning

Start with one question: how do you currently check whether your raters agree, and who owns that check? If the answer is "we don't" or "the vendor does," you have a gap in what you can establish about the measurement process. You cannot yet tell whether disagreement is negligible, expensive, or concentrated in the cases that matter most.

For a team using an LLM judge or buying judged results, I would start with these checks:

  1. Test order and repeatability. Run pairwise comparisons in both orders on a representative audit set, with deliberate coverage of close calls. Repeat identical calls on a subset to estimate ordinary sampling variability. Report inconsistency alongside win rate and assess whether it changes the model choice. For consequential decisions, include both orders in the decision protocol itself.
  2. Keep unresolved cases visible. Report agreement with ties included, the tie and inconsistency rates, and a chance-corrected statistic appropriate to the design. Preserve the distinction between an explicit tie and an inconsistency recoded as one. If a vendor gives you a decisive-vote agreement figure, ask for the full-case result and the number of cases each figure includes.
  3. Check stability and sensitivity. Test edits that should preserve judged quality, such as answer order, alongside controlled changes that should improve or worsen it. Verify that the supposedly irrelevant edits really are irrelevant to the rubric. A judge that never changes its verdict can be perfectly stable and still miss every meaningful improvement.
  4. Gate the complete judging procedure against a documented standard. Use held-out adjudicated exemplars, preserve defensible disagreement, and set acceptance criteria before looking at the results. Include uncertainty and decision-relevant failure types. Recheck after consequential changes to the model checkpoint, prompt, rubric, reference material, or task distribution.

These checks help establish whether a weakness could change the decision you are making. They will not turn every score into a valid measure of every capability. They will give you a much clearer account of what you can trust and what still needs work.

What Part 2 will test

In Part 2, I will use the same rubric and run controlled experiments across multiple judge models. I want to know when a verdict changes, whether it should have changed, and whether the change affects the conclusion someone would draw from the evaluation.

One set of tests will examine stability under changes intended to preserve quality: position swaps, redundant padding, and style edits. Repeated calls with identical inputs will provide a baseline for sampling variability. The edits need checking too. Padding that makes an answer harder to use may genuinely reduce quality under a usability criterion, even if it adds no new factual content.

A second set will test sensitivity to changes that should matter, such as correcting a factual error or adding a missing step required by the task. I will report whether the verdict moves in the expected direction, where the change is large enough to warrant a different verdict. A third set will introduce conflicting dimensions to examine tradeoffs. Those cases need an explicit policy or a documented unresolved outcome; a verdict change is not automatically a failure.

The central results will be stability under irrelevant changes and appropriate sensitivity to relevant ones, reported separately with uncertainty. I also want to see what happens to model rankings or pass/fail decisions. A flip rate tells you how often something moved. The practical question is whether the movement reveals a measurement problem that would change what you do.

References

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. American Educational Research Association. https://www.testingstandards.net/

Bean, A. M., Rocher, L., Kearns, R. O., Romanou, A., et al. (2025). Measuring what matters: Construct validity in large language model benchmarks. arXiv. https://arxiv.org/abs/2511.04703

Bellibatlu, R. R., Raff, E., & Zhang, W. (2026). JudgeSense: A benchmark for prompt sensitivity in LLM-as-a-judge systems. arXiv. https://arxiv.org/abs/2604.23478

Chen, J., Chen, W., Lin, Z., & Vong, C. M. (2026). A judge should know what changed: Construct validity for LLM-as-a-judge evaluation. arXiv. https://arxiv.org/abs/2608.24419

Dubois, Y., Galambosi, B., Liang, P., & Hashimoto, T. B. (2024). Length-controlled AlpacaEval: A simple way to debias automatic evaluators. arXiv. https://arxiv.org/abs/2404.04475

Husain, H. (2024). Creating a LLM-as-a-judge that drives business results. https://hamel.dev/blog/posts/llm-judge/

Li, T., Chiang, W.-L., Frick, E., Dunlap, L., Wu, T., Zhu, B., Gonzalez, J. E., & Stoica, I. (2025). From crowdsourced data to high-quality benchmarks: Arena-Hard and BenchBuilder pipeline. In Proceedings of the 42nd International Conference on Machine Learning (PMLR 267, pp. 34209–34231). https://proceedings.mlr.press/v267/li25h.html

lmarena. (2024). Arena-Hard-Auto [Computer software]. https://github.com/lmarena/arena-hard-auto

LMSYS Org. (2023). FastChat [Computer software]. https://github.com/lm-sys/FastChat

Pombal, J., Rei, R., & Martins, A. F. T. (2026). Self-preference bias in rubric-based evaluation of large language models. arXiv. https://arxiv.org/abs/2604.06996

Shi, L., Ma, C., Liang, W., Diao, X., Ma, W., & Vosoughi, S. (2025). Judging the judges: A systematic study of position bias in LLM-as-a-judge. In Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (pp. 292–314). https://aclanthology.org/2025.ijcnlp-long.18/

Wang, P., Li, L., Chen, L., Cai, Z., Zhu, D., Lin, B., Cao, Y., Liu, Q., Liu, T., & Sui, Z. (2023). Large language models are not fair evaluators. arXiv. https://arxiv.org/abs/2305.17926

Wataoka, K., Takahashi, T., & Ri, R. (2024). Self-preference bias in LLM-as-a-judge. arXiv. https://arxiv.org/abs/2410.21819

Wu, M., & Aji, A. F. (2023). Style over substance: Evaluation biases for large language models. arXiv. https://arxiv.org/abs/2307.03025

Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., & Stoica, I. (2023). Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems (Vol. 36). https://arxiv.org/abs/2306.05685