Validating LLM-as-a-Judge Systems in the Absence of Gold Labels (2025)

Table of Contents

1 Introduction 2 Preliminaries 2.1 A Probabilistic Model for Ratings 2.2 Evaluation Goals 3 Decomposing Sources of Human Rating Variation in LLM-as-a-Judge Validation 4 Defining the Performance of a Judge System in the Absence of Gold Labels 5 Ranking Judge Systems Under Competing Definitions of Performance 6 Reconciling Definitions of Performance via Synthetic Experiments 7 Case Study: Validating LLM-as-a-Judge Systems for Toxicity Detection 8 Conclusion 9 Impact Statement 10 Acknowledgments References Appendix A Appendix. Appendix B Additional Theoretical Analysis B.1 Proofs B.2 Rank Consistency Under Rater Error Appendix C Examples of Monotonicity Violations Among Pairs of Performance Metrics Appendix D Additional Synthetic Experiment Setup Details and Results D.1 Setup Details Appendix E Additional Case Study Setup Details and Results

Luke Guerdan Solon Barocas Kenneth Holstein Hanna Wallach Zhiwei Steven Wu Alexandra Chouldechova

Abstract

The LLM-as-a-judge paradigm, in which a judge LLM system replaces human raters in rating the outputs of other generative AI (GenAI) systems, has come to play a critical role in scaling and standardizing GenAI evaluations. To validate judge systems, evaluators collect multiple human ratings for each item in a validation corpus, and then aggregate the ratings into a single, per-item gold label rating. High agreement rates between these gold labels and judge system ratings are then taken as a sign of good judge system performance. In many cases, however, items or rating criteria may be ambiguous, or there may be principled disagreement among human raters. In such settings, gold labels may not exist for many of the items. In this paper, we introduce a framework for LLM-as-a-judge validation in the absence of gold labels. We present a theoretical analysis drawing connections between different measures of judge system performance under different rating elicitation and aggregation schemes. We also demonstrate empirically that existing validation approaches can select judge systems that are highly suboptimal, performing as much as 34% worse than the systems selected by alternative approaches that we describe. Based on our findings, we provide concrete recommendations for developing more reliableapproaches to LLM-as-a-judge validation.

Machine Learning, ICML

1 Introduction

To improve efficiency, scalability, and repeatability, GenAI evaluations commonly rely on LLM-based judges as a substitute for human raters when rating system outputs for properties like their “relevance”, “helpfulness”, or “toxicity.” In this LLM-as-a-judge paradigm, illustrated in in Figure1, a judge LLM system is used to rate the outputs of a target GenAI system according to instructions specified in a rating task(e.g., Szymanski etal., 2024; Zheng etal., 2023; Bubeck etal., 2023). However,when using this paradigm,validating that judge systems produce accurate ratings is critical to the reliabilityof the resulting GenAI evaluations.

To validate a judge system, evaluators first curate a validation corpus in which each item in the corpusasks a rater to rate a single target system output according to a set of rating task instructions. Each item is then rated by multiple human raters and by the judge system.The human ratings are then aggregated into a single gold label rating for each item in the validation corpus. Judge system performance is then assessed by calculating the agreement rate between these gold labels and judge system ratings(Lu & Zhong, 2024; Kim etal., 2024; Jung etal., 2024; Dong etal., 2024; Es etal., 2023; Dubois etal., 2024; Shankar etal., 2024).

Validating LLM-as-a-Judge Systems in the Absence of Gold Labels (1)

Although this approach is well-motivated when there is a single “correct” rating for each item, judge systems are often used in settings where this gold-label assumption is violated. Items or rating criteria may be ambiguous, or there may be principled disagreement among human raters, leading to multiple “correct” ratings. We say that such rating tasks are indeterminate. In practice, indeterminate rating tasks are very common (Figure 9). For example, raters may reasonably disagree about the “helpfulness” of an item based on its length(Li etal., 2024a), or note that the “toxicity” of an item depends on the cultural context (Goyal etal., 2022).

We show that task indeterminacy substantively affectshow judge systems can be validated, which judge systems are selected, and most critically, the resulting conclusions that are drawn about target systems. We introduce a framework for understanding different approaches to LLM-as-a-judge validation as determined by the rating task design, including the rating elicitation scheme; the rating aggregation scheme; and the metric used to quantify human–judge agreement. We illustrate both theoretically and empirically the implications of these choices on judge system selection and on the resulting conclusions that are drawn about targetsystems. Our main contributions are listed below:

•
We provide the first framework for LLM-as-a-judge validation under rating task indeterminacy—i.e., where many items may have multiple “correct” ratings. This framework enables us to compare existing validation approaches and to develop principled alternatives.
•
We use this framework to compare existing validation approachesto alternative approaches that account for task indeterminacy. We demonstrate that the best-performing judge systems using the former can be among the worst-performing using the latter (§5), and explain how this arises from differences in how human raters and judge systems resolve ambiguities in forced-choice rating tasks (§6). These findings highlight the importance of the rating elicitation scheme, as well as pinpointing a mechanism by which rating task indeterminacy can confound LLM-as-a-judge validation.
•
We conduct an empirical study in which we use five different commercial LLMs as judge systems to rate the “toxicity” of a target system’s outputs using the Civil Comments dataset (§7). We demonstrate that the judge system selected by commonly used existing validation approaches performs 34% worse than the judge system selected when using our framework to explicitly account for the effects of rating task indeterminacy.

Our findings demonstrate that existing validation approaches can be highly suboptimal when used for indeterminate rating tasks. Because the resulting conclusions that are drawn about target systems depend critically on judge system performance, this then jeopardizes the validity of GenAI evaluations.We draw on our findings to offer concrete recommendations for developing more reliable approaches to LLM-as-a-judge validation under rating task indeterminacy.

2 Preliminaries

Let $\mathcal{G}_{\text{judge}}$ denote the judge system and let $\mathcal{G}_{\text{target}}$ denote the target system. We treat $\mathcal{G}_{\text{target}}$ as a black box that produces a string of output tokens given a string of input tokens. Output tokens can be represented as an arbitrary modality (e.g., text, images, audio, video) depending on the system.

Rating Task. We express a target system evaluation as a rating task consisting of $n$ items $\mathcal{T}=\{t_{1},t_{2},...,t_{n}\}$ . Each item $t_{i}\in\mathcal{T}$ consists of 1) an output generated by $\mathcal{G}_{\text{target}}$ , 2) instructions for rating that output, and 3) a set of response options—i.e., possible ratings—for that output.As is often the case in practice, we assume that the rating instructionsand response options are the same for all items.

Our framework is compatible with two types of rating tasks (Zheng etal., 2023): Single output grading tasks instruct a rater (either human or judge system) to rate an output generated by $\mathcal{G}_{\text{target}}$ for a specific property like “helpfulness” or “toxicity”). Pairwise comparison tasks instruct a rater to provide a preference among two outputs generated by one or more target systems. Both types of rating tasks often include a rubric, annotation guidelines, and few-shot examples specifying how outputs should be rated. Letting $\mathcal{O}=\{o_{1},o_{2},...o_{q}\}$ denote the ordered set of response options for a rating task, two common choices for response options are $\mathcal{O}=\{\text{Yes},\text{No}\}$ for single output grading tasks and $\mathcal{O}=\{\text{Win},\text{Tie},\text{Lose}\}$ for pairwise comparison tasks.

Rating Elicitation. We examine two rating elicitation schemes for eliciting ratings from human raters and judge systems: forced choice and response set (Figure 2). Forced choice elicitation instructs a rater to select a single option from $\mathcal{O}$ . Response set elicitation instructs a rater to select all options from $\mathcal{O}$ that are reasonable. Letting $\mathcal{P}(\mathcal{O})$ denote the power set of $\mathcal{S}$ , we define $\mathcal{Q}=\{\mathcal{S}_{1},\mathcal{S}_{2},...,\mathcal{S}_{w}\}$ to be the ordered set of all admissible response sets. Response set elicitation instructs a rater to select a response set $\mathcal{S}\in\mathcal{Q}$ . For some tasks, $|\mathcal{O}|=|\mathcal{Q}|$ , in which case, the twoelicitationschemes carry equivalent information. However, differences arise when rating tasks are underspecified.

Definition 2.1 (Underspecified Rating Task).

A rating task is underspecified if $|\mathcal{O}|\leq|\mathcal{Q}|$ .

For example, consider the single output grading task shown in Figure 2, where $\mathcal{O}=\{\text{Yes},\text{No}\}$ and $\mathcal{Q}=\{\text{Yes},\text{No},\{\text{Yes},\text{No}\}\}$ . This task is underspecified because a rater can select a response set containing both Yes and No under response set elicitation if they determine both options are reasonable. We can construct a fully specified variant of this rating task by adding a Maybe option, and instructing raters to always select Maybe if they determine both Yes and No are reasonable options. When tasks are underspecified, forced choice elicitation capturesless information than response set elicitation.

Validating LLM-as-a-Judge Systems in the Absence of Gold Labels (2)

Human Raters. Let $\mathcal{R}$ denote a population of human raters, such as all target system users in a geographic region, a demographic group (e.g., females over 45), or a set of domain experts (e.g., licensed radiologists). We let $R$ denote a random variable modeling the selection of raters from $\mathcal{R}$ .

Judge System. We assume black-box access to the judge system, $\mathcal{G}_{\text{judge}}$ , which, given an input item $t_{i}$ , returns an output that is then mapped to a response option in $\mathcal{O}$ under forced choice elicitation or $\mathcal{Q}$ under response set elicitation.

2.1 A Probabilistic Model for Ratings

We model the distribution of human and judge system ratings via a joint distribution $\mathbb{P}(T,O^{j},S^{j},O^{h},S^{h},R)$ . Here, the random variables $O^{j}$ and $S^{j}$ denote the forced choice and response set ratings, respectively, returned by the judge system for a random item $T$ . Similarly, $O^{h}$ and $S^{h}$ denote the forced choice and response set ratings, respectively, that a randomly drawn rater $R$ assigns to an item $T$ . We evaluate the quality of judge system ratings by comparing them with human ratings a per-item basis. For each item $i$ , let $\mathbb{P}_{i}^{h}=\mathbb{P}(O^{h},S^{h}\mid T=t_{i})$ denote the human rating distribution and let $\mathbb{P}_{i}^{j}=\mathbb{P}(O^{j},S^{j}\mid T=t_{i})$ denote the rating distribution of $\mathcal{G}_{\text{judge}}$ . We omit $h$ and $j$ when referring to a generic rater (human or judge system) and let $\mathbb{P}_{T}$ denote a rating distribution conditioned on a random item $T$ .

Aggregation Functions. We introduce aggregation functions $a:\Delta\rightarrow\mathcal{Y}$ to consolidate the full rating distribution into a rating vector. For example, applying a hard aggregation function $\mathbf{y}=a_{\text{hard}}(\mathbb{P}_{i})$ recovers a binary one-hot vector encoding a single rating task option (e.g., Yes) (see $\S$ 4). The rating space $\mathcal{Y}=\{a(\mathbb{P}_{i}):\mathbb{P}_{i}\in\Delta\}$ contains all rating vectors that can be recovered from an aggregation function.

We use aggregation functions to define random variables over rating vectors. Specifically, let $Y=a(\mathbb{P}_{T})$ denote the random rating vector obtained by applying an aggregation function to the rating distribution of a random item $T$ . This random variable setup enables us to reason probabilistically about aggregated ratings – e.g., by computing the expected agreement between aggregated rating vectors recovered from humans and the judge system (see $\S$ 2.2). Let $(a^{h},Y^{h},\mathbf{y}^{h},\mathcal{Y}^{h})$ and $(a^{j},Y^{j},\mathbf{y}^{j},\mathcal{Y}^{j})$ denote the aggregation function, random rating vector, rating vector realization, and rating space for humans and $\mathcal{G}_{\text{judge}}$ , respectively.

2.2 Evaluation Goals

Our goal is to characterize the validity of using a judge system as a surrogate for human ratings in evaluations of the target system. We approach validation of the judge system from two complementary angles: first, by directly measuring human–judge agreement ( $\S$ 2.2.1), and second, by examining the extent to which relying on judge system ratings in place of human ratings affects the conclusions drawn from downstream evaluations of $\mathcal{G}_{\text{target}}$ ( $\S$ 2.2.2).

2.2.1 Measuring Human-Judge Agreement

The standard approach for validating judge systems involves computing a measure of human–judge agreement (Table 1). Specifically, given the joint distribution $\mathbb{P}(\cdot)$ , we evaluate

M(Y^{j},Y^{h})=\mathbb{E}_{(T,Y^{j},Y^{h})}[m(Y^{j},Y^{h})],

(1)

where $m:\mathcal{Y}^{j}\times\mathcal{Y}^{h}\rightarrow\mathbb{R}$ is an agreement metric (e.g., Hit-Rate, Cohen’s $\kappa$ , KL-Divergence). The expectation is taken over the joint distribution of random items $T$ and corresponding aggregated rating vectors $Y^{j}$ and $Y^{h}$ .

While Eq.(1) assumes that we know the rating distribution, in practice, we only have access to a small corpus of ratings. Therefore, we also estimate the agreement rate,

\hat{M}(\hat{Y}^{j},\hat{Y}^{h})=\mathbb{E}_{(T,Y^{j},Y^{h})}[m(\hat{Y}^{j},%\hat{Y}^{h})].

(2)

Above, we estimate $\hat{Y}^{h}$ from a corpus of human ratings $\mathcal{C}=\{(T_{v},R_{v},T_{v})\}_{v=1}^{N}\overset{\text{iid}}{\sim}\mathbb%{P}(\cdot)$ . We assume this corpus only contains forced choice ratings, as this is the format used in existing GenAI evaluations. For each item, we estimate $\hat{Y}^{j}$ by repeatedly sampling a response from $\mathcal{G}_{\text{judge}}$ .

2.2.2 Measuring the Performance of a Judge System on Downstream Evaluation Tasks

We also examine how selecting judge systems based on certain human–judge agreement metrics impacts judge system performanceon downstream evaluation tasks.¹¹1This is analogous to how in classical supervised learning we might select a probabilistic classifier $\hat{f}=\hat{f}_{\lambda}$ by choosing $\lambda$ to minimize cross-validated negative log-likelihood. But then the model may get applied to a task where its misclassification error at a particular threshold is the more relevant performance metric. We focus on two specific downstream tasks. In content filtering tasks, the judge system is used to identify which outputs from $\mathcal{G}_{target}$ to allow or suppress. In prevalence estimation tasks, the judge system is used to estimate the prevalence of a certain property (e.g., contain “toxic” language) in $\mathcal{G}_{target}$ outputs.

Both content filtering and prevalence estimation require making binary categorizations of each item (e.g., whether an item contains “toxicity”). We categorize items using a threshold function $s_{k}^{\tau}(\mathbf{y})=\mathbbm{1}[\mathbf{y}_{k}\geq\tau]$ that labels an item as positive if the kth option has at least a $\tau$ chance of being selected. The cutoff $\tau$ is a policy determination. For example, authors of the Civil Comments toxicity classification dataset (Borkan etal., 2019) use $\tau=0.5$ whidetermining whether to categorize an item as toxic or non-toxic.xic

Content Filtering. We evaluate $\mathcal{G}_{\text{judge}}$ on content filtering tasks by measuring how often it makes the same allow/suppress decisions as the population of human raters. We quantify this via the decision consistency metric:

C^{\tau}_{k}(Y^{j},Y^{h})=\mathbb{E}_{(T,Y^{j},Y^{h})}[\mathbbm{1}[s^{\tau}_{k%}(Y^{j})=s^{\tau}_{k}(Y^{h})]].

High consistency indicates that, if deployed, $\mathcal{G}_{\text{judge}}$ would often make the same decisions as human raters on which outputs from $\mathcal{G}_{\text{target}}$ to allow or suppress (at the cutoff $\tau$ ).

Prevalence Estimation. In prevalence estimation tasks, $\mathcal{G}_{\text{judge}}$ is used to evaluate the proportion of outputs from $\mathcal{G}_{\text{target}}$ that have a certain property. For example, in single-output rating tasks, prevalence estimation recovers the proportion of $\mathcal{G}_{\text{target}}$ outputs that are “relevant” or “toxic.” In pairwise comparison tasks, a prevalence estimation recovers the win rate — i.e., the proportion of items where an output from one target model $\mathcal{G}_{\text{target}}^{z}$ is rated as preferable to an output from a second target model $\mathcal{G}_{\text{target}}^{w}$ (Chiang etal., 2024). We measure the estimation bias between estimates obtained from human raters versus the judge system via

B^{\tau}_{k}(Y^{j},Y^{h})=\mathbb{E}[s^{\tau}_{k}(Y^{j})]-\mathbb{E}[s^{\tau}_%{k}(Y^{h})].

For example, when using $\mathcal{G}_{\text{judge}}$ to rate $\mathcal{G}_{\text{target}}$ responses to automated red-teaming attacks designed to elicit toxicity (Mazeika etal., 2024; Ganguli etal., 2022), $B<0$ indicates that $\mathcal{G}_{\text{judge}}$ underestimates the prevalence of toxic outputs (i.e., the attack success rate) as compared to human ratings.

Prevalence estimation has become a central task in the GenAI evaluation literature. The popular Chatbot Arena leaderboard uses a Bradley-Terry model to estimate the win rate (a form of prevalence estimate) in pairwise comparison tasks (Chiang etal., 2024). Similarly, Prediction Powered Inference (PPI) is increasingly used to improve the sample-efficiency of prevalence estimators by combining gold label human ratings with judge system ratings (Angelopoulos etal., 2023b, a; Boyeau etal., 2024; Eyre & Madras, 2024).²²2PPI setups assume that humans and the judge system rate mutually exclusive subsets of items. In contrast, we validate the judge system by assuming that a subset of items have been rated by both humans and the judge system. Nevertheless, our findings challenge the reliability of using a single per-item human rating as a gold label – a foundational assumption in the PPI framework. In this work, we show that current approaches used to elicit and aggregate human ratings can yield misleading evaluations of target systems. Because human ratings also serve as a foundation for PPI estimators and the Bradley-Terry model, our findings in turn call into question the reliability of these widely-used methodologies for GenAI evaluation.

3 Decomposing Sources of Human Rating Variation in LLM-as-a-Judge Validation

With this framework in place, we now develop a model that decomposes sources of rating variation in the LLM-as-a-judge validation pipeline. Because human ratings are the benchmark for validating judge systems, disentangling “meaningful signal” from noise in rating variationis critical. Our model disentangles: (1) genuine differences in how raters interpret a rating task; (2) inconsistencies introduced by lapses in attention (i.e., error); and (3) variation introduced by requiring a rater to select a single option when they determine that more than one is reasonable — i.e., forced choice selection effects. We show that failing to account for each factor can lead to misleading evaluation results ( $\S$ 6-7).

Our rating model decomposes the human rating distribution for each item: $\mathbb{P}_{i}^{h}=\mathbb{P}(S_{*}^{h},S^{h},O^{h},\mid t_{i})$ . To capture the potential for rater error, we distinguish between a human rater’s stable response set $S_{*}^{h}$ — i.e., the options they would consistently endorse when carefully completing the rating task — and the observed response set $S^{h}$ they provide through response set elicitation (Figure 3). A rater’s stable response set can differ from their observed response set if they fail to identify one or more options that could reasonably apply to a rating instruction (or erroneously endorse others). We describe differences between the stable and observed response set via an error matrix $\boldsymbol{\mathbf{E}}_{i}\in\mathbb{R}^{|\mathcal{Q}|\times|\mathcal{Q}|}$ , where each entry encodes the probability that a rater endorses $\mathcal{S}_{v}$ given that their stable response set is $\mathcal{S}_{v^{*}}$ . We assume that error rates are constant across all raters:

Assumption 3.1 (Error Independence).

$S^{h}\perp R\mid S_{*}^{h},T$ .

While a rich literature exists on rater-dependent error modeling (Klie etal., 2023; Gordon etal., 2021), we make this simplifying assumption to examine the aggregate effects of rating error on downstream evaluations of judge systems.

We use a transition matrix $\mathbf{F}_{i}\in\mathbb{R}^{|\mathcal{O}|\times|\mathcal{Q}|}$ to represent how raters pick an option from their response set. Each element in $\mathbf{F}_{i}$ contains the probability of a rater selecting the $k$ th option (e.g., Yes) given that they would select the $v$ th response set (e.g., both Yes and No). As with the error matrix, we also assume that $\mathbf{F}_{i}$ is fixed across raters:

Assumption 3.2 (Forced Choice Independence).

$O^{h}\perp\{S_{*}^{h},R\}\mid S^{h},T$ .

Both $\mathbf{E}_{i}$ and $\mathbf{F}_{i}$ have a reverse matrix, denoted as $\mathbf{E}^{\prime}_{i}$ and $\mathbf{F}^{\prime}_{i}$ , respectively, that encode conditional probabilities in the reverse direction. Entries in $\mathbf{F}^{\prime}_{i}$ denote the probability of a rater endorsing the $v$ th (observed) response set (e.g., Yes and No) given that they selected the $k$ th forced choice option (e.g., Yes). Entries in $\mathbf{E}^{\prime}_{i}$ denote the probability of a rater endorsing $\mathcal{S}_{v^{*}}$ given that their observed response set is $\mathcal{S}_{v}$ .

Our rating model connects different representations of human rating variation (Fig. 3). The response set distribution $\boldsymbol{\theta}^{*}_{i}=\mathbb{P}(S_{*}^{h}=s_{v^{*}}\mid t_{i})$ represents genuine differences in how a population of raters interprets an item in a rating task. This rating distribution, which is uncorrupted by error or forced choice selection effects, is our target parameter. In contrast, the forced choice distribution $\boldsymbol{O}_{i}=\mathbb{P}(O^{h}=o_{k}\mid t_{i})$ describes the probability distribution that is observed under rater error and forced choice selection effects. The following result shows that we can decompose response set distribution into rater error, forced choice selection effects, and the forced choice distribution:

Theorem 3.3.

(Rating Decomposition) Assume 3.1 and 3.2 hold on $\mathbb{P}(\cdot)$ . Then $\boldsymbol{O}_{i}=\mathbf{F}_{i}(\mathbf{E}_{i}\boldsymbol{\theta}^{*}_{i})$ and $\boldsymbol{\theta}^{*}_{i}=\mathbf{E}_{i}^{\prime}(\mathbf{F}_{i}^{\prime}%\mathbf{O}_{i})$ holds for all conditional rating distributions $\mathbb{P}_{i}^{h}\in\Delta$ .

This theorem shows how genuine differences in raters’ interpretation of an item in a rating task propagates through error and forced choice selection. It also provides a mechanism for recovering $\boldsymbol{\theta}^{*}_{i}$ from $\boldsymbol{O}_{i}$ by applying the reverse error and forced choice transition matrices. Given this decomposition, we might wonder when the response set distribution is identifiable from $\mathbf{O}_{i}$ . The following result shows that this is only possible when a rating task is fully specified:

Theorem 3.4 (Response Set Identifiably).

Assume 3.1 and 3.2 hold on $\mathbb{P}(\cdot)$ . Further, assume that $\mathcal{O}\subseteq\mathcal{Q}$ and let $\mathbf{E}_{i}$ be the identity matrix. Then $\boldsymbol{\theta}^{*}_{i}$ is identifiable from $\mathbf{O}_{i}$ if and only if the rating task is fully specified.

This theorem shows that, even in an idealized setting with no rater error, an information loss occurs when compressing a response set rating into a forced choice rating. A practical implication is that rating tasks should be fully specified when possible to enable direct recovery of $\boldsymbol{\theta}_{i}^{*}$ from $\mathbf{O}_{i}$ . In $\S$ 6-7, we show that underspecified rating tasks yield substantive discrepancies in validations of judge systems.

4 Defining the Performance of a Judge System in the Absence of Gold Labels

Our model for human rating variation establishes how to define the performance of a judge system under indeterminacy. In particular, let $a^{h}$ , $a^{j}$ denote human and judge aggregation functions used to consolidate rating variation (i.e., represented via the forced choice or response set distributions) into a rating vector. Given a judge performance metric $m$ , we call $\mathbf{p}=(a^{h},a^{j},m)$ a definition of performance. As we describe next, many such definitions could reasonably be used to validate a judge system. We describe each definition of performance by enumerating over aggregation functions:

Hard aggregation. The hard aggregation function is defined $a_{\text{hard}}(\mathbb{P}_{i})=\mathbf{e}_{k^{*}}$ , where $\mathbf{e}_{k^{*}}$ is an $|\mathcal{O}|$ -dimensional basis vector and $k^{*}=\arg\max_{k}\mathbf{O}_{i,k}$ is the mode of the forced choice distribution. Performance measures that rely on hard aggregation are consistent with categorical human–judge agreement metrics (e.g., Krippendorff’s $\alpha$ ). Measures relying on hard aggregation impose a gold-label assumption, and are the status quo in existing judge system validations (Lu & Zhong, 2024; Jung etal., 2024; Dong etal., 2024; Es etal., 2023; Dubois etal., 2024; Bubeck etal., 2023; Zheng etal., 2023; Faisal etal., 2024; Gu etal., 2024; Thakur etal., 2024; Li etal., 2024b; Chen etal., 2024; Chiang etal., 2024; Dorner etal., 2024; Mirzakhmedova etal., 2024; Chaudhary etal., 2024; Kim etal., 2024; Dettmers etal., 2024).

Soft aggregation. The soft aggregation function $a_{\text{soft}}(\mathbb{P}_{i})=\mathbf{O}_{i}$ returns a probability vector over forced choice responses. Each entry $\mathbf{O}_{i,k}$ represents the probability that the $k$ th option is selected by a rater under forced choice elicitation. Definitions of performance that rely on soft aggregation are consistent with distributional human–judge agreement metrics (e.g., KL-Divergence). Prior work has proposed soft label aggregation with distributional agreement metrics for evaluating ML systems under indeterminacy (Uma etal., 2020; Peterson etal., 2019; Collins etal., 2022). However, soft aggregation is seldom used in judge system validations.

Our rating model ( $\S$ 3) connects these categorical and distributional definitions of performance and multi-label definitions. Multi-label definitions provide a more granular representation of rating variation over response set data. Let $\boldsymbol{\Lambda}_{i}\in\{0,1\}^{|\mathcal{O}|\times|\mathcal{Q}|}$ be a binary matrix indicating whether the $k$ th option is in the $v$ th response set. We define the multi-label vector as $\mathbf{\Omega}_{i}=\boldsymbol{\Lambda}_{i}(\mathbf{E}_{i}\boldsymbol{\theta}%^{*}_{i})$ . Each entry in $\mathbf{\Omega}_{i,k}$ describes the probability that a rater selects the $k$ th option in their observed response set under response set elicitation.³³3Unlike the forced choice and response set distributions, entries in the multi-label vector need not sum to one. Let $\mathbf{\Omega}^{*}_{i}=\boldsymbol{\Lambda}_{i}(\boldsymbol{\theta}^{*}_{i})$ denote the corresponding multi-label vector that is uncorrupted by rater error. Two additional aggregation functions are consistent with multi-label vectors:

Hard Response Set. The hard response set (hrs) function $a_{\text{hrs}}(\mathbb{P}_{i})=\mathbbm{1}\{\mathbf{\Omega}_{i}\geq\tau\}$ maps the response set distribution to a binary multi-label vector. The $k$ th entry of this vector is one if there is at least a $\tau$ probability of a response set containing option $k$ being selected during response set elicitation. This aggregation function is consistent with measuring the coverage of a predicted judge system response in a response set containing multiple “correct” options.

Soft Response Set. The soft response set (srs) function $a_{\text{srs}}(\mathbb{P}_{i})=\mathbf{\Omega}_{i}$ directly returns the non-thresholded multi-label vector. Each entry $\mathbf{\Omega}_{i,k}$ denotes the probability that a rater endorsed the $k$ th option during response set elicitation. Definitions of performance that apply srs aggregation to the human rating distribution are consistent with continuous metrics such Mean Squared Error and Binary Cross Entropy.

Table 1 in Appendix A lists many definitions of performance that are consistent with these aggregation functions. This table also summarizes definitions of performance commonly used in (1) LLM-as-a-judge validations and (2) prior work studying evaluation under indeterminacy.

5 Ranking Judge Systems Under Competing Definitions of Performance

Given that there are many ways of defining performance under indeterminacy, it is unclear when one approach is preferable over another. One way to distinguish among competing definitions is by examining their downstream impact on judge system validation: when do two performance definitions yield a consistent ranking of judge systems? We now use our framework to formally investigate this question.

Let $\mathcal{G}_{judge}^{w}$ and $\mathcal{G}_{judge}^{z}$ denote two judge systems, each described by their conditional rating distributions $\mathbb{P}_{T}^{j,z}$ and $\mathbb{P}_{T}^{j,w}$ , respectively. We can compare these systems with respect to a performance definition $\mathbf{p}$ via,

	$\displaystyle\delta_{\textbf{p}}(z,w)=\mathbb{E}_{\mathbb{P}^{*}}\bigl{[}$	$\displaystyle m\bigl{(}a^{j}(\mathbb{P}_{T}^{j,z}),a^{h}(\mathbb{P}_{T}^{h})%\bigr{)}$		(3)
	$\displaystyle-$	$\displaystyle m\bigl{(}a^{j}(\mathbb{P}_{T}^{j,w}),a^{h}(\mathbb{P}_{T}^{h})%\bigr{)}\bigr{]}$		(3)

where $\mathbb{P}^{*}$ represents the full joint distribution over responses returned by both judge systems and human ratings.

To formalize a comparison between two systems, we let $\mathcal{G}_{judge}^{z}\succeq_{\mathbf{p}}\mathcal{G}_{judge}^{w}$ denote that $\delta_{\mathbf{p}}(z,w)\geq 0$ . For instance, when using Hit Rate with hard aggregation, $\succeq_{\mathbf{p}}$ implies that $z$ achieves greater agreement with a majority vote over human ratings than $w$ .⁴⁴4For metrics where lower values indicate better performance, like KL-divergence, we invert the definition such that $\mathcal{G}_{judge}^{z}\succeq_{\mathbf{p}}\mathcal{G}_{judge}^{w}\iff\delta_{%\mathbf{p}}(z,w)\leq 0$ . Now, suppose that we would like to compare judge systems under a different definition of performance, denoted by $\mathbf{p}^{\prime}=(a_{h}^{\prime},a_{j}^{\prime},m^{\prime})$ . The following condition describes when these two definitions are guaranteed to yield an equivalent ranking of judge systems:

Definition 5.1 (Rank Consistency).

We say that $\mathbf{p}$ and $\mathbf{p}^{\prime}$ are rank consistent if for all $\mathbb{P}^{*}$ , $\mathcal{G}_{judge}^{z}\succeq_{\mathbf{p}}\mathcal{G}_{judge}^{w}\;\iff\;%\mathcal{G}_{judge}^{z}\succeq_{\mathbf{p}^{\prime}}\mathcal{G}_{judge}^{w}$ .

While there are many possible relationships between two definitions of performance, monotonicity captures one key property we might expect: when one system’s performance improves with respect to $\mathbf{p}$ , it should also improve with respect to $\mathbf{p}^{\prime}$ if the two definitions are compatible. We formalize this notion in the following definition:

Definition 5.2 (Monotone Transformation).

$\mathbf{p}$ is a monotone transformation of $\mathbf{p}^{\prime}$ if there exists a monotone increasing function $f$ such that $m^{\prime}\bigl{(}a^{\prime}_{j}(\mathbb{P}_{i}^{j}),a^{\prime}_{h}(\mathbb{P}%_{i}^{h})\bigr{)}=f\bigl{(}m\bigl{(}a^{j}(\mathbb{P}_{i}^{j}),a^{h}(\mathbb{P}%_{i}^{h})\bigr{)}\bigr{)}$ for all $(\mathbb{P}_{i}^{j},\mathbb{P}_{i}^{h})\in\Delta\times\Delta.$

The following result shows that if two performance definitions are not monotone transformations of one another, there exist judge systems and a distribution over human ratings such that the definitions will yield contradictory rankings:

Theorem 5.3.

(Necessary Condition for Rank Consistency)If $\mathbf{p}$ is not a monotone transformation of $\mathbf{p}^{\prime}$ , then $\mathbf{p}$ and $\mathbf{p}^{\prime}$ are not rank consistent.

Theorem 5.3 provides a useful tool for comparing definitions of performance: we can show that two definitions are not rank consistent by demonstrating a monotonicity violation.

We provide two examples of monotonicity violations in Appendix C. The first shows a violation between Hit Rate (defined over $\mathbf{O}$ ) and KL-Divergence (defined over $\mathbf{O}$ ). The second shows a violation between KL-Divergence (defined over $\mathbf{O}$ ) and Mean Squared Error (defined over $\mathbf{\Omega}$ ). This second example illustrates a pernicious issue arising in underspecified tasks: using Theorem 3.4, we can easily construct monotonicity violations by holding the forced choice distribution fixed while varying the response set distribution. This suggests that monotonicity, and by extension rank consistency, is unlikely to hold between definitions of performance defined over the forced choice distribution (i.e., categorical, distributional) and multi-label definitions.

Validating LLM-as-a-Judge Systems in the Absence of Gold Labels (3)

Validating LLM-as-a-Judge Systems in the Absence of Gold Labels (4)

Validating LLM-as-a-Judge Systems in the Absence of Gold Labels (6)

6 Reconciling Definitions of Performance via Synthetic Experiments

Given that a single violation in monotonicity can yield an inversion in system rankings, we might wonder how often inversions incur in practice. Yet merely documenting rank inversions would not provide a means of selecting among two definitions of performance that rank judge systems differently. Therefore, our experiments examine how well judge systems selected using different human–judge agreement metrics perform on downstream evaluation tasks.

Experiment Design. We use our rating decomposition ( $\S$ 3) to sample human and judge system rating distributions:

Human Rating Distribution. For each item, we sample the response set distribution $\boldsymbol{\theta}^{*,h}_{i}\sim\text{Dir}(\mathbf{1}_{|\mathcal{Q}|})$ . We let $\epsilon$ denote the probability that a rater selects an observed response set that differs from their stable response set. We construct the error matrix $\mathbf{E}^{h}_{i}$ such that diagonal entries denote the probability of no rating error $(1-\epsilon)$ and off-diagonal entries denote the probability of rating error ( $\epsilon$ ). We use a skew parameter $\eta$ to control how errors are distributed across response sets. We let $k=0$ denote the index of the option used to categorize items (e.g., as “toxic”). The skew parameter controls whether errors systematically favor ( $\eta>0$ ) or disfavor ( $\eta<0$ ) response sets containing option $k=0$ .

We model $\mathbf{F}_{i}^{h}$ by sampling an exponential decay function $\mathbf{F}^{h}_{i,k,v}\propto\exp(-\gamma^{h}\cdot r_{k})$ . Here, $r_{k}$ denotes the rank (low to high) of the $k$ th option in the $v$ th response set and $\gamma^{h}$ controls the strength of selection effects. We measure the magnitude of forced choice selection effects via

\Gamma^{h}=\frac{1}{|\mathcal{Q}_{+}|}\sum_{\mathcal{S}_{v}\in\mathcal{Q}}%\frac{1}{|\mathcal{S}_{v}|}P(O^{h}=o_{0}\mid o_{0}\in\mathcal{S}_{v}),

where $\mathcal{Q}_{+}$ denotes the set of response sets containing option at index $k=0$ . A value of $\Gamma^{h}=1$ indicates random chance of the first option being selected (i.e., no selection effects), while $\Gamma^{h}=2$ and $\Gamma^{h}=0.5$ indicate a rater is twice or half as likely as chance to select the first option (i.e., positive and negative selection effects, respectively). We then compute the forced choice distribution via $\mathbf{O}^{h}_{i}=\mathbf{F}_{i}^{h}(\mathbf{E}^{h}_{i}(\mathbf{\boldsymbol{%\theta}}^{*,h}_{i}))$ .

Judge Rating Distribution. We model judge systems by sampling an ensemble of distributions with varying similarity to the human rating distribution. We control the deviation of the $z$ th judge’s rating distribution via $\sigma^{z}\sim\mathcal{U}(\sigma_{\text{min}},\sigma_{\text{max}})$ . We then sample the $z$ th judges’ response set distribution by applying $\boldsymbol{\theta}_{i}^{*,z}=\Pi_{\Delta}\{\boldsymbol{\theta}^{*}_{i}+%\boldsymbol{\epsilon}_{i}^{z}\}$ , where $\boldsymbol{\epsilon}_{i}^{z}\sim\mathcal{N}(0,(\sigma^{z})^{2}\mathbf{I})$ and $\Pi_{\Delta}$ projects onto the probability simplex. We sample $\mathbf{F}_{i}^{z}$ following the same procedure used to sample the human rating distribution and let $\Gamma^{z}$ denote the magnitude of the $z$ th judge system’s forced choice selection effects. We assume that all variation in a judge systems’ response set distribution is captured in stable response sets – i.e., the judge system is not affected by rater error. As such, we let $\mathbf{E}_{i}^{z}$ be the identity matrix and compute $\mathbf{O}^{z}_{i}=\mathbf{F}_{i}^{z}(\mathbf{E}^{z}_{i}(\mathbf{\boldsymbol{%\theta}}^{*,z}_{i}))$ .

We refer to forced choice selection effects between humans and the $z$ th judge system as symmetric when $\text{sign}(\Gamma^{h}-1)=\text{sign}(\Gamma^{z}-1)$ . In contrast, we refer to forced choice selection effects as asymmetric when $\text{sign}(\Gamma^{h}-1)\neq\text{sign}(\Gamma^{z}-1)$ . We provide additional experiment setup details and describe our finite sample estimation approach in Appendix D.

Results. Figure 6 reports the performance of judge systems selected via different human–judge agreement metrics. The left panel shows the performance of judge systems on a fully specified rating task with no rating error. The center panel shows the performance of judge systems on an underspecified rating task with symmetric (left) and asymmetric (right) forced choice selection effects and no rating error. The right panel introduces additional random ( $\epsilon=0.3$ , $\eta=0$ ) and additive ( $\epsilon=0.3$ , $\eta=1$ ) rater error to the rating process.⁵⁵5The rightmost column is labeled “additive error” as the positive forced choice selection effects ( $\Gamma=2$ ) and positive skew ( $\eta=1$ ) jointly shift probability mass toward option $k=0$ .

Finding 1: Fully specified rating tasks make more effective use of a limited annotation budget. Figure 6 shows that the annotation budget — i.e., the number of human ratings collected for each item in the evaluation corpus — has a significant impact on the quality of selected judge systems. While current practice is to select judge systems via a single rating per item, increasing the budget to three ratings per item yields a larger benefit than accounting for other factors manipulated in our experiments (i.e., forced choice selection effects and the human–judge agreement metric). Further, fully specifying rating tasks enables more effective use of a limited annotation budget. Judge systems selected with just one rating per item on a fully specified task (Figure 6, left) match the performance of those selected with three ratings per item on an underspecified task (Figure 6, center) – a 66% reduction in annotation budget.

Finding 2: Categorical agreement metrics are unreliable in underspecified rating tasks with asymmetric selection effects. As shown in the center panel of Figure 6, using categorical agreement metrics to select judge systems is unreliable when (1) rating tasks are underspecified and (2) selection effects are asymmetric. Figure 6 corroborates these findings by indicating weak spearman correlation between categorical human–judge agreement metrics and downstream performance metrics under asymmetric selection effects. Because asymmetric selection effects cannot be detected from forced choice ratings alone⁶⁶6Prior work has documented that humans and LLMs exhibit asymmetric survey response biases similar to those modeled by forced choice selection effects (Tjuatja etal., 2024)., distributional agreement metrics should be used when selecting judge systems from forced choice ratings on underspecified tasks.

Finding 3: The impact of rating error on judge system selection varies by context and is sometimes less critical than forced choice selection effects. Given the substantial literature investigating rater error (Klie etal., 2023; Plank, 2022; Gordon etal., 2021), its relatively modest impact on judge system selection is unexpected. The right panel of Figure 6 shows that even with high error rates ( $\epsilon=0.3$ ) and adversarial conditions with additive effects ( $\eta=1$ ), error has a less significant affect on judge system selections than forced choice selection effects.⁷⁷7Figure 10 shows that these findings are robust to different choices of $\epsilon$ and $\eta$ . Lemma B.1 (Appendix B.2) pinpoints the mechanism driving these empirical findings. When rater error affects human ratings but not the ratings assigned by a judge system, its impact on the comparative ranking of judge systems is limited. This is in contrast to forced choice selection effects, which affect both human and judge system ratings. Nevertheless, rater error can yield rank inversions under specific conditions that fall outside the scope of our main experiment design (see Figure 11).

We find that rater error has a significant affect on prevalence estimates obtained from human ratings (Figure 12), particularly when error is correlated with the option used to categorize items. Taken together, our results suggest that more research is needed to characterize the effects of rater error at various points in the LLM-as-a-judge validation pipeline and in downstream performance estimates.

Finding 4: Underspecified rating tasks, forced choice selection effects, and finite sample error yield misleading evaluations of target systems. Figure 6 shows that factors in the human rating process affect evaluations of the target system, irrespective of whether a judge system is introduced. The left panel shows the consistency between thresholded decisions obtained from the population rating distribution versus a finite sample approximation. Allow/suppress decisions obtained from sampling just one rating per item disagree with the decision produced by thresholding the population rating distribution in 30% of items. The right panel shows that, for underspecified tasks, prevalence estimates obtained from forced choice ratings underestimate the prevalence of the property of interest (e.g., “toxicity”) as compared to prevalence estimates obtained from response set ratings.⁸⁸8Figure 6 isolates the role of forced choice selection effects by assuming no rater error. Figure 12 in Appendix D characterizes interactions between forced choice selection effects and rater error. These findings underscore that carefully structuring the rating task and accounting for forced choice selection effects is essential for reliable target system evaluations.

7 Case Study: Validating LLM-as-a-Judge Systems for Toxicity Detection

We examine how the choice of performance definition impacts judge system validation in practice by constructing a toxicity rating task. We use the Civil Comments dataset (Borkan etal., 2019) and assume that each comment is an output from $\mathcal{G}_{\text{target}}$ corresponding to an item in a rating task (see $\S$ E). We compare human ratings against ratings provided by five judge systems: Mistral Small, Mistral Large, Claude Sonnet 3.5, GPT 3.5 Turbo, and GPT 4o Mini.

The Civil Comments data contains forced choice ratings elicited from an underspecified rating task. It is possible to recover three options from provided data: $o_{1}=$ Very Toxic, $o_{2}=$ Toxic, and $o_{3}=$ Not Toxic OR Hard to Say. We model how these forced choice options map to response sets via a sensitivity parameter $\beta=\mathbb{P}(o_{2}\in\mathcal{S}\mid O^{h}=o_{3})$ . This parameter denotes the probability that a rater would endorse a response set that contains Toxic given that their forced choice response was Not Toxic or Hard to Say.

Finding 5: The ranking of judge systems depends on the choice of agreement metric. Table 8 illustrates that, even when there is a perfect mapping between forced choice and response set distributions ( $\beta=0$ ), the selected judge system depends on the choice of human–judge agreement metric. Claude Sonnet 3.5 is selected when targeting Hit Hate (h/h), Coverage (h/hrs), and KL-Divergence⁹⁹9where KL-Divergence is measured as the deviation of the judge system rating distribution from the human rating distribution. (s/s). In contrast, Mistral Small is selected when targeting MSE (srs/srs) or KL-Divergence¹⁰¹⁰10where KL-Divergence is measured as the deviation of the human rating distribution from the judge system rating distribution. (s/s). However, Mistral Small has 90% worse decision consistency than Sonnet 3.5 when $\beta=0$ .

This finding illustrates that selecting a judge system based on metrics developed for the determinate task setting can produce a model that performs very suboptimally on downstream content filtering tasks. Current practice is to measure asymmetric distributional metrics (e.g., KL-Divergence) as the deviation of the human rating distribution from the judge system rating distribution (Peterson etal., 2019; Collins etal., 2022; Uma etal., 2020; Fornaciari etal., 2021). While results presented in Appendix D indicate that judge systems selected with this directionality perform well in some settings, we observe the opposite directionality is preferable when $\beta\leq 3$ in Table 8. We recommend using a symmetric metric (i.e., JS-Divergence) while selecting judge systems in practice, as this approach performs reliably across settings.

Finding 6: The ranking of judge systems depends on forced choice selection effects. Table 8 also shows that the ranking of judge systems inverts when $\beta\geq.3$ . Using Hit Rate to select a judge system — as is common practice in the literature (Table 1) — yields Sonnet 3.5 across all settings of $\beta$ . While Sonnet 3.5 has the highest decision consistency at $\beta=0$ , the best model (defined against consistency) inverts to Mistral Small when $\beta\geq.3$ . Yet selecting Sonnet 3.5 when $\beta=.3$ yields a 34.5% reduction in decision consistency as compared to Mistral Small. This underscores the importance of fully specifying rating tasks or using response set ratings when the rating task is underspecified.

One way to assess whether this value of $\beta$ is plausible is to inspect the value recovered by judge systems: Sonnet 3.5 ( $0.32$ ), Mistral Small ( $0.44$ ), GPT 3.5 Turbo ( $0.04$ ), GPT 4o Mini ( $0.06$ ), and Mistral Large ( $0.11$ ). Notably, the two models that align most closely with human ratings – Mistral Small and Sonnet 3.5 – both yield $\beta\geq.3$ . This suggests the performance inversions shown in Table 8 are plausible.

Finding 7: Forced choice ratings underestimate the prevalence of toxicity in target system outputs. Figure 8 shows the prevalence of toxicity in $\mathcal{G}_{\text{target}}$ outputs as a function of the sensitivity parameter ( $\beta$ ) and cutoff ( $\tau$ ). The setting with $\tau=.5$ and $\beta=0$ corresponds to the status quo approach of using forced choice ratings to estimate prevalence. However, we find that small increases to the value of beta ( $\geq.1$ ) yield significant changes to the estimated prevalence of toxicity in $\mathcal{G}_{\text{target}}$ outputs. For example, at $\tau=.5$ , increasing $\beta$ from 0 to 0.05 doubles the prevalence of toxicity in target system outputs. This substantiates findings of our synthetic data experiments (Figure 6, right) and underscores the importance of carefully modeling the rating process to obtain reliable evaluations of the target system.

8 Conclusion

We introduce a framework for LLM-as-a-judge validation under rating task indeterminacy. Our framework provides a methodological foundation for more principled validation of judge systems designed to rate concepts such as “helpfulness”, “toxicity”, and “relevance” in target system outputs. We identify key factors in the LLM-as-a-judge validation pipeline: rating task design, judge performance measures, and rating elicitation and aggregation schemes, which can significantly affect downstream ratings performed by judge systems. We show that current practices for validating judge systems can yield misleading assessments of judge system performance and unreliable evaluations of target systems.

9 Impact Statement

In this paper, we introduce a framework for LLM-as-a-judge validation in the absence of gold labels. We demonstrate that key methodological considerations, including rating task design, measures of judge system performance, and rating elicitation and aggregation schemes, have substantive downstream impacts on judge system ratings. Although our arguments have potential societal consequences, especially if the practices we advocate for see adoption and thus change current GenAI evaluation practice, there are no consequences we feel the need to highlight that are specific to this work rather than applicable to any work aiming to improve upon current evaluation practices. The validation practices described in this paper are not an endorsement for the adoption of a judge system in any particular setting.

10 Acknowledgments

We thank members of the Sociotechnical Alignment Center at Microsoft Research New York City for their helpful comments on early versions of this work. We also thank attendees and reviewers of the Statistical Frontiers in LLMs and Foundation Models (SFLLM) and Evaluating Evaluations (EvalEval) workshops at NeurIPS, and attendees of the Fairness, Explainability, Accountability, and Transparency (FEAT) reading group at Carnegie Mellon University. This work was supported in part by an award from the UL Research Institutes throughthe Center for Advancing Safety of Machine Intelligence (CASMI) at Northwestern University.

References

Angelopoulos etal. (2023a)Angelopoulos, A.N., Bates, S., Fannjiang, C., Jordan, M.I., and Zrnic, T.Prediction-powered inference.Science, 382(6671):669–674, 2023a.
Angelopoulos etal. (2023b)Angelopoulos, A.N., Duchi, J.C., and Zrnic, T.Ppi++: Efficient prediction-powered inference.arXiv preprint arXiv:2311.01453, 2023b.
Bencke etal. (2024)Bencke, L., Paula, F.S., dos Santos, B.G., and Moreira, V.P.Can we trust llms as relevance judges?In Simpósio Brasileiro de Banco de Dados (SBBD), pp. 600–612. SBC, 2024.
Borkan etal. (2019)Borkan, D., Dixon, L., Sorensen, J., Thain, N., and Vasserman, L.Nuanced metrics for measuring unintended bias with real data for text classification.CoRR, abs/1903.04561, 2019.URL http://arxiv.org/abs/1903.04561.
Boyeau etal. (2024)Boyeau, P., Angelopoulos, A.N., Yosef, N., Malik, J., and Jordan, M.I.Autoeval done right: Using synthetic data for model evaluation.arXiv preprint arXiv:2403.07008, 2024.
Bubeck etal. (2023)Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y.T., Li, Y., Lundberg, S., etal.Sparks of artificial general intelligence: Early experiments with gpt-4.arXiv preprint arXiv:2303.12712, 2023.
Chaudhary etal. (2024)Chaudhary, M., Gupta, H., Bhat, S., and Varma, V.Towards understanding the robustness of llm-based evaluations under perturbations.arXiv preprint arXiv:2412.09269, 2024.
Chen etal. (2024)Chen, D., Chen, R., Zhang, S., Wang, Y., Liu, Y., Zhou, H., Zhang, Q., Wan, Y., Zhou, P., and Sun, L.Mllm-as-a-judge: Assessing multimodal llm-as-a-judge with vision-language benchmark.In Forty-first International Conference on Machine Learning, 2024.
Chiang etal. (2024)Chiang, W.-L., Zheng, L., Sheng, Y., Angelopoulos, A.N., Li, T., Li, D., Zhu, B., Zhang, H., Jordan, M., Gonzalez, J.E., etal.Chatbot arena: An open platform for evaluating llms by human preference.In Forty-first International Conference on Machine Learning, 2024.
Collins etal. (2022)Collins, K.M., Bhatt, U., and Weller, A.Eliciting and learning with soft labels from every annotator.In Proceedings of the AAAI conference on human computation and crowdsourcing, volume10, pp. 40–52, 2022.
Dettmers etal. (2024)Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L.Qlora: Efficient finetuning of quantized llms.Advances in Neural Information Processing Systems, 36, 2024.
Dong etal. (2024)Dong, Y.R., Hu, T., and Collier, N.Can llm be a personalized judge?arXiv preprint arXiv:2406.11657, 2024.
Dorner etal. (2024)Dorner, F.E., Nastl, V.Y., and Hardt, M.Limits to scalable evaluation at the frontier: Llm as judge won’t beat twice the data.arXiv preprint arXiv:2410.13341, 2024.
Dubois etal. (2024)Dubois, Y., Li, C.X., Taori, R., Zhang, T., Gulrajani, I., Ba, J., Guestrin, C., Liang, P.S., and Hashimoto, T.B.Alpacafarm: A simulation framework for methods that learn from human feedback.Advances in Neural Information Processing Systems, 36, 2024.
Es etal. (2023)Es, S., James, J., Espinosa-Anke, L., and Schockaert, S.Ragas: Automated evaluation of retrieval augmented generation.arXiv preprint arXiv:2309.15217, 2023.
Eyre & Madras (2024)Eyre, B. and Madras, D.Auto-evaluation with few labels through post-hoc regression.arXiv preprint arXiv:2411.12665, 2024.
Faisal etal. (2024)Faisal, F., Rahman, M.M., and Anastasopoulos, A.Dialectal toxicity detection: Evaluating llm-as-a-judge consistency across language varieties.arXiv preprint arXiv:2411.10954, 2024.
Fisch etal. (2020)Fisch, A., Schuster, T., Jaakkola, T., and Barzilay, R.Efficient conformal prediction via cascaded inference with expanded admission.arXiv preprint arXiv:2007.03114, 2020.
Fornaciari etal. (2021)Fornaciari, T., Uma, A., Paun, S., Plank, B., Hovy, D., Poesio, M., etal.Beyond black & white: Leveraging annotator disagreement via soft-label multi-task learning.In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, 2021.
Ganguli etal. (2022)Ganguli, D., Lovitt, L., Kernion, J., Askell, A., Bai, Y., Kadavath, S., Mann, B., Perez, E., Schiefer, N., Ndousse, K., etal.Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned.arXiv preprint arXiv:2209.07858, 2022.
Gordon etal. (2021)Gordon, M.L., Zhou, K., Patel, K., Hashimoto, T., and Bernstein, M.S.The disagreement deconvolution: Bringing machine learning performance metrics in line with reality.In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1–14, 2021.
Goyal etal. (2022)Goyal, N., Kivlichan, I.D., Rosen, R., and Vasserman, L.Is your toxicity my toxicity? exploring the impact of rater identity on toxicity annotation.Proceedings of the ACM on Human-Computer Interaction, 6(CSCW2):1–28, 2022.
Gu etal. (2024)Gu, J., Jiang, X., Shi, Z., Tan, H., Zhai, X., Xu, C., Li, W., Shen, Y., Ma, S., Liu, H., etal.A survey on llm-as-a-judge.arXiv preprint arXiv:2411.15594, 2024.
Jung etal. (2024)Jung, J., Brahman, F., and Choi, Y.Trust or escalate: Llm judges with provable guarantees for human agreement.arXiv preprint arXiv:2407.18370, 2024.
Kim etal. (2024)Kim, T.S., Lee, Y., Shin, J., Kim, Y.-H., and Kim, J.Evallm: Interactive evaluation of large language model prompts on user-defined criteria.In Proceedings of the CHI Conference on Human Factors in Computing Systems, pp. 1–21, 2024.
Klie etal. (2023)Klie, J.-C., Webber, B., and Gurevych, I.Annotation error detection: Analyzing the past and present for a more coherent future.Computational Linguistics, 49(1):157–198, 2023.
Li etal. (2024a)Li, X., Lipton, Z.C., and Leqi, L.Personalized language modeling from personalized human feedback.arXiv preprint arXiv:2402.05133, 2024a.
Li etal. (2024b)Li, Z., Wang, C., Ma, P., Wu, D., Wang, S., Gao, C., and Liu, Y.Split and merge: Aligning position biases in llm-based evaluators.In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 11084–11108, 2024b.
Lu & Zhong (2024)Lu, H. and Zhong, F.Can vision-language models replace human annotators: A case study with celeba dataset.arXiv preprint arXiv:2410.09416, 2024.
Mazeika etal. (2024)Mazeika, M., Phan, L., Yin, X., Zou, A., Wang, Z., Mu, N., Sakhaee, E., Li, N., Basart, S., Li, B., etal.Harmbench: A standardized evaluation framework for automated red teaming and robust refusal.arXiv preprint arXiv:2402.04249, 2024.
Mirzakhmedova etal. (2024)Mirzakhmedova, N., Gohsen, M., Chang, C.H., and Stein, B.Are large language models reliable argument quality annotators?In Conference on Advances in Robust Argumentation Machines, pp. 129–146. Springer, 2024.
Nie etal. (2020)Nie, Y., Zhou, X., and Bansal, M.What can we learn from collective human opinions on natural language inference data?arXiv preprint arXiv:2010.03532, 2020.
Pavlick & Kwiatkowski (2019)Pavlick, E. and Kwiatkowski, T.Inherent disagreements in human textual inferences.Transactions of the Association for Computational Linguistics, 7:677–694, 2019.
Peterson etal. (2019)Peterson, J.C., Battleday, R.M., Griffiths, T.L., and Russakovsky, O.Human uncertainty makes classification more robust.In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9617–9626, 2019.
Plank (2022)Plank, B.The’problem’of human label variation: On ground truth in data, modeling and evaluation.arXiv preprint arXiv:2211.02570, 2022.
Rahmani etal. (2024)Rahmani, H.A., Yilmaz, E., Craswell, N., Mitra, B., Thomas, P., Clarke, C.L., Aliannejadi, M., Siro, C., and Faggioli, G.Llmjudge: Llms for relevance judgments.arXiv preprint arXiv:2408.08896, 2024.
Shankar etal. (2024)Shankar, S., Zamfirescu-Pereira, J., Hartmann, B., Parameswaran, A.G., and Arawjo, I.Who validates the validators? aligning llm-assisted evaluation of llm outputs with human preferences.arXiv preprint arXiv:2404.12272, 2024.
Szymanski etal. (2024)Szymanski, A., Ziems, N., Eicher-Miller, H.A., Li, T. J.-J., Jiang, M., and Metoyer, R.A.Limitations of the llm-as-a-judge approach for evaluating llm outputs in expert knowledge tasks.arXiv preprint arXiv:2410.20266, 2024.
Takehi etal. (2024)Takehi, R., Voorhees, E.M., and Sakai, T.Llm-assisted relevance assessments: When should we ask llms for help?arXiv preprint arXiv:2411.06877, 2024.
Thakur etal. (2024)Thakur, A.S., Choudhary, K., Ramayapally, V.S., Vaidyanathan, S., and Hupkes, D.Judging the judges: Evaluating alignment and vulnerabilities in llms-as-judges.arXiv preprint arXiv:2406.12624, 2024.
Tjuatja etal. (2024)Tjuatja, L., Chen, V., Wu, T., Talwalkwar, A., and Neubig, G.Do llms exhibit human-like response biases? a case study in survey design.Transactions of the Association for Computational Linguistics, 12:1011–1026, 2024.
Uma etal. (2020)Uma, A., Fornaciari, T., Hovy, D., Paun, S., Plank, B., and Poesio, M.A case for soft loss functions.In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, volume8, pp. 173–177, 2020.
Zheng etal. (2023)Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., etal.Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in Neural Information Processing Systems, 36:46595–46623, 2023.

Appendix A Appendix.

This work contains the following appendices:

•
Appendix A Table 1 provides a summary of aggregation functions adopted in prior work on (1) validating LLM-as-a-judge systems, and (2) evaluation under task indeterminacy.
•
Appendix B provides additional theoretical analysis, including (i) proofs ( $\S$ B.1); and (ii) additional theoretical results characterizing rank consistency of MSE under error-free versus error-corrupted human ratings ( $\S$ B.2).
•
Appendix C provides examples of monotonicity violations among pairs of human–judge agreement metrics (see $\S$ 5).
•
Appendix D provides further synthetic experiment setup details and empirical results.
•
Appendix E provides further setup details and empirical results for the toxicity detection case study.

Validating LLM-as-a-Judge Systems in the Absence of Gold Labels (7)

Metric Type	Metric	Judge Aggregation	Human Aggregation	Relevant Work
	Hit Rate ( $\uparrow$ )	$a^{j}_{\text{hard}}$	$a^{h}_{\text{hard}}$	Lu & Zhong (2024); Jung etal. (2024); Dong etal. (2024); Es etal. (2023); Dubois etal. (2024); Bubeck etal. (2023); Zheng etal. (2023); Faisal etal. (2024); Gu etal. (2024); Thakur etal. (2024); Li etal. (2024b); Chen etal. (2024); Chiang etal. (2024); Dorner etal. (2024)
Categorical	Krippendorff’s $\alpha$ ( $\uparrow$ )	$a^{j}_{\text{hard}}$	$a^{h}_{\text{hard}}$	Mirzakhmedova etal. (2024); Chaudhary etal. (2024)
	Fleiss’ $\kappa$ ( $\uparrow$ )	$a^{j}_{\text{hard}}$	$a^{h}_{\text{hard}}$	Kim etal. (2024); Dettmers etal. (2024); Bencke etal. (2024)
	Cohen’s $\kappa$ ( $\uparrow$ )	$a^{j}_{\text{hard}}$	$a^{h}_{\text{hard}}$	Rahmani etal. (2024); Bencke etal. (2024)
	Scott’s $\pi$ ( $\uparrow$ )	$a^{j}_{\text{hard}}$	$a^{h}_{\text{hard}}$	Thakur etal. (2024)
	KL Divergence ( $\downarrow$ )	$a^{j}_{\text{soft}}$	$a^{h}_{\text{soft}}$	Nie etal. (2020); Fornaciari etal. (2021)
Distributional	Cross-Entropy ( $\downarrow$ )	$a^{j}_{\text{soft}}$	$a^{h}_{\text{soft}}$	Peterson etal. (2019); Pavlick & Kwiatkowski (2019); Collins etal. (2022)
	JS Divergence ( $\downarrow$ )	$a^{j}_{\text{soft}}$	$a^{h}_{\text{soft}}$	Nie etal. (2020); Fornaciari etal. (2021)
CategoricalMulti-label	Coverage ( $\uparrow$ )	$a^{j}_{\text{hard}},a^{j}_{\text{hrs}}$	$a^{h}_{\text{hard}},a^{h}_{\text{hrs}}$	Fisch etal. (2020); Takehi etal. (2024)
	PredictiveEfficiency ( $\downarrow$ )	$a^{j}_{\text{hard}},a^{j}_{\text{hrs}}$	N/A	Fisch etal. (2020)
	Recall ( $\uparrow$ )	$a^{j}_{\text{hard}},a^{j}_{\text{hrs}}$	$a^{h}_{\text{hard}},a^{h}_{\text{hrs}}$
	Precision ( $\uparrow$ )	$a^{j}_{\text{hard}},a^{j}_{\text{hrs}}$	$a^{h}_{\text{hard}},a^{h}_{\text{hrs}}$
ContinuousMulti-label	Binary Cross Entropy ( $\downarrow$ )	$a^{j}_{*}$	$a^{h}_{\text{hard}},a^{h}_{\text{hrs}}$
ContinuousMulti-label	Mean SquaredError ( $\downarrow$ )	$a^{j}_{*}$	$a^{h}_{*}$

Appendix B Additional Theoretical Analysis

B.1 Proofs

B.1.1 Theorem 3.3

Proof.

The forward model $\boldsymbol{O}_{i}=\mathbf{F}_{i}(\mathbf{E}_{i}\boldsymbol{\theta}^{*}_{i})$ follows by the following factorization¹¹¹¹11We omit $i$ from all subscripts for brevity. :

$\displaystyle\mathbb{P}(o_{k}\mid t)$	$\displaystyle=\sum_{v,{v^{}},r}\mathbb{P}(o_{k}\mid s_{v},s_{v^{}},r,t)\cdot%\mathbb{P}(s_{v},s_{v^{*}},r\mid t)$
	$\displaystyle=\sum_{v,{v^{}},r}\mathbb{P}(o_{k}\mid s_{v},t)\cdot\mathbb{P}(s%_{v},s_{v^{}},r\mid t)$	(4)
	$\displaystyle=\sum_{v,{v^{}},r}\mathbb{P}(o_{k}\mid s_{v},t)\cdot\mathbb{P}(s%_{v}\mid s_{v^{}},r,t)\cdot\mathbb{P}(s_{v^{*}},r\mid t)$
	$\displaystyle=\sum_{v,{v^{}},r}\mathbb{P}(o_{k}\mid s_{v},t)\cdot\mathbb{P}(s%_{v}\mid s_{v^{}},t)\cdot\mathbb{P}(s_{v^{*}},r\mid t)$	(5)
	$\displaystyle=\sum_{v}\mathbb{P}(o_{k}\mid s_{v},t)\cdot\sum_{v^{}}\mathbb{P}%(s_{v}\mid s_{v^{}},t)\cdot\sum_{r}\mathbb{P}(s_{v^{*}},r\mid t)$
	$\displaystyle=\sum_{v}\boldsymbol{\mathbf{F}}_{k,v}\cdot\sum_{v^{}}%\boldsymbol{\mathbf{E}}_{v,v^{}}\cdot\boldsymbol{\theta}^{}_{v^{}}.$

Above, (4) holds by forced choice independence and (5) holds by error independence. The reverse model $\boldsymbol{\theta}^{*}_{i}=\mathbf{E}_{i}^{\prime}(\mathbf{F}_{i}^{\prime}%\mathbf{O}_{i})$ follows by the following factorization:

$\displaystyle\mathbb{P}(s_{v^{*}}\mid t)$	$\displaystyle=\sum_{r,v,k}\mathbb{P}(s_{v^{*}}\mid r,s_{v},o_{k},t)\cdot%\mathbb{P}(s_{v}\mid r,o_{k},t)\cdot\mathbb{P}(o_{k},r\mid t)$
	$\displaystyle=\sum_{r,v,k}\mathbb{P}(s_{v^{*}}\mid r,s_{v},t)\cdot\mathbb{P}(s%_{v}\mid r,o_{k},t)\cdot\mathbb{P}(o_{k},r\mid t)$	(6)
	$\displaystyle=\sum_{r,v,k}\left(\frac{\mathbb{P}(s_{v}\mid r,s_{v^{}},t)\cdot%\mathbb{P}(r,s_{v^{}}\mid t)}{\mathbb{P}(r,s_{v}\mid t)}\right)\cdot\left(%\frac{\mathbb{P}(o_{k}\mid r,s_{v},t)\cdot\mathbb{P}(r,s_{v}\mid t)}{\mathbb{P%}(r,o_{k}\mid t)}\right)\cdot\mathbb{P}(o_{k},r\mid t)$
	$\displaystyle=\sum_{r,v,k}\mathbb{P}(s_{v}\mid s_{v^{}},t)\cdot\mathbb{P}(r,s%_{v^{}}\mid t)\cdot\mathbb{P}(o_{k}\mid s_{v},t)$	(7)
	$\displaystyle=\sum_{v,k}\left(\frac{\mathbb{P}(s_{v}^{}\mid s_{v},t)\cdot%\mathbb{P}(s_{v}\mid t)}{\mathbb{P}(s_{v^{}}\mid t)}\right)\cdot\left(\frac{%\mathbb{P}(s_{v}\mid o_{k},t)\cdot\mathbb{P}(o_{k}\mid t)}{\mathbb{P}(s_{v}%\mid t)}\right)\cdot\sum_{r}\mathbb{P}(r,s_{v^{*}}\mid t)$
	$\displaystyle=\sum_{v,k}\mathbb{P}(s_{v}^{*}\mid s_{v},t)\cdot\mathbb{P}(s_{v}%\mid o_{k},t)\cdot\mathbb{P}(o_{k}\mid t)$
	$\displaystyle=\sum_{v}\mathbf{E}_{v^{*},v}^{\prime}\cdot\sum_{k}\mathbf{F}_{v,%k}^{\prime}\cdot\mathbf{O}_{k}$

where (6) holds by forced choice independence and (7) holds by forced choice and error independence.∎

B.1.2 Theorem 3.4

Proof.

We remove dependence on $i$ from all terms for brevity. To begin, note that $\boldsymbol{\theta}^{*}$ is identifiable from $\mathbf{O}$ if and only if $\boldsymbol{\theta}^{*}=\mathbf{E}^{\prime}(\mathbf{F}_{i}^{\prime}\mathbf{O})%=\mathbf{F}\boldsymbol{\theta}^{*}$ is fully determined (where the system simplifies by taking $\mathbf{E}$ as the identity matrix). The system system $\mathbf{F}\boldsymbol{\theta}^{*}$ must be consistent because Theorem 3.3 establishes a solution. A consistent system with $n_{o}=|\mathcal{O}|$ equations and $n_{s}=|\mathcal{Q}|$ unknowns is fully determined if and only if rank( $\boldsymbol{\mathbf{F}}$ ) $=n_{s}$ .

We will first show that $\mathcal{Q}=\{\{o_{k}\}:o_{k}\in\mathcal{O}\}$ implies that rank( $\boldsymbol{\mathbf{F}}$ ) $=n_{s}$ . To begin, note that (1) $\sum_{k}\mathbf{F}_{k,v}=1,\;\forall v\in\{1,...,n_{s}\}$ because each column in $\mathbf{F}$ represents a valid probability distribution; and (2) $\mathbf{F}_{k,v}=0,\;\;\forall a\neq k,\;\forall v\in\{1,...,n_{s}\}\;$ because $\mathbf{\Lambda}_{k,v}=0\implies\boldsymbol{F}_{k,v}=0$ . This implies that

1=\sum_{a}\mathbf{F}_{a,v}=\sum_{a\neq k}\mathbf{F}_{a,v}+\mathbf{F}_{k,v}=%\mathbf{F}_{k,v},\quad\forall v\in\{1,...,n_{s}\}.

Thus, each singleton set $\{o_{k}\}\in\mathcal{S}$ maps to a standard basis vector $\mathbf{e}_{k}\in\mathbb{R}^{n_{o}}$ . Further, because $n_{s}=n_{o}$ by definition of $\mathcal{Q}$ and $\mathcal{O}\subseteq\mathcal{Q}$ , each option must appear in exactly one set, giving us exactly $n_{s}=n_{o}$ distinct basis vectors. The rank of a matrix is equal to the number of linear independent column vectors. Because each of the $k$ standard basis vectors must be linearly independent, it follows that rank( $\mathbf{F}$ ) $=n_{o}=n_{s}$ .

We will show the reverse implication that rank( $\mathbf{F}$ ) = $n_{s}$ $\implies$ $\mathcal{Q}=\{\{o_{k}\}:o_{k}\in\mathcal{O}\}$ by contradiction. Suppose there exists a set $\mathcal{S}_{v}\in\mathcal{Q}$ containing more than one option, i.e., $|\mathcal{S}|>1$ . Let $\mathbf{v}$ denote the column of $\mathbf{F}$ corresponding to $\mathcal{S}_{v}$ . Since $\mathcal{O}\subseteq\mathcal{Q}$ , for each option $o_{k}\in\mathcal{S}$ , there exists a column in $\mathbf{F}$ that is the standard basis vector $\mathbf{e}_{k}$ , as shown above. Therefore, $\mathbf{v}$ can be written as a linear combination of these basis vectors: $\mathbf{v}=\sum_{k}\alpha_{k}\mathbf{e}_{k}$ where $\alpha_{k}=[0,1]$ . This shows that column $\mathbf{v}$ is linearly dependent with the columns corresponding to singleton sets ${o_{k}}$ for $o_{k}\in\mathcal{S}$ . This implies $\mathbf{F}$ cannot have $n_{s}$ linearly independent columns, contradicting rank( $\mathbf{F}$ ) = $n_{s}$ .

∎

B.1.3 Theorem 5.3.

Proof.

Proof by contradiction. Let $\mathbf{p}$ and $\mathbf{p}^{\prime}$ denote pairs of performance definitions with increasing cardinality (i.e., higher values being better). Let $Y^{z}=a^{j}(\mathbb{P}_{T}^{j,z}),\;Y^{w}=a^{j}(\mathbb{P}_{T}^{j,w}),\;Y^{h}=%a^{h}(\mathbb{P}_{T}^{h})$ denote random functions of $T$ corresponding to definition $\mathbf{p}$ . Let $Y^{z}_{*}=a_{j}^{\prime}(\mathbb{P}_{T}^{j,z}),\;Y^{w}_{*}=a_{j}^{\prime}(%\mathbb{P}_{T}^{j,w}),\;Y^{h}_{*}=a_{h}^{\prime}(\mathbb{P}_{T}^{h})$ correspond to definition $\mathbf{p}^{\prime}$ . Since $\mathbf{p}$ is not a monotone transformation of $\mathbf{p}^{\prime}$ , by definition there must exist distributions $\{(\mathbb{P}_{i}^{j,z},\mathbb{P}_{i}^{h}),(\mathbb{P}_{i}^{j,w},\mathbb{P}_{%i}^{h})\}\in\Delta\times\Delta$ corresponding to realizations of these random variables satisfying

m(Y^{z},Y^{h})<m(Y^{w},Y^{h}),\quad\quad m^{\prime}(Y^{z}_{*},Y^{h}_{*})>m^{%\prime}(Y^{w}_{*},Y^{h}_{*}).

Now suppose that $\mathbb{P}^{*}$ places all marginal probability mass over $T$ on the $i$ ’th item – i.e., $\mathbb{P}^{*}(T=t_{i})=1$ . Then:

	$\displaystyle\delta_{\mathbf{p}}(z,w)$	$\displaystyle=\mathbb{E}_{\mathbb{P}^{*}}[m(Y^{z},Y^{h})-m(Y^{w},Y^{h})]<0$
	$\displaystyle\delta_{\mathbf{p}^{\prime}}(z,w)$	$\displaystyle=\mathbb{E}_{\mathbb{P}^{}}[m^{\prime}(Y^{z}_{},Y^{h}_{})-m^{%\prime}(Y^{w}_{},Y^{h}_{*})]>0$

Thus, rank consistency is violated because there exists a distribution $\mathbb{P}^{*}$ for which $\mathcal{G}_{judge}^{z}\succ_{\mathbf{p}^{\prime}}\mathcal{G}_{judge}^{w}$ but $\mathcal{G}_{judge}^{w}\succ_{\mathbf{p}}\mathcal{G}_{judge}^{z}$ . This provides a contradition, proving the result.

∎

B.2 Rank Consistency Under Rater Error

Lemma B.1 (Rank Consistency of MSE (srs/srs) Under Rater Error ).

Let $\boldsymbol{\theta}^{*}$ and $\boldsymbol{\theta}=\mathbf{E}\boldsymbol{\theta}^{*}$ denote the stable and observed response set distributions for human raters.¹²¹²12We omit subscript $i$ from all terms for brevity. We also omit superscript $h$ from human response set distribution, error, and multi-label vectors where the context is clear. Let $\boldsymbol{\theta}^{j,z}$ and $\boldsymbol{\theta}^{j,w}$ denote observed response set distributions for judge systems $\mathcal{G}^{z}_{judge}$ and $\mathcal{G}^{w}_{judge}$ where both judge systems have a rater error matrix $\mathbf{E}^{j,z}$ and $\mathbf{E}^{j,w}$ that is the identity. Let $\mathbf{\Lambda}$ be the binary matrix mapping response sets to options and define $\delta^{*}=\text{MSE}(\boldsymbol{\Omega}^{*},\boldsymbol{\Omega}^{j,z})-\text%{MSE}(\boldsymbol{\Omega}^{*},\boldsymbol{\Omega}^{j,w})$ as the difference in MSE under error-free conditions. The ranking of judge systems using MSE with soft response set aggregation is preserved under human rating error if and only if:

\text{sign}((\boldsymbol{\theta}^{*}-\boldsymbol{\theta})^{T}\mathbf{\Lambda}^%{T}\mathbf{\Lambda}(\boldsymbol{\theta}^{j,w}-\boldsymbol{\theta}^{j,z}))=%\text{sign}(\delta^{*}).

(8)

This lemma provides conditions under which measuring the performance of a judge system against error-corrupted versus error-free human ratings yields a consistent ranking of judge systems (when measured against MSE(srs/srs)). The condition essentially requires that the direction of the error-induced shift in human ratings ( $\boldsymbol{\theta}^{*}-\boldsymbol{\theta}$ ) matches the direction of the stable response set shift across judge systems ( $\boldsymbol{\theta}^{j,w}-\boldsymbol{\theta}^{j,z}$ ) when projected to the multi-label space. If human rating error and judge system performance differences shift the response set distribution in the same direction, the ranking of judge systems will be consistent for error-free and error-corrupted ratings. Conversely, rankings can invert under an inverse relationship.

E.q. (8) is satisfied in our experimental setup ( $\S$ 6) because the ensemble of judge system rating distributions is generated by adding random perturbations (i.e., uncorrelated with rater error) to the human stable response set vector. Thus we see little change in the reliability of MSE (srs/srs) across settings with no rater error (Figure 6, center) and rater error (Figure 6, right).

Proof of Lemma B.1.

For brevity, let $\mathbf{M}=\mathbf{\Lambda}^{T}\mathbf{\Lambda}$ . The difference in judge system MSE measured against the multi-label human rating vector $\mathbf{\Omega}^{*}=\mathbf{\Lambda}\boldsymbol{\theta}^{*}$ derived from the stable response set distribution is given by:

	$\displaystyle\delta^{*}$	$\displaystyle=\text{MSE}(\mathbf{\Omega}^{},\;\mathbf{\Omega}^{j,z})-\text{%MSE}(\mathbf{\Omega}^{},\;\mathbf{\Omega}^{j,w})$
		$\displaystyle=\|\|\mathbf{\Lambda}\boldsymbol{\theta}^{}-\mathbf{\Lambda}%\boldsymbol{\theta}^{j,z}\|\|^{2}_{2}-\|\|\mathbf{\Lambda}\boldsymbol{\theta}^{}-%\mathbf{\Lambda}\boldsymbol{\theta}^{j,w}\|\|^{2}_{2}$
		$\displaystyle=(\boldsymbol{\theta}^{}-\boldsymbol{\theta}^{j,z})^{T}\mathbf{M%}(\boldsymbol{\theta}^{}-\boldsymbol{\theta}^{j,z})-(\boldsymbol{\theta}^{}-%\boldsymbol{\theta}^{j,w})^{T}\mathbf{M}(\boldsymbol{\theta}^{}-\boldsymbol{%\theta}^{j,w})$
		$\displaystyle=(\boldsymbol{\theta}^{})^{T}\mathbf{M}\boldsymbol{\theta}^{}-(%\boldsymbol{\theta}^{})^{T}\mathbf{M}\boldsymbol{\theta}^{j,z}-(\boldsymbol{%\theta}^{j,z})^{T}\mathbf{M}\boldsymbol{\theta}^{}+(\boldsymbol{\theta}^{j,z}%)^{T}\mathbf{M}\boldsymbol{\theta}^{j,z}$
		$\displaystyle\quad\quad-(\boldsymbol{\theta}^{})^{T}\mathbf{M}\boldsymbol{%\theta}^{}+(\boldsymbol{\theta}^{})^{T}\mathbf{M}\boldsymbol{\theta}^{j,w}+(%\boldsymbol{\theta}^{j,w})^{T}\mathbf{M}\boldsymbol{\theta}^{}-(\boldsymbol{%\theta}^{j,w})^{T}\mathbf{M}\boldsymbol{\theta}^{j,w}$
		$\displaystyle=(\boldsymbol{\theta}^{j,z})^{T}\mathbf{M}\boldsymbol{\theta}^{j,%z}-(\boldsymbol{\theta}^{j,w})^{T}\mathbf{M}\boldsymbol{\theta}^{j,w}+2(%\boldsymbol{\theta}^{*})^{T}\mathbf{M}(\boldsymbol{\theta}^{j,w}-\boldsymbol{%\theta}^{j,z})$

Let $\mathbf{\Omega}=\mathbf{\Lambda}(\mathbf{E}\boldsymbol{\theta})$ denote the multi-label vector recovered from the observed response set distribution. Applying the same derivation as above to the error-corrupted MSE metric yields:

	$\displaystyle\delta$	$\displaystyle=\text{MSE}(\mathbf{\Omega},\;\mathbf{\Omega}^{j,z})-\text{MSE}(%\mathbf{\Omega},\;\mathbf{\Omega}^{j,w})$
		$\displaystyle=(\boldsymbol{\theta}^{j,z})^{T}\mathbf{M}\boldsymbol{\theta}^{j,%z}-(\boldsymbol{\theta}^{j,w})^{T}\mathbf{M}\boldsymbol{\theta}^{j,w}+2(%\mathbf{E}\boldsymbol{\theta}^{*})^{T}\mathbf{M}(\boldsymbol{\theta}^{j,w}-%\boldsymbol{\theta}^{j,z})$

Observe that the first two terms appear in both expansions. Thus we need to focus on the third term while showing the conditions required for rank consistency — i.e., sign( $\delta$ ) = sign( $\delta^{*}$ ).

•

Case 1: $\delta^{*}<0$ ( $\mathcal{G}^{z}_{judge}$ is better than $\mathcal{G}^{w}_{judge}$ under no rater error.) For both inequalities to hold, we need:

	$\displaystyle 2(\mathbf{E}\boldsymbol{\theta}^{*})^{T}\mathbf{M}(\boldsymbol{%\theta}^{j,w}-\boldsymbol{\theta}^{j,z})$	$\displaystyle\leq 2(\boldsymbol{\theta}^{*})^{T}\mathbf{M}(\boldsymbol{\theta}%^{j,w}-\boldsymbol{\theta}^{j,z})$
	$\displaystyle(\mathbf{E}\boldsymbol{\theta}^{*})^{T}\mathbf{M}(\boldsymbol{%\theta}^{j,w}-\boldsymbol{\theta}^{j,z})$	$\displaystyle\leq(\boldsymbol{\theta}^{*})^{T}\mathbf{M}(\boldsymbol{\theta}^{%j,w}-\boldsymbol{\theta}^{j,z})$
	$\displaystyle(\boldsymbol{\theta}^{})^{T}\mathbf{M}(\boldsymbol{\theta}^{j,w}%-\boldsymbol{\theta}^{j,z})-(\mathbf{E}\boldsymbol{\theta}^{})^{T}\mathbf{M}(%\boldsymbol{\theta}^{j,w}-\boldsymbol{\theta}^{j,z})$	$\displaystyle\geq 0$
	$\displaystyle((\boldsymbol{\theta}^{})^{T}-(\mathbf{E}\boldsymbol{\theta}^{}%)^{T})\mathbf{M}(\boldsymbol{\theta}^{j,w}-\boldsymbol{\theta}^{j,z})$	$\displaystyle\geq 0$
	$\displaystyle(\boldsymbol{\theta}^{}-\mathbf{E}\boldsymbol{\theta}^{})^{T}%\mathbf{M}(\boldsymbol{\theta}^{j,w}-\boldsymbol{\theta}^{j,z})$	$\displaystyle\geq 0$

•
Case 2: $\delta^{*}>0$ ( $\mathcal{G}^{w}_{judge}$ is better than $\mathcal{G}^{z}_{judge}$ under no rater error.).For rank consistency, we need $\delta>0$ as well. Following similar steps, we get:

	$\displaystyle 2(\mathbf{E}\boldsymbol{\theta}^{*})^{T}\mathbf{M}(\boldsymbol{%\theta}^{j,w}-\boldsymbol{\theta}^{j,z})$	$\displaystyle\geq 2(\boldsymbol{\theta}^{*})^{T}\mathbf{M}(\boldsymbol{\theta}%^{j,w}-\boldsymbol{\theta}^{j,z})$
	$\displaystyle-2(\mathbf{E}\boldsymbol{\theta}^{*})^{T}\mathbf{M}(\boldsymbol{%\theta}^{j,w}-\boldsymbol{\theta}^{j,z})$	$\displaystyle\leq-2(\boldsymbol{\theta}^{*})^{T}\mathbf{M}(\boldsymbol{\theta}%^{j,w}-\boldsymbol{\theta}^{j,z})$
	$\displaystyle 2(\mathbf{E}\boldsymbol{\theta}^{*})^{T}\mathbf{M}(\boldsymbol{%\theta}^{j,z}-\boldsymbol{\theta}^{j,w})$	$\displaystyle\leq 2(\boldsymbol{\theta}^{*})^{T}\mathbf{M}(\boldsymbol{\theta}%^{j,z}-\boldsymbol{\theta}^{j,w})$
	$\displaystyle(\boldsymbol{\theta}^{}-\mathbf{E}\boldsymbol{\theta}^{})^{T}%\mathbf{M}(\boldsymbol{\theta}^{j,z}-\boldsymbol{\theta}^{j,w})$	$\displaystyle\geq 0$

∎

Appendix C Examples of Monotonicity Violations Among Pairs of Performance Metrics

{mdframed}

[backgroundcolor=gray!10,linewidth=1pt,linecolor=gray!50,innertopmargin=8pt,innerbottommargin=15pt,innerleftmargin=15pt,innerrightmargin=15pt]Example 1: Hit Rate (Forced Choice) and KL-Divergence (Forced Choice).

Let $\mathbf{p}=(a^{h}_{\text{hard}},a^{j}_{\text{hard}},\text{Hit Rate})$ and let $\mathbf{p}^{\prime}=(a^{h}_{\text{soft}},a^{j}_{\text{soft}},\text{KL-%Divergence})$ . Consider forced choice distributions recovered from human ratings and the judge systems $z$ and $w$ , respectively: $\mathbf{O}^{h}=a^{h}_{\text{soft}}(\mathbb{P}^{h})$ , $\mathbf{O}^{j,z}=a^{j}_{\text{soft}}(\mathbb{P}^{j,z})$ , $\mathbf{O}^{j,w}=a^{h}_{\text{soft}}(\mathbb{P}^{j,w})$ , where we omit $i$ from all terms for brevity.

Suppose these distributions are defined over three options $\mathcal{O}=\{o_{1},o_{2},o_{3}\}$ and for an item $i$ :

\displaystyle\text{Human:}\quad\mathbf{O}^{h}=(0.6,0.3,0.1),\quad\text{Judge $%z$:}\quad\mathbf{O}^{j,z}=(0.8,0.1,0.1),\quad\text{Judge $w$:}\quad\mathbf{O}^%{j,w}=(0.5,0.4,0.1)

Under $\mathbf{p}$ , we have:

\text{HR}(\mathbf{O}^{j,z},\mathbf{O}^{h})=1.0>\text{HR}(\mathbf{O}^{j,w},%\mathbf{O}^{h})=0.0\implies\mathcal{G}_{judge}^{z}\succ_{\mathbf{p}}\mathcal{G%}_{judge}^{w}.

But under $\mathbf{p}^{\prime}$ , we have:

\text{KL}(\mathbf{O}^{h}\|\mathbf{O}^{j,z})\approx 0.15>\text{KL}(\mathbf{O}^{%h}\|\mathbf{O}^{j,w})\approx 0.02\implies\mathcal{G}_{judge}^{w}\succ_{\mathbf%{p}}\mathcal{G}_{judge}^{z}.

Thus, we have identified a pair of conditional rating distributions $\mathbb{P}^{j,z}$ , $\mathbb{P}^{j,w}$ and a corresponding human rating distribution $\mathbb{P}^{h}$ where $\mathbf{p}^{\prime}$ is not a monotone transformation of $\mathbf{p}$ , so rank consistency between $\mathbf{p}$ and $\mathbf{p}^{\prime}$ cannot hold.

{mdframed}

[backgroundcolor=gray!10,linewidth=1pt,linecolor=gray!50,innertopmargin=10pt,innerbottommargin=10pt,innerleftmargin=10pt,innerrightmargin=10pt]Example 2: KL-Divergence (Forced Choice) and MSE (Multi-Label). Let $\mathbf{p}=(a^{h}_{\text{soft}},a^{j}_{\text{soft}},\text{KL-divergence})$ and let $\mathbf{p}^{\prime}=(a^{h}_{\text{srs}},a^{j}_{\text{srs}},\text{MSE})$ . Let $\mathcal{O}=\{o_{1},o_{2}\}$ and $\mathcal{Q}=\{\{o_{1}\},\{o_{2}\},\{o_{1},o_{2}\}\}$ . Suppose that humans have no rater error (i.e., $\mathbf{E}^{h}$ and $\mathbf{E}^{j}$ are both the identity). Let $\mathbb{P}^{h}_{i}$ satisfy the decomposition:

\mathbf{O}^{h}=(.4,.6)^{\top},\;\;\mathbf{F}^{h}=\begin{bmatrix}1&0&0\\0&1&1\end{bmatrix},\;\;\boldsymbol{\theta}^{*,h}=(0.4,0.5,0.1)^{\top},\;\;%\mathbf{\Omega}^{h}=(.5,.6)^{\top}.

Let $\mathbb{P}^{j,z}_{i}$ denote the conditional rating distribution of the judge system satisfying:

\mathbf{O}^{j,z}=(.4,.6)^{\top},\;\;\mathbf{F}^{j,z}=\begin{bmatrix}1&0&1\\0&1&0\end{bmatrix},\;\;\boldsymbol{\theta}^{*,j,z}=(0.0,0.6,0.4)^{\top},\;\;%\mathbf{\Omega}^{j,z}=(.4,.6)^{\top}.

Let $\mathbb{P}^{j,w}_{i}$ denote the conditional rating distribution of the judge system satisfying:

\mathbf{O}^{j,w}=(.5,.5)^{\top},\;\;\mathbf{F}^{*,j,w}=\begin{bmatrix}1&0&1\\0&1&0\end{bmatrix},\;\;\boldsymbol{\theta}^{*,j,w}=(0.4,0.5,0.1)^{\top},\;\;%\mathbf{\Omega}^{j,w}=(.5,.6)^{\top}.

Under $\mathbf{p}$ , we have:

\text{KL}(\mathbf{O}^{h}|\mathbf{O}^{j,z})=0<\text{KL}(\mathbf{O}^{h}|\mathbf{%O}^{j,w})\approx 0.02\implies\mathcal{G}_{judge}^{z}\succ_{\mathbf{p}}\mathcal%{G}_{judge}^{w}.

But under $\mathbf{p}^{\prime}$

\text{MSE}(\mathbf{\Omega}^{j,z},\mathbf{\Omega}^{h})=0.01>\text{MSE}(\mathbf{%\Omega}^{j,w},\mathbf{\Omega}^{h})=0.00\implies\mathcal{G}_{judge}^{w}\succ_{%\mathbf{p}^{\prime}}\mathcal{G}_{judge}^{z}.

yielding a violation of monotonicity. Thus, we have identified a pair of conditional rating distributions $\mathbb{P}^{j,z}_{i}$ , $\mathbb{P}^{j,w}_{i}$ and a corresponding human rating distribution $\mathbb{P}^{h}_{i}$ where $\mathbf{p}^{\prime}$ is not a monotone transformation of $\mathbf{p}$ , so rank consistency between $\mathbf{p}$ and $\mathbf{p}^{\prime}$ cannot hold.

Appendix D Additional Synthetic Experiment Setup Details and Results

Validating LLM-as-a-Judge Systems in the Absence of Gold Labels (8)

Validating LLM-as-a-Judge Systems in the Absence of Gold Labels (9)

Validating LLM-as-a-Judge Systems in the Absence of Gold Labels (10)

D.1 Setup Details

We run all experiments with $50$ judge systems. We use 100 items in all experiments and select option and response set configurations satisfying: $2\leq|\mathcal{O}|\leq 10$ , $2\leq|\mathcal{Q}|\leq 30$ , $|\mathcal{O}|\leq|\mathcal{Q}|$ . We let $\sigma_{\text{min}}=0.02$ and $\sigma_{\text{max}}=.4$ when sampling the ensemble of judge systems. While performing synthetic experiments, we estimate $\hat{\mathbf{O}}_{i}$ where $\hat{\mathbf{O}}_{i,k}=\hat{\mathbb{P}}(O=o_{k})$ for both human and judge rating distributions via the maximum likelihood estimator. We then compute $\hat{\boldsymbol{\theta}}^{*}_{i}=\mathbf{E}_{i}^{\prime}(\mathbf{F}_{i}^{%\prime}\hat{\mathbf{O}}_{i})$ This also allows us to compute $\hat{\mathbf{\Omega}}_{i}=\hat{\boldsymbol{\theta}}^{*}_{i}(\mathbf{\Lambda})$ . We apply aggregation functions to the estimated forced choice distribution and estimated multi-label vector to obtain estimated performance metrics. For example, while applying soft aggregation, we obtain $\hat{Y}^{h}=a^{h}_{srs}(\hat{\mathbf{\Omega}}^{h}_{i})$ and $\hat{Y}^{j}=a^{j}_{srs}(\hat{\mathbf{\Omega}}^{j}_{i})$ after estimating $(\hat{\mathbf{\Omega}}^{h}_{i},\hat{\mathbf{\Omega}}^{j}_{i})$ via the procedure outlined above. The expected performance is then given by: $\hat{M}_{mse}(\hat{Y}^{h},\hat{Y}^{j})=E_{(T,\hat{Y}^{h},\hat{Y}^{j})}[|\hat{Y%}^{h}-\hat{Y}^{j}|_{2}^{2}].$ . We use the same finite sample estimation approach to recover other metrics. This estimator is consistent so long as $\hat{\mathbf{\Omega}}\xrightarrow{p}\mathbf{\Omega}$ as $n\rightarrow\infty$ , where $\xrightarrow{p}$ denotes convergence in probability. This follows by a standard maximum likelihood convergence argument.

Validating LLM-as-a-Judge Systems in the Absence of Gold Labels (11)

Validating LLM-as-a-Judge Systems in the Absence of Gold Labels (12)

Validating LLM-as-a-Judge Systems in the Absence of Gold Labels (13)

Validating LLM-as-a-Judge Systems in the Absence of Gold Labels (14)

Validating LLM-as-a-Judge Systems in the Absence of Gold Labels (15)

Appendix E Additional Case Study Setup Details and Results

We sample $N=200$ comments from the civil comments dataset and suppose that they are outputs from $\mathcal{G}_{\text{target}}$ . We stratify sampled comments by observed disagreement in forced choice responses. Specifically, we select an even sample of comments with toxicity annotations at: 10%, 20%, 30%, 40%, 50%. We use the matrix $\mathbf{F}_{i}$ to construct reverse forced choice transition matrices from observed forced choice ratings. The parametrization shown in Table 2 yields a conservative analysis of forced choice selection effects. Mapping forced choices to a response set containing both Very Toxic and Toxic would flip thresholded decisions from $s_{.5}(\hat{\mathbf{y}}^{h})$ at a smaller magnitude of $\beta$ . Figure 16 shows prompts used to collect ratings from judge systems.

Options		Response Set
Options		VT	T	N/U	VT,T	T,N/U	VT,N/U	VT,T,N/U
ForcedChoice	Very Toxic	1	0	0	0	0	0	0
	Toxic	0	1	0	0	0	0	0
	No/Unsure	0	$\beta$	$1-\beta$	0	0	0	0

Metric	Sensitivity Parameter ( $\beta$ )
	0.0	0.1	0.2	0.3	0.4
Hit Rate (h/h)	Sonnet 3.5
	0.61 (0)	0.66 (0)	0.68 (0)	0.55 (-0.29)	0.42 (-0.58)
Fleiss’s $\kappa$ (h/h)	Sonnet 3.5
	0.61 (0)	0.66 (0)	0.68 (0)	0.55 (-0.29)	0.42 (-0.58)
Cohen’s $\kappa$ (h/h)	Mistral Large
	0.27 (-0.34)	0.38 (-0.28)	0.58 (-0.10)	0.75 (-0.09)	0.77 (-0.23)
MSE (h/h)	Sonnet 3.5
	0.61 (0)	0.66 (0)	0.68 (0)	0.55 (-0.29)	0.42 (-0.58)
KLD(h,j) (s/s)	Mistral Small
	0.06 (-0.55)	0.26 (-0.40)	0.52 (-0.16)	0.84 (0)	1.00 (0)
KLD(j,h) (s/s)	Sonnet 3.5
	0.61 (0)	0.66 (0)	0.68 (0)	0.55 (-0.29)	0.42 (-0.58)
CE(h,j) (s/s)	Mistral Small
	0.06 (-0.55)	0.26 (-0.40)	0.52 (-0.16)	0.84 (0)	1.00 (0)
CE(j,h) (s/s)	Sonnet 3.5
	0.61 (0)	0.66 (0)	0.68 (0)	0.55 (-0.29)	0.42 (-0.58)
JSD (s/s)	Sonnet 3.5
	0.61 (0)	0.66 (0)	0.68 (0)	0.55 (-0.29)	0.42 (-0.58)
MSE (s/s)	Sonnet 3.5
	0.61 (0)	0.66 (0)	0.68 (0)	0.55 (-0.29)	0.42 (-0.58)
Coverage (h/hrs)	Sonnet 3.5			4o Mini	3.5 Turbo
	0.61 (0)	0.66 (0)	0.68 (0)	0.64 (-0.20)	0.96 (-0.04)
MSE (srs/srs)	Mistral Small			3.5 Turbo
	0.06 (-0.55)	0.26 (-0.40)	0.52 (-0.16)	0.79 (-0.05)	0.96 (-0.04)
Consistency ( $\tau=.5$ )	Sonnet 3.5			Mistral Small
	0.61	0.66	0.68	0.84	1.00

Metric	Sensitivity Parameter ( $\beta$ )
	0.0	0.1	0.2	0.3	0.4
Hit Rate (h/h)	Sonnet 3.5
	0.37 (0)	0.17 (0)	0.09 (0)	0.42 (+0.35)	0.58 (+0.57)
Fleiss’s $\kappa$ (h/h)	Sonnet 3.5
	0.37 (0)	0.17 (0)	0.09 (0)	0.42 (+0.35)	0.58 (+0.57)
Cohen’s $\kappa$ (h/h)	Mistral Large
	0.71 (+0.34)	0.51 (+0.34)	0.26 (+0.17)	0.07 (0)	0.24 (+0.23)
MSE (h/h)	Sonnet 3.5
	0.37 (0)	0.17 (0)	0.09 (0)	0.42 (+0.35)	0.58 (+0.57)
KLD(h,j) (s/s)	Mistral Small
	0.94 (+0.57)	0.74 (+0.57)	0.49 (+0.40)	0.16 (+0.09)	0.01 (0)
KLD(j,h) (s/s)	Sonnet 3.5
	0.37 (0)	0.17 (0)	0.09 (0)	0.42 (+0.35)	0.58 (+0.57)
CE(h,j) (s/s)	Mistral Small
	0.94 (+0.57)	0.74 (+0.57)	0.49 (+0.40)	0.16 (+0.09)	0.01 (0)
CE(j,h) (s/s)	Sonnet 3.5
	0.37 (0)	0.17 (0)	0.09 (0)	0.42 (+0.35)	0.58 (+0.57)
JSD (s/s)	Sonnet 3.5
	0.37 (0)	0.17 (0)	0.09 (0)	0.42 (+0.35)	0.58 (+0.57)
MSE (s/s)	Sonnet 3.5
	0.37 (0)	0.17 (0)	0.09 (0)	0.42 (+0.35)	0.58 (+0.57)
Coverage (h/hrs)	Sonnet 3.5			4o Mini	3.5 Turbo
	0.37 (0)	0.17 (0)	0.09 (0)	0.16 (+0.09)	0.05 (+0.04)
MSE (srs/srs)	Mistral Small			3.5 Turbo
	0.94 (+0.57)	0.74 (+0.57)	0.49 (+0.40)	0.12 (+0.05)	0.05 (+0.04)
Bias (MAE) ( $\gamma=.5$ )	Sonnet 3.5			Mistral Large	Mistral Small
	0.37	0.17	0.09	0.07	0.01

$\displaystyle\mathbb{P}(o_{k}\mid t)$	$\displaystyle=\sum_{v,{v^{}},r}\mathbb{P}(o_{k}\mid s_{v},s_{v^{}},r,t)\cdot%\mathbb{P}(s_{v},s_{v^{*}},r\mid t)$
	$\displaystyle=\sum_{v,{v^{}},r}\mathbb{P}(o_{k}\mid s_{v},t)\cdot\mathbb{P}(s%_{v},s_{v^{}},r\mid t)$	(4)
	$\displaystyle=\sum_{v,{v^{}},r}\mathbb{P}(o_{k}\mid s_{v},t)\cdot\mathbb{P}(s%_{v}\mid s_{v^{}},r,t)\cdot\mathbb{P}(s_{v^{*}},r\mid t)$
	$\displaystyle=\sum_{v,{v^{}},r}\mathbb{P}(o_{k}\mid s_{v},t)\cdot\mathbb{P}(s%_{v}\mid s_{v^{}},t)\cdot\mathbb{P}(s_{v^{*}},r\mid t)$	(5)
	$\displaystyle=\sum_{v}\mathbb{P}(o_{k}\mid s_{v},t)\cdot\sum_{v^{}}\mathbb{P}%(s_{v}\mid s_{v^{}},t)\cdot\sum_{r}\mathbb{P}(s_{v^{*}},r\mid t)$
	$\displaystyle=\sum_{v}\boldsymbol{\mathbf{F}}_{k,v}\cdot\sum_{v^{}}%\boldsymbol{\mathbf{E}}_{v,v^{}}\cdot\boldsymbol{\theta}^{}_{v^{}}.$

	$\displaystyle\delta^{*}$	$\displaystyle=\text{MSE}(\mathbf{\Omega}^{},\;\mathbf{\Omega}^{j,z})-\text{%MSE}(\mathbf{\Omega}^{},\;\mathbf{\Omega}^{j,w})$
		$\displaystyle=\|\|\mathbf{\Lambda}\boldsymbol{\theta}^{}-\mathbf{\Lambda}%\boldsymbol{\theta}^{j,z}\|\|^{2}_{2}-\|\|\mathbf{\Lambda}\boldsymbol{\theta}^{}-%\mathbf{\Lambda}\boldsymbol{\theta}^{j,w}\|\|^{2}_{2}$
		$\displaystyle=(\boldsymbol{\theta}^{}-\boldsymbol{\theta}^{j,z})^{T}\mathbf{M%}(\boldsymbol{\theta}^{}-\boldsymbol{\theta}^{j,z})-(\boldsymbol{\theta}^{}-%\boldsymbol{\theta}^{j,w})^{T}\mathbf{M}(\boldsymbol{\theta}^{}-\boldsymbol{%\theta}^{j,w})$
		$\displaystyle=(\boldsymbol{\theta}^{})^{T}\mathbf{M}\boldsymbol{\theta}^{}-(%\boldsymbol{\theta}^{})^{T}\mathbf{M}\boldsymbol{\theta}^{j,z}-(\boldsymbol{%\theta}^{j,z})^{T}\mathbf{M}\boldsymbol{\theta}^{}+(\boldsymbol{\theta}^{j,z}%)^{T}\mathbf{M}\boldsymbol{\theta}^{j,z}$
		$\displaystyle\quad\quad-(\boldsymbol{\theta}^{})^{T}\mathbf{M}\boldsymbol{%\theta}^{}+(\boldsymbol{\theta}^{})^{T}\mathbf{M}\boldsymbol{\theta}^{j,w}+(%\boldsymbol{\theta}^{j,w})^{T}\mathbf{M}\boldsymbol{\theta}^{}-(\boldsymbol{%\theta}^{j,w})^{T}\mathbf{M}\boldsymbol{\theta}^{j,w}$
		$\displaystyle=(\boldsymbol{\theta}^{j,z})^{T}\mathbf{M}\boldsymbol{\theta}^{j,%z}-(\boldsymbol{\theta}^{j,w})^{T}\mathbf{M}\boldsymbol{\theta}^{j,w}+2(%\boldsymbol{\theta}^{*})^{T}\mathbf{M}(\boldsymbol{\theta}^{j,w}-\boldsymbol{%\theta}^{j,z})$