Learning From Pairwise Preferences: An Introduction to the Bradley Terry Model

assumes the availability of absolute labels. For example, an instance belongs to a class, a document receives a score, an observation is assigned a probability, a product is rated on a fixed scale. In practice, however, human judgment often appears in a more local and comparative form. People may not know whether an answer deserves 7.4 out of 10, but they can often say which of two answers is better. They may hesitate to assign an absolute quality score to a candidate, but they can say which of two candidates seems stronger. In many real systems, comparison is much easier than calibration.

This is the setting in which the Bradley-Terry model becomes especially useful by offering a mathematically clean way to learn from pairwise preferences. Rather than asking for absolute judgments, it starts from simple head-to-head outcomes and uses them to infer a latent ordering over items to give a coherent probabilistic ranking.

The Core Idea: Each Item Has a Latent Strength

The model begins with a simple assumption. Each item i is associated with an unobserved positive strength parameter, written as πᵢ > 0. When item i is compared with item j, the probability that i is preferred to j is defined as:

and, symmetrically we can write:

This form is quite attractive because it is both simple and interpretable. If the two items have equal strength, then each has probability 1/2 of winning. If πᵢ is much larger than πⱼ, then i becomes much more likely to win. The Bradley Tery model translates hidden relative strengths into observable pairwise probabilities.

A second and often more convenient way to write the same model is to express each positive strength as the exponential of a real-valued score:

Substituting this into the probability expression yields:

which can also be written as:

This makes an important fact visible. The probability that i beats j depends only on the difference βᵢ − βⱼ. Bradley-Terry is therefore closely related to logistic modeling. This is the same structural idea that appears in logistic regression. In logistic regression, a binary outcome is modeled by applying the logistic function to a linear score. In Bradley-Terry, the binary outcome is the result of a head-to-head comparison, and the relevant score is simply the difference between the two latent strengths. Equivalently, the log-odds that i beats j are linear in βᵢ − βⱼ, which makes Bradley-Terry a particularly natural model for pairwise preference data.

More specifically, what matters is not the absolute level of an item’s score, but its position relative to the other item in the comparison.

A Simple Example

Consider three candidate answers generated by a language model: A, B, and C. Suppose human annotators produce the following preferences:

• A is preferred to B
• A is preferred to C
• B is preferred to C

Even without any numeric ratings, a structure is already visible. A appears strongest, B next, and C weakest. The Bradley-Terry model formalizes this intuition by finding latent strengths that make these observed outcomes plausible under the model.

This is the first conceptual step worth noticing. The model does not begin with global scores and then derive pairwise outcomes. It does the reverse. It begins with local comparisons and infers the latent scores that best explain them.

Fitting the Model From Data

Now suppose that comparisons are repeated many times across a larger collection of items. For each ordered pair (i, j), let wᵢⱼ denote the number of times that item i beats item j, and let wⱼᵢ denote the number of times that j beats i.

The Bradley-Terry model fits the parameters by choosing values of the strengths that make the observed comparison data as likely as possible. This is done through maximum likelihood estimation.

For a single pair of items i and j, the likelihood contribution is:

The interpretation is straightforward. If item i beat item j many times, then the fitted model should assign a high probability to i beating j. If j also won some comparisons, then the model should account for that as well. The likelihood rewards parameter settings that place high probability on the outcomes that were actually observed.

Across all item pairs, the full likelihood is obtained by multiplying these terms together. In practice, one works instead with the log-likelihood, because it is easier to optimise. The log-likelihood is:

The fitting problem is then to find the parameter values that maximise this quantity.

A Deeper Look at Bradley-Terry Model Fitting

At an intuitive level, the optimization process adjusts the latent strengths so that the model’s predicted probabilities align with the empirical comparison outcomes.

If an item wins frequently, its strength should rise. If it loses frequently, its strength should fall. If two items split their contests roughly evenly, their strengths should move closer together. These are the informal consequences. The technical mechanism behind them is the gradient of the log-likelihood.

Using the parameterisation πᵢ = exp(βᵢ), the gradient with respect to βᵢ can be written as:

This expression is the central learning signal in the Bradley-Terry model, and it has a very clean interpretation.

• The first term, wᵢⱼ, is the number of wins item i actually achieved against item j.
• The second term, (wᵢⱼ + wⱼᵢ) P(i ≻ j), is the number of wins the current model expects item i to achieve against item j.

So the gradient is measuring a discrepancy between two quantities: observed wins and expected wins.

Gradient descent adjusts the latent strength as follows:

• If item i is winning more often than the current model predicts, then the gradient is positive, and βᵢ should increase.
• If item i is winning less often than predicted, then the gradient is negative, and βᵢ should decrease.

Learning proceeds by repeatedly correcting these discrepancies until the model’s expected outcomes are brought into as close an alignment as possible with the observed data. This is the most useful way to think about Bradley-Terry fitting. Learning is adjusting the latent strengths until expected pairwise behaviour matches empirical pairwise behaviour.

There is, however, an important subtlety of note with the Bradely Terry model. The model does not identify an absolute scale of quality. Only relative strengths matter. If every strength parameter is multiplied by the same positive constant c, the pairwise probabilities do not change:

This means the model learns a relative ranking structure rather than an absolute score in some external unit. In practice, one usually fixes the scale by imposing a normalisation, such as setting one β-value to zero or constraining the parameters to sum to a constant.

From Local Judgments to Global Structure

The deeper appeal of the Bradley-Terry model lies in the way it converts many local judgments into a single global representation. Each individual comparison says very little on its own. It tells us only that, in one head-to-head contest, one item was preferred to another. Yet when these local observations are aggregated across a dataset, a broader structure begins to emerge. The model reconstructs that structure in the form of latent strengths and pairwise probabilities.

This is why Bradley-Terry remains such a useful model, but is still arguably less well known in the Data Scientist’s toolkit. It offers a principled bridge between noisy comparative judgments and global probabilistic ranking. It respects the fact that human supervision is often easier to obtain in relative rather than absolute form, and it turns that relative evidence into something mathematically tractable.

A natural next question is why pairwise comparisons are often more stable and more reliable than direct scoring in the first place. That is where the practical appeal of comparative supervision becomes much clearer still.

Why Pairwise Comparisons Are Often Better Than Direct Scores

One of the main practical advantages of the Bradley-Terry setting is that pairwise judgments are often easier for humans to make than absolute ones. This is partly a matter of cognitive burden. Asking whether answer A is better than answer B requires a local comparison. Asking whether answer A deserves a score of 7.8 out of 10 requires an internal standard, calibration against prior examples, and a stable interpretation of what the numeric scale is meant to represent. In many domains, people are much better at the former than the latter.

This difference matters because supervision noise is not all of the same kind. Direct scores often suffer from scale inconsistency. One annotator may use the full range from 1 to 10, while another compresses nearly all judgments into the interval from 6 to 8. One reviewer may treat 5 as average, another as poor. Even the same person may score more harshly in the morning than in the afternoon. The problem is not simply disagreement about quality. It is disagreement about the meaning of the scale itself.

Pairwise comparisons avoid much of this difficulty. They do not require the annotator to anchor a judgment to a global numerical frame. They ask only for a relative decision: which of these two items is better? This is a simpler and often more stable question. As a result, comparative judgments are frequently less noisy, easier to collect consistently, and more robust across annotators.

There is also a structural reason that pairwise data is attractive. In many real systems, ranking is the true downstream objective. A search engine needs to order results. A recommender system needs to place better items ahead of worse ones. A reward model for language generation needs to distinguish preferred outputs from less preferred ones. In these settings, absolute scores may be an unnecessary intermediate abstraction. Pairwise supervision is closer to the decision problem the system is ultimately trying to solve.

This does not mean pairwise judgments are free of difficulty. They can be expensive when the number of items is very large, and they can contain cycles or inconsistencies. One annotator may prefer A to B, B to C, and yet C to A. Different annotators may disagree sharply. Even so, pairwise supervision often remains attractive because it shifts the problem from asking humans to provide perfectly calibrated scores to asking a model to infer latent structure from local comparative evidence.

That is precisely what Bradley-Terry is designed to do. It takes a collection of small, possibly noisy, head-to-head outcomes and fits a global probabilistic ranking that best explains them. The model is valuable not because pairwise judgments are perfect, but because they are often the most natural and reliable signal available.

Going Deeper: Identifiability, Curvature, and Optimization

The basic Bradley-Terry model is easy to state, but its technical structure becomes more interesting once one asks how the parameters are actually estimated and under what conditions that estimation is well behaved.

Identifiability

A first issue is identifiability. In the parameterization using positive strengths πᵢ, the probabilities are unchanged if every parameter is multiplied by the same positive constant. The reason is simple:

depends only on the ratio of the strengths, not on their common scale. If every πᵢ is replaced by cπᵢ for some c > 0, the probabilities remain exactly the same.

The same issue appears in the log-strength parameterization πᵢ = exp(βᵢ). Adding the same constant to every βᵢ leaves all pairwise probabilities unchanged, since only differences such as βᵢ − βⱼ matter. The model therefore has one redundant degree of freedom.

In practice, this is handled by imposing a normalization. Common choices include:

or

These constraints do not change the fitted probabilities. They simply fix a reference level so that the solution becomes unique.

There is also a graph-theoretic aspect to identifiability. If the comparison graph is disconnected, then the relative strengths of items in different connected components cannot be determined from the data. More generally, to estimate a meaningful global ranking, the observed comparisons must connect the items sufficiently well. Otherwise the data only identifies separate local rankings within isolated subsets.

The Log-Likelihood Again

Recall the log-likelihood:

This is the objective function we maximize. Its gradient with respect to βᵢ is:

As discussed earlier, this is observed wins minus expected wins. That gives the gradient a particularly appealing interpretation. The model increases an item’s score when the item wins more often than predicted, and decreases it when the item wins less often than predicted.

At the optimum, these discrepancies balance out as well as possible over the full comparison network.

The Hessian and Curvature

To understand the geometry of the optimization problem, it helps to examine the second derivatives. For the Bradley-Terry log-likelihood, the diagonal second derivative takes the form:

and for i ≠ j, the off-diagonal second derivative is:

whenever items i and j are compared, and 0 otherwise. Several things follow from this structure:

• First, the Hessian is negative semidefinite, which means the log-likelihood is concave in β up to the identifiability issue already discussed. This is an important property. It implies that, once the scale ambiguity is fixed, the optimization problem has a well-behaved global optimum rather than many unrelated local maxima.
• Second, the curvature depends on the term P(i ≻ j) P(j ≻ i). This quantity is largest when the contest is uncertain, that is, when the two items have similar strength and each has a substantial chance of winning. It becomes small when one item is overwhelmingly stronger than the other. Intuitively, comparisons that are already almost deterministic contribute less local curvature, because the model is already quite certain about them.

This is a useful point to mention in a technical article because it connects the mathematics to the data geometry. The most informative comparisons are often those between items of roughly similar quality. They are the contests that provide the strongest local signal about relative ordering.

Gradient Ascent

The most direct optimization approach is gradient ascent. Starting from an initial guess for the parameters, one repeatedly updates:

where η is the learning rate.

Because the log-likelihood is concave after normalization, this procedure is conceptually straightforward. At each step, the parameters are moved in the direction that increases the fit between model expectations and observed outcomes. In small or medium-sized problems, this is often perfectly adequate.

That said, plain gradient ascent is not always the most efficient approach. Its convergence rate depends on the learning rate and on the local curvature of the objective. If η is too small, learning is slow; if it is too large, updates may overshoot.

Newton and Second-Order Methods

Because the gradient and Hessian are available in closed form, Bradley-Terry can also be fitted with Newton or quasi-Newton methods. A Newton step takes the form:

where H is the Hessian matrix and ∇ℓ is the gradient vector.

The advantage of second-order methods is that they account for curvature directly. Instead of moving only according to slope, they also use information about how sharply the objective bends. This often yields faster convergence, especially near the optimum.

The drawback is computational. Computing and inverting the Hessian can be expensive when the number of items is large. For that reason, practical implementations often prefer quasi-Newton methods or specialized iterative schemes.

MM Updates

One of the classic fitting procedures for Bradley-Terry is an MM algorithm, where MM stands for minorization-maximization or majorization-minimization depending on the convention. These methods replace the difficult objective with a simpler surrogate function that is easier to optimize at each step.

For Bradley-Terry, the MM update for the positive strengths can be written in a form such as:

where:

is the total number of wins for item i, and

is the total number of comparisons between i and j.

This update has an appealing interpretation. The numerator counts how often item i actually won. The denominator reflects how much winning opportunity it had under the current parameterization. The algorithm repeatedly rescales each strength so that these quantities come into better alignment.

MM methods are popular for Bradley-Terry because they preserve positivity automatically and often behave stably in practice.

A Statistical Interpretation of the Optimum

The first-order condition for optimality is especially revealing. Setting the gradient to zero gives:

for each item i.

This says that, at the optimum, the total observed wins of item i equal the total wins expected for item i under the fitted model. In other words, the estimated strengths are those for which the model reproduces the empirical win counts as closely as possible in expectation.

This is perhaps the cleanest interpretation of Bradley-Terry learning. The model is fitted when its internal probabilistic account of the world is in equilibrium with the comparison data.

Contextual Bradley-Terry: When Strength Depends on Setting

The standard Bradley-Terry model assigns a single latent strength to each item. This is a useful simplification, but it is also an important limitation. In practice, the strength of an item often depends on the circumstances of the comparison. A language model may perform well on mathematical reasoning but poorly on creative writing. A chess player may be stronger in rapid formats than in classical time controls. A product may be preferred in one market segment but not in another.

The contextual Bradley-Terry model addresses this by allowing the latent strength to vary as a function of observable covariates. Instead of a fixed parameter βᵢ for each item, one writes:

where xᵢ is a vector of features associated with item i in the current comparison context, and w is a coefficient vector shared across all items that is estimated from data. The comparison probability becomes:

This formulation reveals a structural equivalence that is worth pausing on. If one defines the design vector for a comparison as dᵢⱼ = xᵢ − xⱼ, then the contextual Bradley-Terry model becomes:

where σ is the logistic function. This is simply logistic regression on the difference of feature vectors. Each pairwise comparison is treated as a binary classification problem, and the features are the element-wise differences between the two items’ covariate vectors.

This equivalence has a practical consequence. Any software package that fits logistic regression can be used to fit a contextual Bradley-Terry model. One constructs a training set in which each row corresponds to a comparison, the features are dᵢⱼ = xᵢ − xⱼ, and the label is 1 if i was preferred and 0 otherwise. The estimated coefficient vector w then determines how each feature contributes to the probability of winning.

What Covariates Capture

The choice of covariates determines what the model can express. In the setting of language model evaluation, relevant covariates might include the topic of the prompt (mathematics, coding, creative writing), the difficulty of the prompt (estimated from annotator agreement rates or from embedding-based predictors), the length of the prompt, or the conversational turn at which the comparison was made.

With these covariates, the model no longer estimates a single global strength for each language model. Instead, it estimates a strength profile across the feature space. A model may have high estimated strength on coding prompts but lower strength on open-ended creative tasks. The learned coefficient vector w quantifies how much each contextual feature shifts the outcome probability.

This is a meaningful departure from the standard model. In the non-contextual case, the model answers the question: “Which item is stronger overall?” In the contextual case, it answers: “Under what conditions is each item stronger, and by how much?”

Application: The Chatbot Arena

The most prominent contemporary application of contextual Bradley-Terry modelling is the LMSYS Chatbot Arena (Chiang et al., 2024), a platform for crowdsourced evaluation of large language models. Users submit prompts, receive responses from two anonymised models, and indicate which response they prefer.

The challenge facing the Arena is that naive Bradley-Terry ranking treats all comparisons as equally informative. In practice, easy prompts produce nearly indistinguishable outputs from most models, while difficult prompts reveal meaningful quality differences. A comparison on a trivial factual question contributes far less ranking signal than a comparison on a complex multi-step reasoning problem.

The Arena addresses this by incorporating prompt-level covariates into the Bradley-Terry framework. Prompt difficulty, topic category, and other linguistic properties are included as features, allowing the system to estimate context-specific ratings for each model. The result is not a single Elo score per model but a learned strength profile across the space of prompts and tasks.

Bootstrap confidence intervals are computed by resampling the comparison data and re-estimating the Bradley-Terry coefficients for each bootstrap sample, providing a measure of uncertainty in the rankings.

Bayesian Extension: TrueSkill

A related but distinct extension is the Bayesian treatment of item strengths. Microsoft’s TrueSkill system (Herbrich et al., 2006; Minka et al., 2018) replaces point estimates with posterior distributions. Each item’s strength is modelled as a Gaussian random variable with mean μᵢ and variance σᵢ². After observing each comparison, the posterior is updated:

where τ² is a system noise parameter that accounts for draws and upsets. The variance σᵢ² shrinks as more comparisons are observed, reflecting increasing confidence in the estimated strength.

The key practical benefit of this approach is that it provides a natural measure of uncertainty. An item with few comparisons has high variance and therefore a wide credible interval. An item with many comparisons has low variance and a more precise estimate. This uncertainty information can be used for adaptive matchmaking: pairing items with high uncertainty against each other accelerates the convergence of the ranking.

TrueSkill does not incorporate covariates in the same way as the contextual Bradley-Terry model, but the two ideas are complementary. One could place Bayesian priors on context-dependent strengths, maintaining posterior distributions that vary across the feature space. This remains an active area of research.

Benefits of Contextualisation

The practical benefits of the contextual extension can be summarised as follows.

• First, interpretability. Instead of a single opaque rating per item, the model provides a strength profile that reveals under which conditions an item performs well and under which it does not.
• Second, data efficiency. By leveraging the structure of the feature space, contextual models can extract more ranking signal from fewer comparisons. An item that has been compared only on coding prompts can still receive an estimated strength on mathematics prompts if the model has learned how topic affects performance from other items.
• Third, generalisation to new items. In the standard model, a new item with no comparison history has no estimated strength. In the contextual model, if the new item’s feature vector is available, its strength can be estimated via the learned coefficient vector w, without any direct comparisons. This is a form of cold-start prediction that is particularly valuable when the number of items is large relative to the number of comparisons.

Accounting for Noisy Raters: When Not All Comparisons Are Equal

The Bradley-Terry model, in both its standard and contextual forms, assumes that every observed comparison is an equally reliable draw from the model’s probability distribution. This assumption is often violated. In crowdsourced settings, where comparisons are collected from many human annotators, the quality of individual judgments varies substantially.

Some annotators are careful, consistent, and knowledgeable about the domain. Others may rush through comparisons, apply idiosyncratic criteria, or produce answers that are effectively random. A small fraction may be adversarial or inattentive. If the model treats all comparisons equally, the estimated strengths will be distorted by the noise from unreliable annotators, and the resulting rankings will be less trustworthy than the data warrants.

The Standard Model’s Implicit Assumption

Consider the standard Bradley-Terry likelihood for a single comparison in which annotator k reports that item i is preferred to item j:

This expression does not reference the annotator at all. It assumes that the outcome is a noisy observation of the true comparison probability, with no variation in noise level across annotators. The implicit model is that every annotator, regardless of expertise or engagement, has the same probability of correctly identifying the better item.

In practice, this is rarely the case. Different annotators bring different levels of skill, attention, and domain knowledge to the task. Ignoring this heterogeneity leads to biased strength estimates, overconfident rankings, and an inability to diagnose or correct for poor-quality annotations.

CrowdBT: Joint Estimation of Items and Annotators

Chen et al. (2013) proposed CrowdBT, a model that addresses this problem by jointly estimating item strengths and annotator reliabilities. The key idea is to introduce a per-annotator reliability parameter ρₖ ∈ [0, 1] that governs the quality of annotator k’s comparisons.

The comparison probability under CrowdBT is modelled as a mixture:

The interpretation of this mixture is intuitive. With probability ρₖ, the annotator observes the true Bradley-Terry outcome and reports it correctly. With probability 1 − ρₖ, the annotator produces a uniformly random answer. A perfectly reliable annotator has ρₖ = 1 and behaves exactly as in the standard model. A completely unreliable annotator has ρₖ = 0 and contributes only noise.

This formulation captures an important insight about unreliable annotators. They are not assumed to be adversarial (systematically wrong), but rather noisy (sometimes right, sometimes random). This is a more realistic model of human annotation behaviour than either assuming perfect reliability or treating low-quality annotations as inverted signals.

Estimation via the EM Algorithm

The full log-likelihood under CrowdBT is:

where Cₖ is the set of comparisons made by annotator k. This objective is optimised via the expectation-maximisation (EM) algorithm.

In the E-step, for each observed comparison, the algorithm computes the posterior probability that the annotator was behaving reliably (as opposed to guessing randomly), given the current estimates of β and ρ. Let zₖᵢⱼ denote this latent indicator. Its posterior is:

In the M-step, the item strengths β are updated to maximise the likelihood of the comparisons that are attributed to reliable behaviour, and the annotator reliabilities ρₖ are updated based on the fraction of their comparisons that the E-step attributes to genuine expertise rather than random guessing.

The algorithm alternates between these two steps until convergence. The result is a set of item strengths that have been denoised by downweighting unreliable annotators, together with a set of annotator reliability scores that can be used for quality control and diagnosis.

Practical Implications

The CrowdBT model has several practical consequences that are worth highlighting.

• First, it provides automatic quality control. Rather than requiring a separate step to identify and remove bad annotators, the model learns annotator quality as a byproduct of fitting the ranking. Annotators with low estimated ρₖ can be flagged for review, retrained, or excluded from future tasks.
• Second, it improves ranking accuracy. By downweighting noisy comparisons, the model produces item strength estimates that are less sensitive to annotation quality. This is particularly important when the annotator pool is heterogeneous, as is typical in crowdsourcing platforms.
• Third, it enables a diagnosis of annotation difficulty. If many annotators have low reliability on comparisons involving a particular pair of items, this may indicate that the two items are genuinely difficult to distinguish rather than that the annotators are poor. The model’s output can help separate annotator noise from item-level ambiguity.

Extensions: Beyond a Single Reliability Parameter

Subsequent work has extended the CrowdBT formulation in several directions.

One natural extension is to decompose annotator behaviour into reliability and bias. The single parameter ρₖ captures noise but not systematic preferences. An annotator who consistently favours a particular item regardless of its quality is not well modelled by the reliability parameter alone. Adding a per-annotator bias term allows the model to distinguish between noise (random errors) and systematic distortion (consistent favouritism).

A second extension is to allow annotator reliability to vary by domain or topic. An annotator who is an expert in mathematics may produce highly reliable comparisons on mathematical questions but much noisier comparisons on creative writing tasks. Modelling domain-specific reliability as ρₖ,c, where c indexes the comparison category, captures this heterogeneity.

A third extension, developed in the Bayesian setting, places a prior distribution on the reliability parameters. A natural choice is a Beta prior:

which encodes a prior belief about the distribution of annotator quality. This Bayesian formulation, sometimes referred to as BBQ (Bayesian Bradley-Terry with Quality estimation), provides posterior distributions over both item strengths and annotator reliabilities. It handles the case where individual annotators contribute only a small number of comparisons, using the prior to regularise the reliability estimates.

Connection to the Broader Crowdsourcing Literature

The problem of aggregating judgments from multiple noisy annotators has a substantial history in the statistical and machine learning literature. The foundational model is the Dawid-Skene model (1979), which addresses the same problem in the setting of categorical labelling. In Dawid-Skene, each annotator is characterised by a confusion matrix that describes their probability of reporting each label given the true label. The EM algorithm jointly estimates the true labels and the annotator confusion matrices.

CrowdBT can be understood as an adaptation of this principle to the pairwise comparison setting. Instead of a confusion matrix, each annotator is characterised by a reliability parameter. Instead of categorical labels, the true signal is a Bradley-Terry comparison probability. The conceptual structure is the same: jointly estimate the latent ground truth and the annotator quality, using each to inform the other.

The broader lesson from this literature is that models which jointly estimate item parameters and annotator parameters consistently outperform models that treat either dimension as fixed. Treating all annotators as equally reliable discards information about annotation quality. Treating item quality as known discards the signal that annotators are providing. The most effective approach is to learn both simultaneously, which is precisely what CrowdBT and its extensions are designed to do.

Summary

The standard Bradley-Terry model provides a clean framework for learning from pairwise comparisons, but it assumes that all comparisons are equally reliable. In practice, annotator quality varies, and this variation can distort the estimated rankings.

The CrowdBT model addresses this by introducing a per-annotator reliability parameter that governs the probability of observing a genuine comparison versus a random guess. The EM algorithm jointly estimates item strengths and annotator reliabilities, producing denoised rankings and annotator quality scores as a natural byproduct.

Extensions to domain-specific reliability, Bayesian priors, and bias modelling provide additional flexibility for applications where annotator heterogeneity is particularly pronounced. Together with the contextual extensions discussed in the preceding section, these methods transform the basic Bradley-Terry model from a tool for simple ranking into a rich framework capable of handling the complexities of real-world comparative evaluation.

Disclaimer: The views and opinions expressed in this article are my own and do not represent those of my employer or any affiliated organizations. The content is based on personal experience and reflection, and should not be taken as professional or academic advice.

📚 Further Reading

R. A. Bradley and M. E. Terry (1952) — Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons — The foundational Bradley–Terry paper. It introduced one of the canonical statistical models for pairwise comparison data, and it remains the natural starting point for any discussion of preference-based ranking.

Manuela Cattelan (2012) — Models for Paired Comparison Data: A Review with Emphasis on Dependent Data — A clear review of the paired-comparison literature, especially useful for understanding how classical models such as Bradley–Terry and Thurstone are extended when comparisons are not independent.

Xi Chen, Paul N. Bennett, Kevyn Collins-Thompson, and Eric Horvitz (2013) — Pairwise Ranking Aggregation in a Crowdsourced Setting — A useful reference for ranking under noisy human judgments. The paper focuses on how to aggregate pairwise comparisons in crowdsourced settings while accounting for annotator quality and label efficiency.

Wei-Lin Chiang et al. (2024) — Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference — The main reference for Arena-style evaluation of language models through large-scale human pairwise voting. It is especially relevant if your article connects paired comparison models to modern LLM benchmarking.

A. P. Dawid and A. M. Skene (1979) — Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm — The classic Dawid–Skene paper on estimating latent truth and annotator reliability from noisy labels. It is foundational for crowd-label aggregation and for thinking carefully about judge quality in evaluation pipelines.

Ralf Herbrich, Tom Minka, and Thore Graepel (2006) — TrueSkill: A Bayesian Skill Rating System— The original TrueSkill paper, introducing a Bayesian framework for inferring latent skill from repeated competitive outcomes. It is highly relevant when pairwise wins and losses are used to build dynamic rankings over time.

Tom Minka, Ryan Cleven, and Yordan Zaykov (2018) — TrueSkill 2: An Improved Bayesian Skill Rating System— A later refinement of TrueSkill that incorporates richer signals and improves predictive accuracy. It is helpful if you want to gesture beyond simple win/loss aggregation toward more expressive Bayesian ranking systems.