Position bias is an AI judge's tendency to favor an answer because of where it appears in the prompt, such as first or second, regardless of its quality.
Where sales teams run into it
Sales teams meet it whenever a model compares two things side by side: two calls, two reps' answers to an objection, two drafts of a follow-up email, two coaching replies. The research literature treats it as a known weakness of using models as judges. A 2025 study of 15 model judges across more than 150,000 evaluation instances found position bias "is not due to random chance" and is "strongly affected by the quality gap between solutions" (Shi et al., IJCNLP-AACL 2025). Close calls are where it bites.
In our call-scoring research (August to October 2026, more than 5,000 tests and more than 250,000 scoring decisions), a model asked to compare two calls head to head picked the first call it read 62% of the time (field guide).
The standard fix is to run every comparison in both orders and count a win only when it survives the swap, with the judge blind to which source wrote which answer, and with more than one judge. Our benchmark for our AI sales experts uses blind, multi-judge pairwise judging; the method is described on the research hub, and the call-scoring study is the subject of Show Your Working, our series with PeerLab.
Two calls, two orders, two answers
In this illustrative case, an enablement lead asks a model which of two reps ran the better discovery call, call A pasted first. The model picks A. She swaps the order and asks again; it picks B. Neither answer means anything. She scores each call alone against the team scorecard instead, then compares the scores. The full finding is in AI picked the first call it read 62% of the time.
- 01Paste call A firstThe model picks A
- 02Swap the orderThe model picks B
- 03Score each call aloneAgainst the team scorecard
- 04Compare the scores
Related terms
- LLM-as-a-judge: using a language model to evaluate outputs
- Rank agreement: how closely a scorer's order matches the experts'
- Call scoring: rating a call against a scorecard, skill by skill
FAQ
How do you reduce position bias in an AI judge?
Run each comparison twice with the order swapped and count only results that hold both ways. Hide which source produced which answer, and use more than one judge. Where possible, score each item alone against a rubric and compare the scores.
Does position bias affect AI call scoring?
Position bias affects AI call scoring mainly when the model compares calls side by side. In our research a model picked the first of two calls 62% of the time. Scoring each call alone against a scorecard avoids it, which is how our final scoring method works.
Is position bias the same in every model?
Position bias is not the same in every model: the 2025 study found it varies significantly across judges and tasks. Test the model you use, on your own material, in both orders.