The finding
Asked which of two sales calls was stronger, an AI model picked the first one it read 62% of the time. With no bias the figure would sit near 50%. That result comes from our field guide Can AI judge a sales call?, and it is why the final scoring method we built never asks a model to compare two calls.
The finding matters beyond call scoring. Every team that asks an AI to pick the better of two answers, two emails or two reps is running the same experiment, often without checking the order. This piece is one finding from the sales coaching research hub.
What position bias is
Position bias is the tendency of a language model acting as a judge to prefer an answer because of where it appears in the prompt. When a model sees two items and is asked which is better, the order alone moves the verdict.
It is a known property of LLM-as-a-judge setups in general. The useful question for a sales leader is how big it is on sales calls. A discovery call is long, and two of them side by side ask the model to hold both in view and weigh them on many skills at once. That is a hard judgment, and a hard judgment is where a shortcut like "the first one was better" gets room to operate.
Why it happens
Position bias happens because a language model reads its input in order, and the order leaves a mark on what it weighs. Researchers who study the effect generally trace it to how these models are built and trained, and report that it varies a lot between models and between tasks. We did not test the mechanism ourselves. What we can say from our own data is where it showed up and what removed it.
Two practical points follow from the general picture; we saw the first directly:
- Close calls are the most exposed. When one item is clearly better, most judges pick it whatever the order. When two items are close, the order has room to decide. Two discovery calls from reps on the same team are often close.
- Long inputs are harder to hold in view. Two full call transcripts in one prompt make a long input. Material in the middle of a long prompt can get less weight than material at the start or the end, which is one more reason to give each call its own conversation.
Position bias and the other judge biases
Position is one of several shortcuts an AI judge can take when the judgment is hard. Others that teams meet often:
| Bias | What the judge prefers | Quick check |
|---|---|---|
| Position | The item in a particular place, often first | Swap the order and rerun |
| Length | The longer answer or the longer call | Compare a short strong item with a long weak one |
| Fluency | The answer that reads smoothly | Include a well-written answer that is wrong |
| Anchoring | Whatever score or verdict it was shown first | Remove other scores from the prompt |
| Drift | Harsher or softer marks as a conversation gets long | Start a fresh conversation per item |
The fixes overlap. Written criteria, fresh conversations and single-item scoring reduce most of them at once.
How it showed up in our study
The research ran from August to October 2026: more than 5,000 tests, more than 250,000 scoring decisions and 15 AI models. Trained assessors from PeerLab marked real enterprise discovery roleplays blind against a 20-skill capability scorecard, and never saw a machine's answer. That gave us a human order of calls to test every setup against.
One setup we tried in our call-scoring research was head-to-head comparison. Here the model saw two calls and said which rep performed better. In principle, enough pairwise verdicts produce a ranking. In practice the model picked whichever call it read first 62% of the time, so the ranking partly reflected the order we fed the calls in. We dropped the approach.
The final method scores every call on its own. A checklist model, TypeSafe's Jev, answers 80 yes/no questions per call ("did the rep reach at least level 2? 3? 4? 5?" for each of 20 skills), and GPT-6 Luna scores the whole call blind. The two rankings are averaged. That reached 0.89 rank agreement with the experts' order. No call is ever placed next to another call inside a prompt.
Related failures from the same study
Position bias was one of several ways a model's judgment bent to something other than the call. The field guide records each.
| Failure | What happened | Figure |
|---|---|---|
| Head-to-head calls | Model favored the call it read first | 62% first-call picks |
| Single turns out of context | Judging individual turns of a call stripped the context a skill depends on | Ranking fell to 0.57 to 0.63 |
| Showing another model's scores | The model anchored on them and stopped spotting the best calls, even with a warning | Top marks lost |
| One long scoring conversation | A scoring account with history drifted harsher | Within 18 hours; a fresh account fixed it |
The common thread is that the model's verdict is shaped by whatever else is in its context: an earlier call, another model's numbers, its own earlier answers. The fix in each case was to remove that context, which is why the final method gives every call a fresh conversation and keeps the second scorer blind to the first.
How to run a fair comparison
Run a fair comparison by judging every pair twice, once in each order, and counting only the verdicts that agree. Sometimes a head-to-head comparison is the right tool: two versions of a coach's answer, two email drafts, two talk tracks. If you need one, these steps keep the order from deciding it.
- Run every pair in both orders. Ask A-then-B and B-then-A in separate, fresh conversations. If the verdict flips with the order, count it as a tie.
- Hide the labels. Strip model names, rep names and anything that says which version is "new". The judge should see two unlabeled items.
- Use more than one judge. A judge from a single model family can carry that family's tastes. Several judges, each blind, give you a majority that no single quirk decides.
- Give the judge criteria in plain words. A judge with no criteria falls back on length, fluency and position. Written criteria from the people who own the standard narrow what it can prefer.
- Keep a control. Include pairs where both items are identical. A fair judge splits them evenly; a biased one shows its lean.
- Score single items where you can. If a scorecard exists, score each item against it separately and compare the scores. That removes the order from the prompt altogether.
A worked example
This example is illustrative. The team and the calls are invented to show the procedure.
An enablement lead wants to know which of two reps, Dana and Morgan, ran the better discovery call on the same fictional account. She pastes both transcripts into an assistant, Dana's first, and asks which was stronger. The answer: Dana.
She runs it again in a fresh conversation with Morgan's call first. The answer: Morgan. By the rule above, that is a tie, and the first verdict told her about the prompt more than about the reps.
So she switches to single-item scoring. She scores each call on its own against the team's scorecard, in a fresh conversation each time, and compares the two results skill by skill. Dana scores higher on gaining commitment because she agreed a dated next step with the budget holder. Morgan scores higher on personas and drivers because he got the buyer to say what the operations director cares about. Now she has something to coach on both calls, and she did not need a winner at all.
The pairwise question, "who was better?", is rarely the question a manager needs answered. "What is each rep missing on this deal?" usually is.
Where order can bend a sales team's AI verdicts
Head-to-head call scoring is the obvious case. Order can also matter anywhere a model sees several items in one prompt and is asked to choose or rank.
- Ranking a list of deals. "Which of these ten deals is most at risk?" puts ten deals in one prompt. Shuffle the list and ask again. If the top three change, the ranking depends partly on the order you pasted them in.
- Picking the example call. An enablement lead pastes three calls and asks which to use in training. The call that went first or last may win for reasons unrelated to the selling.
- Choosing between drafts. Two versions of a cold email, a proposal summary or a coach's answer, compared in one message.
- Summarizing several calls. A summary of five calls in one prompt can lean on whichever calls the model weighed most. Check that the summary's claims trace back to every call in it.
- Vendor bake-offs. If you ask an AI to compare two vendors' outputs on your deal, the same rule applies: run it both ways.
In each case the fix is the one above: separate conversations, both orders, and a single-item score wherever a written standard exists.
How to audit a tool you already use
If a tool in your stack compares or ranks items, you can check it for order effects in an hour.
- Pick ten pairs. Use real items the tool would normally compare: calls, emails, talk tracks.
- Run each pair twice. Once as A then B, once as B then A, each in a fresh session.
- Count the flips. A pair whose verdict changes with the order is a tie. Write down how many of the ten flipped.
- Add three identical pairs. Give the tool the same item twice. Count how often it picks the first copy. An unbiased judge should split them roughly evenly over enough runs.
- Report the result to the vendor. Ask how they control for order, and whether verdicts shown to users were run in both orders.
If more than a couple of the ten pairs flip, treat the tool's single-order verdicts as unreliable until the vendor changes the method.
What we do instead
For calls, we score each one alone against the team's scorecard, as above. For the coaches themselves, where the question is which of two coaching answers is better, we use pairwise judging because there is no fixed scorecard for a good answer to an open question. Every pair is judged blind, in both orders, by more than one judge, against written criteria set by methodology experts. The test is whether the rep or manager is likely to act on the answer. The method is described in how we benchmark AI sales coaches.
The 62% figure from our call-scoring research is the reason for the both-orders rule. A coach that wins half its comparisons because it was read first has not won anything.
What this means for a sales team
For a sales team, an AI verdict that one call beat another is worth trusting only when it was run in both orders. If a tool tells you one rep's call beat another's, ask how it compared them. If the answer is "the AI read both and picked", ask whether it ran both orders. The same question applies to a vendor's claim that its coach beats a rival's, and to an enablement test where an AI picked which talk track to roll out; a pair that flips goes to the managers who will coach it. The bias does not announce itself, because each verdict reads as confident and well argued whichever order produced it.
A trustworthy score is where coaching starts. With our AI sales experts the rep talks the live deal through on a call, the expert works out what the deal is missing, and the two prepare the next conversation with the buyer. Typing is the fallback. Our call-scoring research was built around the failures in the table above.
At the 62% first-read rate we measured, a single-order verdict is close to a coin with a thumb on it. Related findings: newer AI models mark reps harsher, average error flatters a lazy scorer and marker reasons beat textbook examples. The research is discussed on camera in Episode 01 of Show Your Working.
FAQ
What is position bias in LLMs?
Position bias is when a language model judging two or more items prefers one because of where it appears in the prompt. In our call-scoring study, a model asked which of two sales calls was stronger picked the first one it read 62% of the time, where an unbiased judge would land near 50%.
Can AI fairly compare two sales calls?
Not reliably when it reads both in one prompt. In our tests the model picked the first call 62% of the time. Scoring each call alone against a scorecard avoided the problem: our final method ranked calls at 0.89 agreement with trained human assessors without ever putting two calls side by side.
How do you reduce position bias in an LLM judge?
Run every comparison in both orders in fresh conversations and treat flipped verdicts as ties. Hide labels, use more than one blind judge, give the judge written criteria, and include identical pairs as a control. Where a scorecard exists, score each item separately and compare the scores.
Does an AI scorer get biased by other information in the prompt?
An AI scorer can be biased by other information in its prompt, and in our tests it was. In our call-scoring research, showing a model another model's scores made it stop spotting the best calls, even with a warning. A long-running scoring conversation drifted harsher within 18 hours. Judging single turns out of context dropped ranking to 0.57 to 0.63. Whole-call scoring, fresh conversations and blind scorers fixed them.
How does Replicate Labs avoid position bias?
In our research, the final method scored every call on its own with two scorers kept blind to each other, so no call sat beside another in a prompt. When we compare coaching answers, every pair is judged blind, in both orders, by more than one judge, against a written standard of what a good answer looks like.
Does an AI judge always prefer the first item?
An AI judge does not always prefer the first item; the direction depends on the model and the setup. In our study the model favored the call it read first. We did not test every model, prompt or input length, so a different setup could lean the other way. Test the tool you use in both orders, without assuming a direction, and treat any verdict that flips with the order as a tie.