The finding
Each newer AI model generation we tested marked sales reps harder than the one before, and from GPT-5.1 onward every model scored them below the trained human assessors. On a shared scale where negative means harsher, GPT-4.1 sat at +0.18 against the human marks and GPT-6 Sol at −1.1, according to our field guide Can AI judge a sales call?.
The most expensive model in the study finished eighth. The newest top-tier model ranked no better than the engine we already ran in production. For any team scoring calls with AI, that makes a model upgrade a change to the measuring instrument, and it needs re-testing before anyone coaches from the new numbers.
This piece is one finding from our sales coaching research hub, which collects the full program.
How we measured it
The research ran from August to October 2026: more than 5,000 tests, more than 250,000 scoring decisions and 15 AI models. Trained assessors from PeerLab marked real enterprise discovery roleplays blind against a 20-skill capability scorecard and never saw a machine's answer. Each model scored the same calls, and we compared its marks with the human marks on two measures: bias (how far above or below the humans it marked on average) and rank agreement (how closely its order of calls matched the human order, where 1.0 is identical).
The results come from one scorecard and one cohort, so the exact gaps may differ on yours; the trend across generations is what to test.
Bias by model
| Model | Bias against human marks | What it means |
|---|---|---|
| GPT-4.1 | +0.18 | Slightly more generous than the experts |
| GPT-5.1 | −0.13 | Closest to the experts' marks |
| GPT-5.4 | −0.65 | Harsher |
| GPT-5.6 family | −0.75 to −1.09 | Harsher, range across the family |
| Claude Sonnet 5 | −1.01 | Harsher |
| GPT-6 Sol | −1.1 | Harshest tested |
Source: field guide, model-generation tests through October 2026.
Bias alone does not decide whether a scorer is useful. A scorer that marks everyone a point low but keeps them in the right order can be corrected with a fixed offset. The trouble is that the harsher models did not rank better either.
| Model | Rank agreement with human order | Re-run behavior |
|---|---|---|
| GPT-6 Sol (newest top-tier model tested) | 0.73, three runs ranging 0.67 to 0.78 | 15% to 17% of scores changed between identical runs |
| GPT-5.1 (production engine at the time) | 0.74 | Level with Sol on average |
| GPT-6 Luna (cheapest current model) | 0.77 to 0.82 | Even its lowest run beat both averages |
GPT-5.1 also won our original eleven-model comparison (the first eleven models tested, before the newer releases were added) on the other measures: the lowest average error of any real model (0.43), the closest to the experts' marks (bias −0.13) and the best exact agreement (59%).
Why public leaderboards do not answer this question
Public model leaderboards rank models on general tests: reasoning, coding, knowledge, plus price and speed. Some fold many tests into a single intelligence index. They are useful for choosing a model for general work. They say nothing about whether a model's 3 on your scorecard means what your best assessor's 3 means, which is the only thing a call scorer is for.
There is a second gap. A public benchmark is a fixed set of questions that labs know about, so scores on it can rise without the model getting better at anything new. A team's own marked calls avoid both problems. The model has never seen them, and the answer key comes from the people whose judgment the team already trusts.
A model can climb a general leaderboard and drift further from your assessors on the same release, which is what the results on this page show. A scoring decision should rest on the test that uses your calls.
Price did not predict quality
In our AI call scoring guide, cost and quality ran in opposite directions often enough to matter. GPT-6 Luna, the cheapest current model we tested, out-ranked both the newest top-tier model and the production engine on average, and even its lowest run (0.77) beat both averages.
The final setup we landed on uses neither the biggest model nor a single model. A checklist model, TypeSafe's Jev, answers 80 yes/no questions per call, and GPT-6 Luna scores the whole call blind; their rankings are averaged. That blend reached 0.89 rank agreement, and about $0.007 per call covers scores plus written feedback. One pass of a top-tier model costs 6 to 28 times more. The build is covered in how giving the model the assessors' reasons improved it.
Why the newest models mark harder
Out of the box, the newest models struggle to tell great work from good work. The field guide's working explanation is that early chat models were tuned to please and would call any idea brilliant. As the labs moved into work tasks, they trained much of that flattery out, and scoring is where the correction shows most. That explanation is ours and unproven; the measurement stands without it.
A fair challenge is whether the experts set the bar too high. If their marks were off, no scorer would line up with them, and the final setup did line up closely. Trained experts still disagree with each other on 18% of the scores they both give, so they are an imperfect bar, and we measured against them knowing that.
The obvious fix, telling a harsh model to be more generous, failed. Every prompt variant that explained the marking agreed less with the experts, and "use the full range" made the model use less of it. The detail is in five prompt fixes that made AI call scoring worse. Giving GPT-6 Sol the experts' reasons left its ranking at 0.73, exactly where it started.
What it does to coaching
A rep who gets a 2 where the team's experts would give a 3 hears that they are worse than they are. Do that across a team and the debrief turns into a telling-off. Worse, the harsh scorer squeezes strong calls toward the middle, so the manager loses the signal for who is doing the work well and who needs help first.
The second problem is quieter. If the team's scorer moves from one model to another and nobody re-tests, last quarter's 3 and this quarter's 3 are different measurements. A rep's "decline" on a dashboard can be a model change.
On our platform the score sits under a conversation. The rep talks the live deal through with one of our AI sales experts on a call, and the expert works out with them what is missing and what to ask next. A rep who cannot talk at that moment types, and the expert picks up the same deal.
The score is the standard the expert holds the rep to, so it has to match the team's own assessors, and that is why we re-test our scoring on every model release. Coaching quality is a separate test. On that one, our flagship experts were measured against the newest models: in 120 sales situations they ranked first against the newest models from Anthropic, OpenAI and Google, winning 79% of 3,840 blind AI-judge votes (20 October 2026, the benchmark).
How to re-test before an upgrade
Re-test by running the new model on calls your own assessors have already marked, and checking it against those marks before it goes live. The field guide's checklist applies to an in-house scorer and a vendor's alike.
- Keep a marked set. Have a set of your own calls marked blind by people who never see an AI score, with a one-line reason for each mark.
- Score the set with the current model and the candidate. Same transcripts, same scorecard, same prompt, fresh conversation for each call.
- Compare bias and ranking. Check how far each model sits above or below the human marks, and how closely its order of calls matches the human order. A new model can match on ranking and still redefine what a strong call looks like.
- Score the set twice with the candidate. Count how many scores change between identical runs.
- Correct after scoring. If the candidate ranks well but marks low, apply a fixed offset to its output and re-test. Do not add instructions telling it to mark higher.
Keep the transcript, scorecard, model version, prompt and date with every run, so a change can be traced. For the measure to use when comparing, see average error flatters a lazy scorer. For a separate bias that appears when a model compares two calls, see AI picked the first call it read 62% of the time.
Questions to ask a vendor about model changes
Many teams do not run their own scorer. They buy one, and the model underneath can change without a release note. Five questions tell you whether a vendor treats a model change as a change to the measurement.
- Which model scores our calls today, and how will we hear when it changes? A vendor that cannot name the model, or will not commit to telling you, is asking you to trust a number you cannot trace.
- What do you re-test against before switching? Look for a set of calls marked by people, with bias and ranking compared before and after. A demo on new calls proves nothing about consistency with last quarter.
- How many scores move when you run the same call twice? Ask for the figure, and ask how it was measured.
- Can we see old and new scores side by side for our own calls? A short overlap period, where both models score the same calls, shows whether a rep's trend is real.
- What happens to historical scores? If they are not re-scored, dashboards that compare quarters across a model change need a marker on the date it happened.
If the answers are vague, keep your own marked set and run step 3 of the re-test yourself on a sample each quarter.
The models are leaving anyway
OpenAI's API deprecations page, checked on 21 October 2026, lists GPT-5.1 as deprecated on 1 October 2026 and removed from the API on 1 April 2027, with GPT-6 Sol recommended instead, the model that marked harshest in our tests. GPT-4o, GPT-4o mini, GPT-4.1 and GPT-4.1 mini go on 14 January 2027. From 6 January 2027 nobody can start a new fine-tuning job.
Every team scoring on one of those models will change models within months, whether it plans to or not. The research is discussed on camera in Episode 01 of Show Your Working. A separate benchmark of models writing sales emails is at the email LLM benchmark. Talk to us.
FAQ
Do newer AI models score sales calls better?
Newer AI models did not score sales calls better in our tests. Each newer generation marked reps harder than the last, moving from +0.18 for GPT-4.1 to −1.1 for GPT-6 Sol against trained human assessors. GPT-6 Sol ranked calls at 0.73, no better than the older production engine at 0.74, and the most expensive model finished eighth. Re-test any new model on your own marked calls first.
Why do AI models give sales reps low scores?
Newer models mark harder than human experts against the same scorecard. Our working explanation is that labs trained flattery out of their models as they moved into work tasks, and scoring is where that correction shows most. Telling the model to be more generous made agreement worse; a fixed offset after scoring is the safer correction.
Which AI model is best for scoring sales calls?
On its own, GPT-6 Luna ranked best in our study at 0.77 to 0.82, ahead of GPT-6 Sol and GPT-5.1. The best result came from a blend: a checklist model answering 80 yes/no questions plus a blind GPT-6 Luna pass, which reached 0.89 rank agreement at about $0.007 per call for scores and feedback.
Should I switch my call scorer to the newest model?
Switch your call scorer to the newest model only after re-testing it on the same marked calls you use today. Compare its bias against human marks and its ranking, and score the set twice to see how many scores move. GPT-6 Sol changed 15% to 17% of its scores between identical runs in our tests.
Is GPT-5.1 being retired?
Yes, GPT-5.1 is being retired: OpenAI's deprecations page, checked on 21 October 2026, lists it as deprecated on 1 October 2026 and removed from the API on 1 April 2027. GPT-4o, GPT-4o mini, GPT-4.1 and GPT-4.1 mini go on 14 January 2027. Teams scoring on those models will need to re-test a replacement.