The short answer
An AI scorer can score a sales call: in our 2026 research it ranked calls close to the order trained human assessors gave them, and its yes/no checklist model changed 1.7% of its own scores on a re-run (two trained assessors disagreed on 18% when marking the same calls). The conditions matter, and most setups do not meet them.
Between August and October 2026 we ran more than 5,000 tests and more than 250,000 scoring decisions across 15 AI models, with blind human marking from PeerLab, on recorded enterprise discovery roleplays scored against a 20-skill capability scorecard. The final setup reached 0.89 rank agreement with the experts' order. The first setup we thought had won was retracted.
This piece covers what sits between those two results, and how a reader can check their own scorer against the same bar. The full write-up is the field guide Can AI judge a sales call?, and the research is walked through on camera in Episode 01 of Show Your Working.
What "scoring a call" has to mean
A sales call score is a number on a defined scale that places one call relative to others on the same skill. The useful question is whether the scorer puts calls in the same order a trained expert would, because the order is what a manager acts on. A score that says a rep is a 3 on gaining commitment means little on its own. A score that puts that rep's call below last week's and above the team median tells the manager who to talk to first.
Four properties decide whether a scorer is worth trusting. The figures under each come from our field guide.
Rank agreement. Does the scorer order a set of calls the way the experts do? We measure this as a correlation between the model's ranking and the human ranking. The production engine before the research started sat at 0.74. The final method reached 0.89. For comparison, a second trained human marker, scoring calls independently, ranked them at about 0.95 against the first.
Re-run stability. Score the same calls again with the same setup. How many scores change? The final setup's checklist model moved 1.7%. One of the leading AI models moved 15% to 17% of its scores between identical runs.
Calibration to your experts. Two trained human assessors disagree with each other on 18% of scores. A scorer that is more consistent than that is a usable instrument. A scorer that is less consistent is adding noise to a measurement that already carries some.
Quote fidelity. If the scorer cites evidence, the quote has to exist in the transcript. With cheap evidence extraction, a fifth of the quotes the model pulled were not in the call.
Average error, the metric most vendors quote, is a poor guide. A scorer that never read the transcript and returned the same middling score for every call landed an average error of 0.41. The best real model landed 0.43. Across 34 configurations we tested, average error and ranking correlated at 0.02. Judge a scorer on ranking.
Where AI scoring fails
The failures taught us more than the successes, and most of them will repeat in any team that connects a language model to a scorecard. Every figure in this section is from the field guide.
The retracted winner. In August the best configuration (context plus calibration, with two language models' scores averaged) posted a strong single-run result. Re-run ten times, it averaged 0.55 rank agreement with single runs anywhere from 0.41 to 0.75. The engine it was supposed to beat sat at 0.74. We published the headline and then withdrew it.
Newer models mark harder. Every newer model generation scored reps more harshly than the human assessors, and the gap widened with each release. The most expensive model finished eighth. GPT-6 Sol scored 0.73 across three runs ranging 0.67 to 0.78. Production GPT-5.1 scored 0.74. GPT-6 Luna scored 0.77 to 0.82. Swapping to the newest model is a change to the instrument and needs re-measuring.
Prompt fixes that made it worse. A redesigned system prompt moved up to 40% of scores on a re-run, where the unedited prompt moved 9%. Rewriting the scorecard as "observable checks" ranked 0.62. Telling the model to "use the full range" made it use less of the range. Showing it another model's scores stopped it spotting the best calls. A long-running scoring conversation drifted harsher within 18 hours.
Order bias. Asked to compare two calls head to head, the model picked the first call it read 62% of the time. Scoring single turns out of context dropped ranking to between 0.57 and 0.63.
Fractional scores. 37% of one production engine's scores were fractional on a scale defined in whole levels. Rounding alone added 16 points of exact agreement with the assessors.
Each of these failed silently. The dashboards kept filling and nothing in the output said the ranking had collapsed.
What made it work
The final method in our research scores a call in two independent passes and averages the rankings.
The checklist model (TypeSafe's Jev) answers 80 yes/no questions per call: for each of 20 skills, "is this call at least level 2? level 3? level 4? level 5?" Each question carries marker reasons written by the trained assessors, in their words, explaining what each level looks like in a real transcript. Those marker reasons were the single biggest gain in the program. Adding them lifted average fit with human scores from 0.62 to 0.72. Textbook examples of good behavior added nothing. The skills that moved most with marker reasons were framing, gaining commitment, emotional control and negotiation.
A second judge, GPT-6 Luna, scores the whole call blind, without seeing the first pass. The two rankings are averaged. Luna then writes the feedback from the final scorecard, and every quote in that feedback is checked against the transcript before it reaches a rep.
The cost is about $0.007 per call for scores and feedback. A single pass of a leading general-purpose model costs 6 to 28 times more and, as above, ranks worse. The method held at 0.79 to 0.83 rank agreement with up to a third of the words dropped from the transcript and the speaker labels swapped.
| Setup | Rank agreement with experts | Re-run change | Note |
|---|---|---|---|
| Production engine (GPT-5.1) | 0.74 | Held its lead over GPT-6 Sol | |
| August "winner" (two models averaged) | 0.55 average over 10 runs | Single runs 0.41 to 0.75 | Headline retracted |
| Most expensive model tested | Finished eighth | Price did not predict ranking | |
| GPT-6 Sol alone | 0.73 (0.67 to 0.78) | 15% to 17% | |
| GPT-6 Luna alone | 0.77 to 0.82 | ||
| Final method (Jev yes/no questions + blind Luna, rankings averaged) | 0.89 | 1.7% (checklist model) | About $0.007 per call including feedback |
All figures are from the field guide. How scorecards are configured on our platform is in the scorecard docs.
What to give the scorer before it scores anything
The scorer can only be as good as the standard it is handed. The research points to four things to prepare.
Levels defined in whole steps. Each skill needs a scale with named levels and a description of each. If the scale has five levels, the scorer should return one of five, and anything fractional should be treated as a fault to fix.
Marker reasons in your assessors' words. For each level, have the people who know the standard best write what that level looks like in a real call: what the rep said, what they skipped, what the buyer did next. This is the step that carried the largest gain, so give it the most time.
Calls from one stage. A discovery call and a negotiation call test different skills. Mixing them in one scored set makes the ranking hard to read and harder to act on.
A plain instruction, left alone. The redesigned prompt above moved up to 40% of scores on a re-run. Keep the instruction simple, change it rarely, and put the effort into the marker reasons.
Once those are in place, the test below shows whether the scorer holds to them.
How to test your own scorer in an afternoon
A team with a call recorder and a scorecard can run this today. It takes two managers and about three hours.
- Pick a set of calls from the same stage and the same segment. Discovery calls work well because the skills are visible and the stakes are real.
- Have two managers rank them independently, best to worst, on two skills from your scorecard. Do not let them confer. Keep both rankings.
- Run your AI scorer on the same calls twice, an hour apart, with nothing changed between runs.
- Compare the orders. How far apart are the two managers? That is your human baseline, and there will be disagreement. How far is the AI from each manager's order? How far is the AI from itself across the two runs?
- Read the feedback for one call and check each quote against the transcript. Count the ones that are missing.
If the scorer disagrees with itself more than the managers disagree with each other, it is not ready to be the basis of a conversation with a rep. If it agrees with itself but puts the calls in a different order from both managers, it is measuring something other than your scorecard. If its quotes are missing from the call, nothing downstream of it can be trusted.
What a trustworthy score is for
A score that ranks correctly and holds steady on a re-run is a measurement. On its own it changes nothing. The rep still has the next conversation with that buyer in four days, and the score does not attend.
The job of a score is to start the coaching. On our platform the rep brings the live deal to one of our AI sales experts on a call, talks through what the buyer said and what came back, and the expert works it with the rep: what the buyer has given so far, and what to ask for in the next conversation.
The scorecard sits under that conversation as the standard the expert holds the rep to, criterion by criterion, with the moment on the call each score came from. A rep who cannot talk at that moment can type, and the expert picks up the same deal either way.
This is the reason we ran the research behind our AI call scoring guide at all. A coach that marks harder than the experts, or whose scores move 40% on a re-run, turns every debrief into an argument about the number instead of the next call. To find out where your own discovery calls land, talk to us.
FAQ
Can AI score a sales call as accurately as a human manager?
AI can score a sales call nearly as accurately as a trained human, under the right setup. In our 2026 research the final method agreed with trained assessors' ranking at 0.89, against about 0.95 for a second human marker, and 1.7% of its checklist-model scores changed on a re-run, while two trained experts marking the same calls disagreed on 18% of scores. A single model with a default prompt did not reach that level.
Why do AI call scores change when I run them again?
AI call scores change on a second run because language models are not deterministic, and most scoring setups make it worse. In our research we saw one of the leading AI models move 15% to 17% of its scores between identical runs, and a redesigned prompt move up to 40%. Breaking the judgment into yes/no decisions with expert marker reasons, and averaging a blind second judge, brought checklist-model re-run change down to 1.7%.
Does a newer AI model score sales calls better?
A newer AI model does not score sales calls better by default. Every newer model generation in our tests marked reps more harshly than the human assessors, and the most expensive model finished eighth. GPT-6 Luna did rank well (0.77 to 0.82) and still improved when paired with a checklist model answering yes/no questions. Treat a model swap as a change to the instrument.
What is a good accuracy number for AI call scoring?
A good accuracy number for AI call scoring is rank agreement with your own experts, with average error kept only as a second number. In our tests a scorer that never read the transcript landed 0.41 average error while the best real model landed 0.43; across 34 configurations average error and ranking correlated at 0.02. Our final method reached 0.89 rank agreement. A scorer that cannot show its re-run stability has not been measured.
How much does AI call scoring cost per call?
AI call scoring with our final method costs about $0.007 per call for scores plus written feedback, according to our research. It uses a small checklist model for the yes/no questions and one pass of a cheap model, GPT-6 Luna, for the blind whole-call score. A single pass of a leading general-purpose model on its own costs 6 to 28 times more and ranked worse in our tests.
Can I trust the quotes an AI scorer pulls from a call?
The quotes an AI scorer pulls from a call can be trusted only once each one is checked against the transcript. In our tests, with cheap evidence extraction, a fifth of the quotes the model produced were absent from the transcript. In the final method every quote in the feedback is verified against the call before a rep sees it. Ask any vendor how quotes are checked, and spot-check one call yourself.
Should I test an AI scorer before using it for coaching?
Test an AI scorer on your own calls before using it for coaching, because a scorer that disagrees with itself cannot hold a rep to a standard. Pick a set of calls, have two managers rank them independently, run the scorer twice an hour apart, and compare the orders. If the scorer disagrees with itself more than the managers disagree with each other, it is not ready. The whole test takes one afternoon and needs no vendor involvement.