The finding
A call scorer that never opened the transcript and gave every call the same score landed 0.41 from trained human assessors on average. The best real AI model landed 0.43. On average error, the scorer that read nothing won, according to our field guide Can AI judge a sales call?.
Across 34 scoring configurations, average error and ranking correlated at 0.02. They were unrelated. A buyer who compares AI scorers on average error, the number most vendors quote, can pick one that cannot tell a strong call from a weak one.
This is one finding from the sales coaching research hub, which collects the full program.
The lazy scorer test
The research ran from August to October 2026: more than 5,000 tests, more than 250,000 scoring decisions and 15 AI models, with real enterprise discovery roleplays marked blind by trained PeerLab assessors against a 20-skill capability scorecard. The assessors never saw a machine's answer.
Alongside the real models we ran a baseline that ignores every transcript and returns the same middling score for each call. It is the simplest possible scorer, and every test should include one. The constant score has no opinion about any call, so it cannot find a strong one or a weak one, and those are the calls a manager needs to find. It still beat every model that read the calls.
Four measures and what each misses
The field guide judged every setup on ranking, exact agreement and consistency, and kept average error alongside them.
| Measure | The question it answers | How it can mislead |
|---|---|---|
| Average error | How far, on average, does each mark land from the human mark? | A constant score that reads nothing scored 0.41 against the best model's 0.43 |
| Exact agreement | How often does the machine give the same whole mark as the human? | Fractional scores and rounding can move it without any change in judgment |
| Rank agreement | Does the scorer put calls in the same order as the experts? 1.0 is an identical order | Says nothing about severity: a scorer can rank well and mark everyone low |
| Consistency | Score the same calls twice: how many marks change? | A scorer can repeat a wrong answer perfectly |
No single figure carries it. The guide's checklist asks for ranking, both ends, re-run consistency and bias, with average error kept alongside.
Why ranking matches the manager's job
A manager uses a score to decide where to spend the next hour: whose call to review, which rep needs help on which skill, whether a rep is improving. Each of those decisions depends on order. Is this call better than last week's? Is this rep below the team on gaining commitment?
Ranking measures that directly. The final setup in the call-scoring research, a checklist model answering 80 yes/no questions per call and a blind GPT-6 Luna pass with the rankings averaged, reached 0.89 rank agreement with the experts' order, and 1.7% of its checklist-model scores changed on a re-run. For comparison, the production engine it replaced ranked at 0.74, and an August configuration we first thought had won averaged 0.55 over ten re-runs.
Ranking still needs the ends checked. The call-scoring guide compares the blend with a second human marker: rank agreement 0.89 against about 0.95, strongest calls found 86% against about 81%, weakest calls found 71% against about 86%. The second-marker figures come from calls two of the experts marked independently. The weakest-call gap limits how confidently a manager can use any scorer to find the calls that need help, so ask for both ends.
Severity matters too, and ranking hides it. Newer models in the study marked reps harsher than the humans, down to −1.1 for GPT-6 Sol. A harsh scorer can keep calls in order and still tell every rep they are worse than they are, which is covered in newer AI models mark reps harsher.
A worked example on six calls
The numbers below are illustrative, made up to show the arithmetic. The study's figures are the ones above.
Six calls get human marks on one skill, on a 1 to 5 scale: 2, 3, 3, 3, 3, 4. Scorer A never reads anything and gives every call a 3. Its average error is 2 misses of 1 point over 6 calls, about 0.33, and its ranking is undefined, because every call ties.
Scorer B reads each call and gives 1, 3, 3, 4, 3, 5. It is a point off on three calls, so its average error is 0.5, worse than Scorer A. It also put the weakest call at the bottom and the strongest at the top, which is what a manager needs from it on Monday morning.
On average error, A wins. On every decision a manager would make with the score, B wins. The study saw the same pattern at scale.
Exact agreement has its own trap
One production engine in the study returned a fractional score 37% of the time. A machine's 2.7 can never exactly match a human's 3. Rounding alone added 16 points of exact agreement, without changing a single judgment.
That cuts both ways. A vendor quoting low exact agreement may be running a scorer whose decimals hide decent judgment. A vendor quoting high exact agreement may have rounded its way there. In the field guide's method a rep never sees a decimal: the skill score becomes a whole level on the scorecard, mapped so the scores spread across levels the way your own assessors' scores do. That display mapping is kept separate from the ranking used to compare calls.
GPT-5.1 shows how the measures can disagree. It won our original eleven-model comparison on average error among real models (0.43), on closeness to the experts' marks (bias −0.13) and on exact agreement (59%). On ranking it sat at 0.74, well below the final blend.
How to run the lazy scorer test on your own calls
You can run the same check on any scorer with calls you already have, and it takes an afternoon of marking plus a spreadsheet.
- Pick calls that cover the range. Include a few you know went badly and a few that went well, so there are real ends to find.
- Have people mark them blind. Two assessors score each call against your scorecard without seeing any AI score. Keep both sets of marks, because the gap between them tells you how much your own markers disagree.
- Score the same calls with the AI scorer. Record the model version, prompt and date next to the scores.
- Build the lazy baseline. Give every call the same score: the average of the human marks.
- Compare on two numbers. Work out the average error for both the scorer and the baseline, then the rank agreement for the scorer, using any rank correlation your spreadsheet offers.
- Re-run once. Score the calls again with nothing changed and count the marks that move.
If the scorer's error is no better than the baseline's and its ranking is weak, it is not reading the calls in a way a manager can use. If the order holds and few marks move on the re-run, look next at which end it misses and whether it marks harsher than your people.
The usual slip is marking the calls after the AI has scored them. Once an assessor has seen the machine's number, the test measures how persuasive the machine is.
Questions to put to any scorer
The field guide's checklist applies equally to an in-house build and a vendor's product. Ask for results against blind human marks, and keep the transcript, scorecard, model version, prompt and date with each run. The test above already covers ranking and re-run stability. Three questions go further.
- Which end does it miss? Ask for the share of the strongest and weakest human-marked calls the scorer finds, and look at any call it placed at the wrong end.
- What happens to our baseline when the model changes? Ask for the re-test on the same marked calls before any upgrade.
- Who marked the benchmark, and could they see the AI score? A benchmark marked by people who saw the machine's answer measures agreement with the machine.
Two related failures bear on any vendor's numbers: AI picked the first call it read 62% of the time when comparing calls head to head, and the reasons trained assessors give did more for accuracy than any prompt.
Where the score goes next
A score that ranks correctly is a measurement. The coaching happens around it. When a rep talks a live deal through with one of our AI sales experts (on a call, or typed when they cannot talk), the expert holds the rep to the team's scorecard while working out what the buyer said, what the deal is missing and what to ask in the next conversation. A coach's read of a call has to match how the team's own assessors would judge it, which is why we test our scoring against blind human marks.
If a vendor hands you an average error, ask for rank agreement against your own blind-marked calls before you read anything into it. The research is discussed on camera in Episode 01 of Show Your Working, and to test a scorer against your own blind-marked calls, talk to us.
FAQ
How do you measure the accuracy of an AI call scorer?
Measure rank agreement against blind human marks, then check both ends, re-run stability and bias. In our call-scoring research, average error and ranking correlated at 0.02 across 34 configurations, so average error alone says almost nothing. Our final setup reached 0.89 rank agreement with the experts' order, and 1.7% of its scores changed on a re-run.
Why is average error a bad metric for AI call scoring?
Average error rewards scores near the middle, where most human marks sit. In the study, a scorer that never read the transcript and gave every call the same score landed 0.41 from the experts, while the best real model landed 0.43. The lazy scorer wins on error and cannot find a single strong or weak call.
What is rank agreement?
Rank agreement measures how closely a scorer's order of calls matches the human experts' order, where 1.0 means an identical order. It answers the question a manager uses a score for: which calls are better and which need help. It does not capture severity, so check bias against human marks as well.
Is exact agreement a good measure of AI scoring accuracy?
Exact agreement works as a measure of AI scoring accuracy only with care. In the study, 37% of the scores from one production engine were fractional, and a 2.7 never exactly matches a human 3. Rounding alone added 16 points of exact agreement without changing a single judgment. Ask whether scores were rounded, and look at ranking alongside it.
What accuracy should I expect from an AI call scorer?
Ask for rank agreement against your own blind-marked calls. In our study GPT-6 Sol on its own ranked at 0.73, the older production engine at 0.74, and our final method, which answers 80 yes/no questions per call and averages its ranking with a blind GPT-6 Luna pass, at 0.89. A second human marker ranked at about 0.95.