Human call reviewers vary more than many teams assume: in our research, trained expert assessors scoring the same sales calls against the same scorecard disagreed with each other on 18% of their scores.
Where the 18% comes from
That figure comes from blind marking by PeerLab's trained assessors in our research program, August to October 2026, which ran more than 5,000 tests and more than 250,000 scoring decisions on recorded enterprise discovery roleplays scored against a 20-skill capability scorecard (field guide). These were people trained to score, working from written level descriptions. Front-line managers scoring between deal reviews, with less training and a looser scorecard, would likely disagree more.
The causes are mostly habits, and the section on why reviewers drift apart lists the common ones. The full measurement approach sits in the AI call scoring guide.
Two things narrow the gap. The first is writing the reasons behind each level into the scorecard, so a reviewer can check a call against why a level is earned instead of a one-word label two people read differently.
The nearest evidence we have comes from a machine. Giving our AI scorer the short notes trained assessors wrote on why a real call sat at each level lifted its average fit with human scores from 0.62 to 0.72 in our tests, the largest single gain we measured. That is a finding about an AI scorer. We have not measured the effect on human reviewers, though it points at the same weak spot: a label with no reason behind it leaves the reader guessing.
The second is a regular calibration session where managers score the same calls blind and agree rulings.
Same score or same order
Consistency means two different things, and coaching depends more on the second. Two reviewers can give different levels to the same call and still agree that call A was stronger than call B. For deciding which rep and which skill to coach, the order is what counts. We measured scorers by rank agreement for that reason; across 34 scoring configurations, average error and ranking correlated at only 0.02, so a scorer can look accurate on averages and still put calls in the wrong order.
Machines are not consistent by default either. One general-purpose model in our tests changed 15% to 17% of its scores between identical runs. After testing and fixes, the scorer in our final setup changed 1.7% of scores on a re-run, and the full setup ranked calls like the experts at 0.89. A checked AI scorer can hold the standard steady, while an unchecked one adds a second source of noise.
A ten-call check between two managers (illustrative)
The managers and results here are illustrative. Two sales managers each score the same ten discovery calls alone, on the six skills of the team scorecard. They compare in two ways: how many of the 60 skill scores differ, and whether they ranked the ten calls in the same order. They differ on 17 scores, about 28%, but put the top three and bottom three calls in the same places.
Most of the disagreement sits on one skill, impact, where one manager gives a level 3 whenever the rep asks about cost and the other wants a figure from the buyer. They agree the rule, write it into the level description, and the next month's check shows fewer splits on impact. The rep who kept scoring 2 on impact gets the coaching ask before their next call on a live deal.
Why two reviewers drift apart
Most disagreement between managers comes from a handful of habits, and each one has a fix you can write down.
Scoring the deal outcome. A manager who knows the deal closed reads the discovery call more kindly. A manager who knows it was lost finds the gap that lost it. The call was the same call either way. Score from the transcript and the scorecard, and keep the deal outcome out of the room until the scores are in.
Scoring against their own strength. A manager who was known for closing marks every soft close harshly and lets a thin discovery pass. Another who came up through technical sales does the reverse. Neither is wrong about what good looks like in their own area. They are each applying a private standard to one skill, and the rep hears two different messages.
Filling gaps in a vague level. "Explores impact well" means one thing to a manager who wants a number from the buyer and another to a manager who is happy with a question about cost. Where the level description is vague, each reviewer quietly writes their own. This is where most splits on a single skill come from, as in the ten-call check above.
Letting one moment color the whole call. A strong opening or an awkward pricing answer can drag every other score with it. Scoring each skill on its own evidence, with a quote from the call next to it, keeps one moment from setting the level for everything.
Scoring the wording. If the scorecard says "confirms next steps", one manager wants the phrase "next steps" and another wants a date and an attendee list. Score what the behavior is for, in this case a meeting the buyer has agreed to, and accept any wording that gets there.
How to run a calibration session
A calibration session is the routine version of the ten-call check. It works best once a month, takes about an hour, and needs nothing more than a shared spreadsheet.
- Pick the calls. Choose three to five recent calls on the same stage of the sale, including at least one the team would call strong and one it would call weak. Real calls from your own pipeline beat a training library, because the arguments are about your buyers.
- Score alone, before the meeting. Each manager scores every call on the team scorecard without seeing anyone else's marks, because once one manager says a call was strong, the others drift toward that view. Ask for a sentence on why for any score they had to think about. Those reasons are the useful output of the hour.
- Lay the scores side by side. Count the splits per skill, call by call, since two managers whose averages match can still disagree on most individual calls. Then check whether everyone put the calls in the same order. A team that agrees on the order but splits on levels has a wording problem. A team that disagrees on the order has a standard problem.
- Argue the biggest splits on a call. Take the two or three skills with the most disagreement. Each manager reads out the moment in the transcript that set their score. Agree a ruling, and write it as a sentence a new manager could apply without asking: "Level 3 needs the buyer to put a cost or a number on the problem."
- Change the scorecard, then re-check. Add the ruling to the level description. Next month, look at whether that skill splits less. If it does not, the wording is still open to interpretation.
Keep a running log of rulings. After a few months it is the first thing to hand a new manager who wants to know how the team judges a call.
A level description two managers read the same way
Most of the work is in the wording. Here is one skill before and after a calibration session.
| Level | Before | After |
|---|---|---|
| 1 | Does not explore impact | Never asks what the problem costs or what happens if nothing changes |
| 2 | Some exploration of impact | Asks about impact, but the buyer gives no number, date or consequence |
| 3 | Explores impact well | The buyer states a cost, a deadline or a consequence in their own words |
| 4 | Excellent exploration of impact | The buyer puts a figure on the problem and links it to someone senior who cares about it |
The "after" column can be checked against the transcript. Two managers reading the same call will mostly land on the same row, and when they do not, the argument is about a specific line from the buyer.
Look for patterns across calls
One call is a small sample. A rep can have a weak discovery call because the buyer was distracted, the meeting was cut to fifteen minutes, or the wrong person joined. Before a score turns into a coaching conversation, look for the same gap across three or four calls. If impact is a 2 on every one, that is the skill to coach. If it is a 2 on one call and a 3 on the others, ask the rep about that call first. In either case, ask how they think the call went before sharing any score; the answer shows whether they can see the gap themselves.
The same goes for the team. When most reps score low on the same skill, the problem is usually the training or the scorecard, and coaching reps one by one will not fix it.
Measuring your own team
The ten-call check takes an afternoon and tells you which skill your managers read differently. Can AI score a sales call? compares human and machine consistency in more detail, and How to score a discovery call gives the method for one call. To add an AI scorer as a third reader on the ten calls, talk to us.
FAQ
What is inter-rater reliability for sales calls?
Inter-rater reliability for sales calls is how closely two reviewers agree when they score the same call on the same scorecard. Measure it as the share of scores that differ and as whether the reviewers put calls in the same order. Trained experts in our research disagreed on 18% of individual scores.
Why do two managers score the same call differently?
Two managers usually score the same call differently because the scorecard leaves room to interpret. Vague level descriptions, a strong or weak call outcome, and each manager's own strengths all pull scores apart. The two fixes are writing the reasons behind each level into the scorecard and running regular calibration sessions.
Is AI more consistent than human call reviewers?
A tested AI setup can score very steadily from one run to the next. The scorer in our final method changed 1.7% of its scores on a re-run, and the full method ranked calls like trained experts at 0.89. That re-run figure measures a different thing from two people disagreeing, so the two numbers do not compare directly. Untested models vary much more: one changed 15% to 17% of scores between identical runs.
How often should managers calibrate call scores?
Once a month works for most teams, plus a session whenever the scorecard changes or a new manager joins. Three to five calls, scored blind beforehand, fill an hour.
Should a call score depend on whether the deal closed?
No, a call score should not depend on whether the deal closed. Score the call on what the rep did and what the buyer said. A won deal can follow a weak call and a lost deal can follow a strong one, and mixing the two teaches reps the wrong lesson.