Inter-rater agreement is how closely two or more scorers, human or AI, give the same scores to the same sales calls on the same scorecard.
Why it matters for coaching
If two managers score a call differently, a rep's score depends on who marked it, and coaching built on it inherits the noise. Teams usually measure agreement two ways: the share of scores where scorers match, and rank agreement, which asks whether scorers put a set of calls in the same order. Rank agreement matters most for coaching, because managers act on order.
In our 2026 call-scoring research, trained expert assessors marking the same calls disagreed on 18% of scores. Our final AI method reached 0.89 rank agreement with the experts' order and changed 1.7% of its own scores on a re-run (field guide). The 18% is the useful number for any team: it is roughly how much disagreement to expect from good human scorers before anyone has done anything wrong.
For an AI scorer, add a second test: agreement with itself across identical runs. One model on its own, GPT-6 Sol, changed 15% to 17% of its scores between runs. Avoid average error as the measure. In the same research, a scorer that never read the transcript landed 0.41 on it, against 0.43 for the best real model, so a low average error can come from guessing the middle of the scale.
Two managers, twelve calls
Two managers score twelve discovery calls blind. They match on most scores but rank three of the middle calls in different orders on stakeholder mapping. In a short meeting they settle the boundary between level 2 and level 3 and write it down (illustrative): "Level 3 means the rep asked who else signs off. Level 2 means the buyer volunteered it and the rep moved on." When the team then checks an AI scorer, it compares the AI's ranking with that settled order. The procedure is How to calibrate AI call scores.
Related terms
- Calibration session: the meeting that raises agreement
- Call scoring: the activity being measured
- Human vs AI call scoring: what the blind test found
FAQ
What is a good inter-rater agreement for sales call scoring?
Use your own experts as the bar. Trained assessors in our research disagreed on 18% of scores. An AI scorer's re-run change rate should sit well under that figure; our final method changed 1.7% of scores on a re-run.
What is the difference between inter-rater agreement and inter-rater reliability?
Agreement asks how often scorers give identical scores. Reliability statistics such as kappa adjust that figure for agreement that would happen by chance. For coaching, rank agreement covers what managers act on: whether scorers put calls in the same order.
How do you improve inter-rater agreement?
Write level descriptions from your experienced managers' real reasons for real scores. In our research those reasons lifted fit with human scores from 0.62 to 0.72. Score whole levels, score the whole call, and hold regular calibration sessions.