Rank agreement is a measure of how closely a scorer puts a set of calls in the same order, best to worst, as a trusted reference such as expert reviewers.
Testing a scorer by its order
Score a set of calls with the scorer you want to test, whether a new manager or an AI model, and with your reference reviewers. Rank the calls by each set of scores, then correlate the two rankings; Spearman's rank correlation is the usual method. A result of 1 means identical order, 0 means no relationship.
It matters because coaching decisions are about order: which rep, which call and which skill sits furthest below the standard. Average error misses this.
In our research program (August to October 2026, more than 5,000 tests and more than 250,000 scoring decisions, with blind marking by PeerLab's trained assessors), a scorer that never read the transcript and gave every call the same middling score had an average error of 0.41; the best real model had 0.43. Across 34 configurations, average error and ranking correlated at 0.02, which means a lower error told you nothing about the order (field guide). Our final setup reached 0.89 rank agreement with the experts; the production engine it replaced sat at 0.74. The AI call scoring guide covers the wider topic.
Ten calls in a spreadsheet
A sales director tests an AI scorer on ten discovery calls that the team's two best managers have already ranked. In a spreadsheet the director ranks the calls by each total with RANK, then runs CORREL on the two rank columns, which gives Spearman's correlation when there are no ties. The director runs the AI twice more, because single runs vary: an early setup in our research averaged 0.55 rank agreement across ten re-runs, with single runs anywhere from 0.41 to 0.75. The AI call scoring framework has templates for the full test. To run it on your own calls, talk to us.
- 01Score the same ten callsWith the AI and with your best managers
- 02Rank each set of totalsRANK in a spreadsheet
- 03Correlate the two rankingsCORREL on the rank columns gives Spearman
- 04Run the AI twice moreSingle runs vary
Related terms
- Call scoring: rating a call against a scorecard, skill by skill
- Inter-rater agreement: how closely two scorers land on the same scores
- Calibration session: the meeting that brings scorers into line
FAQ
What is a good rank agreement score for AI call scoring?
Aim for the level your own experts reach with each other. Our final method reached 0.89 against trained assessors, and a production engine sat at 0.74. Check across several runs, since a single run can look far better or worse than the average.
Why not judge a call scorer on average error?
Average error is a poor test because a lazy scorer can win on it. In our tests a scorer that never read the transcript had almost the same average error as the strongest real model. Ranking separates them, and ranking is what coaching decisions depend on.
Is rank agreement the same as Spearman correlation?
Rank agreement is usually computed as Spearman's rank correlation, though the two are not strictly the same thing. Other rank measures exist, but for a manager testing a scorer on ten to twenty calls, Spearman in a spreadsheet is enough.