What calibrating an AI scorer means
Calibrating AI call scores means checking that a model scores your calls the way your most trusted managers would, then adjusting it until it does and holds there. It takes one enablement lead or RevOps owner, two trusted managers and roughly a working day spread over a week, and it needs repeating whenever the model, prompt or scorecard changes.
- 01Two managers score blind12 to 20 calls, whole levels, with reasons
- 02Run the AI twiceBlind to the managers' scores
- 03Compare on three measuresRank agreement, re-run change, human disagreement
- 04Fix with your managers' words
- 05Lock it, re-check on triggersA new model, prompt or scorecard
- 01Two managers score blind12 to 20 calls, whole levels, with reasons
- 02Run the AI twiceBlind to the managers' scores
- 03Compare on three measuresRank agreement, re-run change, human disagreement
- 04Fix with your managers' words
- 05Lock it, re-check on triggersA new model, prompt or scorecard
Most guidance on calibration comes from contact centers, where it means getting human QA reviewers to agree with each other. That still matters. An AI scorer adds a second problem: the instrument itself can drift, and it drifts silently. Vendor pages tend to answer "can AI score a call accurately?" with a yes and no number behind it. A figure is what calibration produces.
The method here comes from our research program (August to October 2026: more than 5,000 tests, more than 250,000 scoring decisions, 15 models, blind marking by PeerLab's trained assessors). It is part of the AI call scoring guide, and the full findings are in the field guide.
The three measures
Calibration compares three numbers. Collect all three every time. The benchmark figures in this guide come from our research, Can AI judge a sales call?
| Measure | Question | How to compute | Reference point from our research |
|---|---|---|---|
| Rank agreement | Does the AI put calls in the same order as your managers? | Rank calls by each scorer's total for a skill, then correlate the ranks (spreadsheet: RANK, then CORREL) | Final method 0.89; production baseline 0.74 |
| Re-run change | Does the AI agree with itself? | Score the same calls twice, nothing changed; count the scores that differ | Final method 1.7%; GPT-6 Sol alone 15% to 17% |
| Human disagreement | How far apart are your managers? | Count the scores where the two managers differ | Trained experts: 18% |
Leave average error out. In our call-scoring research a scorer that never read the transcript, giving every call the same middling score, landed an average error of 0.41. The best real model landed 0.43. Across 34 configurations average error and ranking correlated at 0.02. The property being measured is inter-rater agreement, applied to a model as one of the raters.
How to calibrate AI call scores in seven steps
1. Build a calibration set
Choose 12 to 20 recorded calls of one type (first discovery calls, say) from different reps. Include strong, weak and awkward calls: a buyer who changed direction, a call that ended early. Easy calls produce easy agreement and teach nothing. Good looks like: a set where your managers would struggle to agree on the order of the middle third.
2. Have two managers score blind
Each scores every call alone on your scorecard, in whole levels. Ask each to write a sentence or two on why, for any score they had to think about. Those sentences are marker reasons, and the later steps are built on them. Good looks like: two complete scorecards per call, with reasons, and no conferring.
3. Measure the human baseline
Count where the two managers differ. Then get them on a call to settle the biggest disagreements and record the reason they agree on. If they differ on more than about a fifth of scores, the scorecard descriptors need work before any model is measured against them; the calibration session is the meeting for that. Good looks like: a settled reference scorecard per call and a disagreement figure you can quote.
4. Run the AI twice
Score the same calls with the AI scorer, then again an hour later with nothing changed. Keep the model blind to the managers' scores; when we showed a model another scorer's results, it stopped spotting the best calls. Good looks like: two full AI runs, stored next to the reference scorecards.
5. Compare on the three measures
Compute rank agreement against the reference, re-run change between the two AI runs, and the direction of any gap skill by skill. Every newer model generation in our research marked reps harsher than the human assessors, so check direction as well as size. Good looks like: a one-page table: rank agreement per skill, re-run change, harsher or softer per skill.
6. Fix with your managers' words
Add the managers' marker reasons to each skill's level descriptions and re-run. In our call-scoring research, reasons carried into yes/no level questions lifted average fit with human scores from 0.62 to 0.72, the largest single gain; textbook examples of good behavior added nothing.
Reasons alone did not help a model scoring the whole call (GPT-6 Sol stayed at 0.73); they worked when paired with yes/no level questions, the structure in the worked example below. Change one thing per run. A redesigned system prompt moved up to 40% of scores on a re-run, against 9% for the unedited prompt, and an instruction to "use the full range" made the model use less of it. Good looks like: each change logged with the before and after figures.
7. Lock it and set re-check triggers
When the scorer's ranking sits close to the reference and its re-run change is low enough that a rep's score would not flip between runs, freeze the model version, prompt and scorecard. Re-run the calibration set whenever any of the three changes. Dates are already on the calendar: OpenAI deprecated GPT-5.1 on 1 October 2026 for removal on 1 April 2027, and GPT-4o, 4o mini, 4.1 and 4.1 mini go on 14 January 2027 (checked 21 October 2026). Good looks like: a named owner, a frozen configuration and a short list of triggers.
Worked example
This example is illustrative; the team and figures are invented to show how to read the results.
A RevOps lead calibrates a scorer on 16 first discovery calls. The two sales managers differ on about one score in six, so their scorecard is usable. The AI's first two runs differ from each other on more scores than the managers do, and its ranking on gaining commitment sits well away from both. It is also harsher on that skill than either manager.
The lead adds the managers' marker reasons for gaining commitment, written as they gave them ("Level 3: the buyer said 'sounds reasonable' and the rep moved on without confirming who would attend"). The re-run shows the ranking on that skill moving toward the reference, and the harsh gap narrowing. Re-run change is still high, so the lead splits each skill into yes/no level questions ("at least level 2? level 3?"), the structure our final method used. The next pair of runs rarely disagree, and the ranking holds close to the reference. The configuration is frozen with a re-check scheduled before the model's removal date.
Where it goes wrong
Calibrating on easy calls. Agreement on obvious calls says nothing about the middle of the distribution, where coaching decisions sit.
Trusting one good run. Our August configuration looked like a winner on one run and averaged 0.55 across ten, with single runs from 0.41 to 0.75. Always run twice before believing a number.
Fixing by prompt. Rewriting the scorecard as "observable checks" ranked 0.62, against 0.74 for the production baseline and 0.89 for the final method. Prompt fixes moved scores in ways nobody predicted; your managers' reasons moved them toward the reference.
Calibrating once. A long-running scoring conversation in our tests drifted harsher within 18 hours. Models, prompts and scorecards all change, and the output gives no sign when they do.
Ignoring fractional scores. In our call-scoring research, 37% of one production engine's scores were fractional on a whole-level scale. Rounding alone added 16 points of exact agreement. Force whole levels before measuring anything.
Keeping every rep's scores calibrated
The final method from our research holds these properties by design: 80 yes/no level questions per call, asked by a model that carries the assessors' marker reasons, a blind second judge, rankings averaged, and every feedback quote checked against the transcript. It reached 0.89 rank agreement and 1.7% re-run change at about $0.007 per call. The four templates to run the same loop yourself are in How to build an AI call scoring framework, and the evidence in short form is in Can AI score a sales call?.
Once the scorer passes, the score can start a coaching conversation. When a rep brings a live deal to one of our AI sales experts on a call, the expert scores the call on your scorecard and holds the rep to the standard your managers set. Configuration is described in the scorecard docs. If you want to see what a scored call looks like before your managers mark a single test set, talk to us.
FAQ
How do I calibrate AI call scoring?
Score a set of 12 to 20 calls with two trusted managers working blind, settle their disagreements into a reference, then run the AI scorer twice on the same calls. Compare rank agreement with the reference and re-run change between the two AI runs. Fix gaps by adding the managers' marker reasons to the level descriptions, changing one thing at a time.
How accurate should AI call scores be before reps see them?
The scorer should give the same scores when you run it twice, and rank calls close to the managers' settled order. In our call-scoring research, the final method reached 0.89 rank agreement and changed 1.7% of scores on a re-run, so both are reachable targets.
Why are my AI call scores harsher than my managers'?
Newer models tend to mark harder. In our call-scoring research every newer model generation scored reps more harshly than the human assessors, and the gap grew with each release. Check direction skill by skill during calibration, then add your managers' marker reasons to the descriptors, which moved scores toward the human reference more than any prompt change.
How often should AI call scores be recalibrated?
Re-run your calibration set whenever the model, prompt or scorecard changes, and on a fixed schedule besides. Model retirements force changes: OpenAI removes GPT-4o, 4o mini, 4.1 and 4.1 mini on 14 January 2027 and GPT-5.1 on 1 April 2027. A long scoring conversation in our tests drifted harsher within 18 hours.
Does a better prompt fix inconsistent AI call scores?
A better prompt usually does not fix inconsistent scores. A redesigned system prompt in our call-scoring research moved up to 40% of scores on a re-run, against 9% for the unedited prompt. Asking the model to "use the full range" made it use less of the range. Assessors' marker reasons and splitting each skill into yes/no level questions were what made scores consistent.
What is the difference between call calibration and AI calibration?
Call calibration gets human reviewers to agree on how to apply a scorecard. AI calibration adds the model as a reviewer and also checks that it agrees with itself across runs. Run the human session first, because the managers' settled scores and reasons are what the model is measured and corrected against.