What a call calibration session is for
A call calibration session is a meeting where sales managers who score calls each mark the same recordings alone, compare their scores skill by skill, and agree rulings that make the next round of scores line up. An enablement or RevOps lead runs it with three to six managers. Plan on an hour of scoring per manager beforehand and a 90-minute session.
Almost everything written about calibration comes from contact centers, where the steps start from QA analysts and support supervisors. Sales calls bring their own problems. There is rarely a script, the buyer changes direction, and the skills that matter most (impact, stakeholder mapping, negotiation) depend on judgment. That makes calibration more important for sales teams, and it changes what the session needs to produce. See the calibration session glossary entry for the short definition and the AI call scoring guide for the system it belongs to.
The method draws on our call-scoring research: August to October 2026, more than 5,000 tests and more than 250,000 scoring decisions, with blind marking by PeerLab's trained assessors (field guide).
What to measure
Use two numbers, and compute both per skill.
| Measure | How to compute | Reference point |
|---|---|---|
| Disagreement | Share of scores where two managers gave different levels for the same skill on the same call | Trained experts in our research disagreed on 18% of scores |
| Rank agreement | Does each manager put the calls in the same order? Rank each manager's totals per skill and compare the orders | The useful signal; close order means the scorecard is doing its job |
A total-score variance is the usual contact-center target, and it hides most of the problem. As Yoodli's guide puts it, "A small gap on the total can hide large disagreements on specific behaviors." Two managers can land on the same total for a call while disagreeing on half the skills.
Running the session
1. Choose calls that will cause disagreement
Pick five to eight recorded calls of one type, say first discovery calls, from different reps. Include calls where the buyer pushed back, changed direction or ended early. Easy calls produce easy agreement and teach the group very little. Good looks like: a set where you cannot predict how the managers will order the middle calls.
- 01Choose calls that split the roomFive to eight calls of one type
- 02Every manager scores blindWhole levels, a quote on the hard ones
- 03Lay out the spreadBefore anyone talks
- 04Evidence before numbersRead the quote, then the level
- 05Write the ruling into the levelOne new sentence per disputed skill
- 06Re-score one fresh callThen log the rulings and tell the reps
2. Have every manager score blind
Each manager scores every call alone, on the current scorecard, in whole levels, before the session. For any score they hesitated over, they write a sentence on why and copy the quote that decided it. Good looks like: a full grid of whole-number scores, with reasons on the hard ones, and no manager has seen anyone else's.
3. Lay out the spread before anyone talks
The facilitator puts every manager's scores side by side, skill by skill, and computes the two measures. Sort skills by disagreement. Hide names if seniority will stop people from disagreeing. Good looks like: a one-page grid with the three skills that split the room at the top.
4. Discuss evidence before numbers
Start with the skill that split the room most. Ask each manager to read the quote they scored from, then explain the level. Keep the conversation on what the rep said and what the buyer said back. Good looks like: managers find they heard different moments, or heard the same moment and applied different standards. Both are fixable.
5. Make a ruling and write the reason into the level
For each disagreement, the group agrees the level and why. Write that reason into the scorecard as a marker on the level: "Level 3 requires the buyer to state the cost in their own numbers; a rep's estimate the buyer nods at is level 2." In our research, giving our yes/no scoring model the reasons trained assessors gave for each level lifted its average fit with human scores from 0.62 to 0.72, the largest single gain we measured; GPT-6 Sol given the same reasons stayed at 0.73. Textbook examples of good calls added nothing. Good looks like: each disputed skill leaves the room with one new sentence in its level descriptions.
6. Re-score one fresh call
Before the session ends, everyone scores one call nobody has seen, using the updated descriptions. Compare quickly. Good looks like: fewer disagreements on the skills you just ruled on. If they persist, the wording is not clear yet.
7. Log the rulings and tell the reps
Keep every ruling in one shared document, with the call and quote it came from. Send reps the changed descriptions, so the standard they are coached against is the one managers use. Good looks like: a rep can read why a level 2 is a level 2.
Four managers, six calls (illustrative)
The team, calls and figures are illustrative. Four managers score six first discovery calls on a six-skill scorecard: 144 scores in all. The facilitator's grid shows the managers disagree most on impact and stakeholder mapping, and they rank call 4 anywhere from second to fifth.
On impact, the quote is the buyer saying "the delays are costing us real money." Two managers scored level 3 because the rep followed up; two scored level 2 because no figure came out. The group rules level 2 and writes the marker: "Level 3 needs a figure or a consequence the buyer states." On stakeholder mapping, the split comes from a rep who asked "who else is involved?" and got "just me for now." The ruling: asking the question is level 2; level 3 needs the rep to test the answer, for example by asking who signs contracts of this size. The fresh call in step 6 produces one disagreement on impact out of four managers.
The next week, each manager takes the impact ruling into their deal reviews. A rep with a discovery call scored at level 2 on impact talks the deal through with the manager on a call and leaves with the question that gets the buyer's own figure.
Where it goes wrong
Talking about the numbers first. Once a senior manager says "I gave it a 4," others move toward it. Anchoring hits models too: in our research, showing a model another model's scores stopped it spotting the strongest calls. Read quotes first, then reveal levels.
Easy calls. Five clear calls produce agreement and change nothing in the scorecard.
Arguing about the outcome. A call that won the next meeting gets generous scores. This is outcome bias. Score the skill, then note the outcome separately.
Half scores. A 2.5 avoids the ruling the session exists to make. In our research 37% of one production engine's scores were fractional on a whole-level scale, and rounding alone added 16 points of exact agreement. Insist on whole levels.
Rulings that stay in the room. If the reason does not go into the level description, the next new manager scores the old way.
How often to calibrate
Monthly suits most sales teams once a scorecard is stable. Add a session when the scorecard changes, when a new manager starts scoring, when the sales motion changes (a new product or price), and when your AI scorer's model changes. Model changes are coming on fixed dates: OpenAI removes GPT-4o, 4o mini, 4.1 and 4.1 mini on 14 January 2027 and GPT-5.1 on 1 April 2027 (field guide).
Carrying the rulings to every call
A session calibrates the managers in the room, on the calls they scored. To hold the same standard on every call, run your AI scorer on the session's calls, compare its ranking with the managers' settled ranking, and add the session's marker reasons to its level descriptions. How to calibrate AI call scores has the steps, and Can AI score a sales call? shows what our final blended setup reached in our call-scoring research: 0.89 rank agreement with experts, with 1.7% of its scores changing on a re-run.
With our AI sales experts, the rulings carry into coaching too. The calibrated scorecard sits under every coaching call, so when a rep talks a live deal through with Caty, the rep is held to the level descriptions your managers just rewrote. If you want help running the first session against your own calls, talk to us.
FAQ
How long should a call calibration session take?
About 90 minutes for five to eight calls, plus an hour of blind scoring per manager beforehand. Most of the session goes on the two or three skills with the widest disagreement. If the session runs long every time, the call set is too big or the scorecard has too many skills for that call type.
Who should attend a sales call calibration session?
Every manager who scores calls, plus a facilitator from enablement or RevOps who does not score. Three to six scorers works best; more than that and the discussion stalls. Invite a top rep occasionally to hear the rulings, since the standard is theirs to meet.
What is an acceptable level of disagreement between call reviewers?
Expect some disagreement: trained experts in our research disagreed on 18% of individual scores, so a team of managers starting out will usually disagree more. Watch the trend per skill and whether managers put calls in the same order. Rank order matters more than matching every individual score.
How do I calibrate an AI call scorer with my managers?
Run the AI scorer on the calls from your calibration session, compare its order of calls with the managers' agreed order, and add the managers' marker reasons to the scorecard it uses. Re-run it twice on the same calls to check it agrees with itself. Change one thing at a time.
What is the difference between calibration and a call review?
A call review is one manager coaching one rep on one call. Calibration is several managers scoring the same calls to agree how the scorecard is applied. Calibration makes call reviews fair, because every rep is held to the same reading of each level.