What an AI call scoring framework is
An AI call scoring framework is the set of fixed artifacts that let a language model score sales calls the way your trained experts would, and let you prove that it does. It has four parts: a case set of real calls with an expert ranking, a criterion card per skill with marker reasons (the assessors' short written notes on why a real call sat at a level), a judging protocol that fixes how the model is asked and how its answers are checked, and an improvement log that re-runs the whole thing on a schedule and records what moved. The model sits inside it and can be swapped.
- Case setReal calls with an expert ranking attached
- Criterion cardOne per skill, with marker reasons
- Judging protocolHow the model is asked and checked
- Improvement logOne row per run, the only way in for a change
- Case setReal calls with an expert ranking attached
- Criterion cardOne per skill, with marker reasons
- Judging protocolHow the model is asked and checked
- Improvement logOne row per run, the only way in for a change
This is the structure we built over more than 5,000 tests and more than 250,000 scoring decisions across 15 models between August and October 2026, with blind human marking from PeerLab. The end state was 0.89 rank agreement with the experts' order and 1.7% of checklist-model scores changing on a re-run, at about $0.007 per call including written feedback. The full record is in the field guide Can AI judge a sales call?. This piece is the build.
The model is the smallest part
Every model has a removal date. As of 21 October 2026, OpenAI lists GPT-5.1 as deprecated on 1 October 2026 and removed on 1 April 2027; GPT-4o, GPT-4o mini, GPT-4.1 and GPT-4.1 mini go on 14 January 2027; fine-tuning winds down from 6 January 2027. A scoring setup that is a prompt and a model name is a configuration with an expiry date.
The swap is never neutral. In our call-scoring research every newer model generation marked reps more harshly than the human assessors, and the gap widened with each release. The most expensive model finished eighth. GPT-6 Sol ranked at 0.73 and moved 15% to 17% of its scores between identical runs. A framework exists so that when the model changes, you re-run the case set the same night and know by morning what the new model did to your scores.
Template 1: the case set
The case set is a fixed collection of real calls with an expert ranking attached. Everything else is measured against it.
| Field | What goes in it | Why |
|---|---|---|
| Calls | Recorded calls from one stage and one segment (discovery is the usual start) | Mixed stages make skills incomparable across calls |
| Transcripts | Verbatim, speaker-labeled, unedited | The final method held at 0.79 to 0.83 with a third of words dropped and labels swapped, so imperfect transcripts are survivable; edited ones are not comparable |
| Expert scores | Two or more trained assessors, scoring independently in whole levels on every skill | In our call-scoring research, trained experts disagreed on 18% of scores; two assessors give you that baseline |
| Expert ranking | The calls ordered best to worst per skill, from the averaged expert scores | Ranking is the target; average error correlated with ranking at 0.02 across 34 configurations |
| Marker reasons | Each assessor's note, verbatim, on why a call sat at its level | Feeds Template 2; the single largest gain in our research program |
| Holdout | A fifth of the calls never used for tuning | Lets you detect a change that fits the tuning set and nothing else |
A case set with no expert ranking is a pile of calls. The ranking costs money and time, and it is what makes every later number mean something.
Template 2: the criterion card
One card per skill. The research scorecard carries 20 skills; the structure is identical at twelve.
| Field | Content |
|---|---|
| Skill | Name and one-sentence definition |
| Levels | 1 to 5, each with a written descriptor of what that level looks like in a transcript |
| Marker reasons | Up to five verbatim assessor notes per level, taken from other calls in the case set |
| Threshold questions | Four yes/no questions: is this call at least level 2? at least level 3? at least level 4? at least level 5? |
| Applicability | The condition under which the skill is scored at all; otherwise N/A |
| Evidence rule | Every cited quote must appear verbatim in the transcript |
The threshold questions are the mechanism. A small checklist model answers 80 of them per call, four per skill for 20 skills, each carrying the marker reasons for that level. Asking "is this at least a 3?" with the assessors' own words for what a 3 looks like is a smaller and more stable decision than "score this 1 to 5". In our tests, adding marker reasons lifted average fit with human scores from 0.62 to 0.72. Textbook examples of good behavior added nothing. The skills that moved most were framing, gaining commitment, emotional control and negotiation.
Two things to leave off the card. A line telling the model to "use the full range" made it use less of the range. Another model's scores shown as a reference stopped it spotting the best calls.
A worked example: the card for gaining commitment
Take an illustrative company, Quarrendon Freight, whose mid-market reps run discovery calls on a route-planning product. The assessors start with gaining commitment, because it is the skill the managers argue about most. Their card looks like this:
| Field | Quarrendon's entry |
|---|---|
| Skill | Gaining commitment: the call ends with a specific next step the buyer has agreed to |
| Level 2 | The rep suggests a follow-up and leaves the buyer to propose a time |
| Level 3 | A dated next step is agreed, with no stated purpose or named attendees |
| Level 4 | A dated next step with a purpose the buyer has put in their own words |
| Level 5 | Level 4, plus the buyer has agreed to bring the other people who must be there |
| Marker reason, level 3 | "Got the Tuesday slot but never asked who signs off on routing changes" |
| Marker reason, level 4 | "Buyer said the next call is to see whether it fits their depot rota. That is the buyer's goal, in the buyer's words" |
| Threshold questions | Is this call at least level 2? 3? 4? 5? |
| Applicability | Scored when the call reaches the close; a call the buyer cuts short is N/A |
| Evidence rule | The quote showing the agreed step must appear verbatim in the transcript |
Now one call. Jordan, a Quarrendon rep, ends a discovery call with Robin, the operations lead at a regional grocer, by saying "Shall we pick this up next week?" Robin replies "Send me something and I'll find a time." The checklist model answers yes to "at least level 2" and no to "at least level 3", because nothing dated was agreed. The call sits at level 2 and the cited quote is Robin's reply.
The feedback Jordan reads names that moment and carries the level 3 marker reason, so the next step is plain: get a date before hanging up, then ask who else needs to be in the room. When the assessors and the model disagree on a call like this, the first place to look is the marker reasons for the two levels in question. The prompt is the last.
Template 3: the judging protocol
The protocol fixes how the model is asked, what is hidden from it, and what is checked before a score is accepted. It is the part most teams never write down, and it is where most of the drift came from.
| Rule | Setting | Figure behind it |
|---|---|---|
| Two passes | A checklist model answers the threshold questions; a second model scores the whole call blind | Rankings averaged across the two reached 0.89; either alone was lower |
| Blind | The second judge never sees the first pass | Showing a model another model's scores hid the best calls |
| Whole call | Score the full transcript as one unit | Single-turn scoring ranked 0.57 to 0.63 |
| Fresh context | Every call scored in a new conversation | A long-running scoring conversation drifted harsher within 18 hours |
| Order | Pairwise comparisons, when used, run both orders | In our research, the model picked the first call it read 62% of the time head to head |
| Whole levels | Scores shown to reps as whole levels; continuous scores kept for ranking | In our research, 37% of one production engine's scores were fractional; rounding alone added 16 points of exact agreement |
| Quote check | Every quote in feedback matched to the transcript before release | A fifth of cheaply extracted quotes were absent from the call |
| Prompt freeze | The system prompt changes only through the improvement log | In our research, a redesigned prompt moved up to 40% of scores on a re-run and the unedited prompt moved 9% |
The feedback a rep reads is written by the second judge from the final scorecard, after the quote check. How scorecards are set up on our platform is in the scorecard docs.
Template 4: the improvement log
The log is a table with one row per run, and the only place a change is allowed to enter.
| Column | Content |
|---|---|
| Run date and time | When the run started, for matching to model releases |
| What changed | Model version, prompt edit, new marker reasons, new cases, or "nothing" for a control run |
| Rank agreement | Correlation with the expert ranking, per skill and overall |
| Re-run change | Percentage of scores that differ from the previous run with identical settings |
| Harshness | Mean model score minus mean expert score, per skill |
| Quote errors | Quotes in feedback that are absent from the transcript, as a percentage |
| Holdout agreement | Rank agreement on the holdout fifth only |
| Decision | Keep, revert, or investigate |
The control run is the row people skip. Our August "winner" posted a strong single result; run ten times unchanged it averaged 0.55 with single runs from 0.41 to 0.75, against a production engine at 0.74. We retracted the headline. A change that cannot be told apart from run-to-run noise is noise.
Running the loop overnight
The loop runs on a schedule and is read in the morning.
- Freeze the case set and the cards at the start of the week.
- Each night, run the full case set through the protocol twice: once with any proposed change, once with nothing changed.
- Write both rows into the log. Rank agreement, re-run change, harshness, quote errors, holdout.
- Apply the thresholds in the morning. Keep a change if rank agreement rises on the holdout, re-run change stays under 5%, and harshness does not grow. Revert if any one of those fails. Investigate if the two control runs disagree with each other by more than they did last week.
- On a model swap, run three nights of control before any other change.
- Each quarter, add 10 to 15 new cases with fresh expert scores and marker reasons, and retire the same number of the oldest.
The whole run costs about $0.007 per call for scores and feedback with the two-pass method; a single pass of a top general-purpose model costs 6 to 28 times more and ranked worse.
Real-time scoring or post-call scoring?
Post-call scoring is the one to build on, because whole-call scoring is the setting that held up. Single-turn scoring ranked 0.57 to 0.63 in our tests, well below the two-pass method. A live call has no whole yet. Prompts shown during a call, such as a reminder to ask about budget, are a different tool with a different job, and they should never be reported as a rep's level on a skill.
Use the post-call score for coaching and for the improvement log. If the team also wants live prompts, keep them out of the case set and out of the log, so a change to the prompts cannot move the numbers you rely on.
Rolling the scores out to reps
Reps trust a score when they can see why it landed where it did. Show three things with every score: the level, the quote from their own call that decided it, and the assessor's marker reason for the next level up. A bare number starts an argument. A quote with a reason gives the rep something to change on the next call.
Start with one skill and one team. Run the framework for a few weeks on the calls the managers already review, and put the model's score next to the manager's in the 1:1. Where the two differ, the manager reads the quote and decides, and the disagreements become new marker reasons on the card. Widen to the next skill after that.
Keep scores out of compensation and league tables until the log shows a stable run of control nights. Reps notice a score that moves when nothing changed long before managers do, and the trust it costs is slow to rebuild.
Where the score goes
The framework produces a score a team can trust, with the moment on the transcript that decided it and the assessor's reason for the level. The rep still has the next conversation with that buyer, and the score on its own does not change what they say in it.
On our platform the rep takes the deal behind the score to Caty, our deal expert. The rep brings the deal on a call, talks through what the buyer said, and Caty works the deal: what the buyer agreed to, what is missing and what to ask for when the buyer is back on the line. She runs the team's own method on that deal (MEDDPICC or the team's own playbook; Gap Selling teams work with Keenan and ValueSelling Framework® teams with Val) and remembers every earlier conversation about it. A rep who cannot talk can type. A manager gets a Monday read on which deals need attention this week from Bryce, our manager coach.
The framework keeps that expert honest when a model ships or is deprecated. Models change; the case set, the cards, the protocol and the log are the part a team owns. Talk to us.
FAQ
What is an AI call scoring framework?
An AI call scoring framework is the fixed set of artifacts that lets a language model score sales calls the way your trained experts would, and lets you prove it. It has four parts: a case set with an expert ranking, a criterion card per skill with marker reasons, a judging protocol, and an improvement log. The model sits inside it and can be swapped.
How do I calibrate AI call scoring to my own team?
Build a case set of real calls scored independently by two trained assessors, keep their written reasons for each level, and attach those reasons to the criteria. In our research, marker reasons from assessors lifted fit with human scores from 0.62 to 0.72, the largest single gain. Then measure rank agreement against the experts' order, with average error kept as a second number.
How many criteria should an AI scoring rubric have?
Our research scorecard uses 20 skills, each with four yes/no threshold questions (at least level 2, 3, 4, 5), which is 80 decisions per call. Twelve skills with the same structure is a workable start. The count matters less than whether each level carries marker reasons from real calls and the scorer records whole levels.
Should I use the newest AI model for call scoring?
Use it only after it passes your case set, because every newer model generation in our call-scoring research marked reps more harshly than the human assessors. The most expensive model finished eighth. On a model swap, run the case set unchanged for three nights and read every column of the log, rank agreement first, before changing anything else.
How often should an AI call scoring framework be re-run?
Nightly, with a control run alongside any proposed change, and read in the morning. Our August result looked strong on one run and averaged 0.55 across ten; the control row is how that gets caught. Add fresh expert-scored cases each quarter and run three control nights after any model change.
How much does it cost to run AI call scoring with a validated framework?
About $0.007 per call for scores plus written feedback using a small checklist model for the threshold questions and one pass of a cheap model, GPT-6 Luna, for the blind whole-call score. A single pass of a top general-purpose model on its own costs 6 to 28 times more and ranked worse in our tests. Two runs a night come to about $0.014 per call.
What happens when my AI scoring model is deprecated?
Re-run the case set on the replacement with nothing else changed and read the log. As of 21 October 2026, OpenAI has GPT-5.1 removed on 1 April 2027 and the GPT-4o and 4.1 families on 14 January 2027. A framework with a frozen case set and an improvement log survives the swap; a prompt and a model name do not.