How does AI call scoring work?

AI call scoring works by transcribing a recorded call, having a language model judge each skill on a scorecard against the transcript, and returning a level for each skill with the quotes that justify it.

The outline every vendor gives

The outline is simple: transcribe the call, score it against a scorecard the manager sets, and hand the score to the manager as the start of a coaching conversation. Aircall describes the same three parts (Aircall). That outline is right, and it hides the part that decides whether you can trust the result, which is how the model turns a transcript into a level.

We spent August to October 2026 testing that step: more than 5,000 tests, more than 250,000 scoring decisions and 15 AI models, against blind marking by PeerLab's trained assessors (field guide). A single model reading the call and scoring each skill reached between 0.73 and 0.82 rank agreement with the experts, depending on the model. The design below ranked calls in the same order as the experts at 0.89, and its checklist model, the step that answers yes or no for each level, changed 1.7% of its scores on a re-run. The AI call scoring guide is the full treatment.

Inside the scoring step

Questions per level. For each of 20 skills, a checklist model answers yes or no to "is this at least level 2? 3? 4? 5?", 80 questions per call. Each question carries the reasons trained assessors gave for scoring a real call at that level. In our tests, adding those reasons was the largest single gain we measured, lifting average fit with human scores from 0.62 to 0.72.

Inside the scoring step

1. 01Transcribe the call
2. 02Questions per levelYes or no up each ladder, 80 per call
3. 03A second, blind judgeScores the whole call without the first answers
4. 04Average the rankingsInto the final scorecard
5. 05Check the feedbackEvery quote checked against the transcript

A second, blind judge. A different model reads the whole call and scores it without seeing the first model's answers. In our tests, showing a model another model's scores stopped it spotting the strongest calls.

Averaged rankings. The two rankings are averaged into the final scorecard.

Checked feedback. A model writes the feedback from the final scorecard, and every quote it uses is checked against the transcript before anyone sees it.

In our call-scoring research, all of it cost about $0.007 per call for scores and feedback; one pass of one of the leading general-purpose models cost 6 to 28 times more. It also held up on messy input, ranking at 0.79 to 0.83 with up to a third of the words dropped and speaker labels swapped.

Where it breaks

Bias does not go away because a model does the scoring. Hyperbound's FAQ says AI call scoring "eliminates bias" (Hyperbound). Our research found the opposite on three counts. Every newer model generation marked reps more harshly than the human assessors. Asked to compare two calls head to head, a model picked the first one it read 62% of the time. And when a cheap model extracted evidence quotes, a fifth of them were not in the call. Each of these is fixable, and none is visible unless you test for it. Human vs AI call scoring covers when each kind of scorer is the better choice.

Scoring one skill, step by step (illustrative)

The call and answers here are illustrative. On a discovery call the buyer says shipments run late and "it costs us." The rep asks how much; the buyer says "a lot," and the rep moves to the demo. The checklist model answers the impact questions: at least level 2, yes (the rep asked about cost); at least level 3, no (no figure or consequence was established). The blind judge reads the whole call and also puts impact low.

The final score is level 2, with the "a lot" exchange quoted as the evidence. The manager reads it in a minute, and before the follow-up the rep talks it through with Caty on a call to work out how to size the cost with the buyer's finance lead.

What AI call scoring can judge from a transcript

An AI scorer can judge anything that shows up in the words of the call, and it struggles with anything that does not. Most scorecard items fall into one of five kinds, and they are not equally easy to score.

Kind of item	Example	How well a model scores it
Yes or no	Did the rep confirm the next meeting with a date?	Reliably, if the transcript is complete
Count	How many open questions did the rep ask before the demo?	Reliably, though counts say little about quality
Buyer response	Did the buyer put a figure or a consequence on the problem?	Well, when the level descriptions name what the buyer says
Judgment	Did the rep connect the product to the buyer's stated cost?	Only with clear level descriptions and a second judge
Tone and delivery	Did the rep sound confident?	Poorly from text alone; audio helps, and managers disagree here too

Sales scorecards lean on the third and fourth rows. Most AI call scoring was built for contact centers first, where a support call is scored on compliance and resolution and a keyword or a checklist gets most of the way. A keyword match cannot tell a buyer's real answer from a polite one, so a compliance scorer can look accurate in a demo and miss most of what a sales manager cares about. Test any tool on your hardest skills, such as impact or gaining commitment.

How many skills, and how to weight them

A scorecard a manager fills in by hand works best with a handful of skills. Our research scorecard had 20, which an AI scorer can handle because it does not tire, but more skills only help if each one can be found in a transcript and has its own level descriptions.

Weighting matters more than the count. If ten easy items (greeting, agenda, recap) carry the same weight as finding out what the problem costs, a rep can score well on a call that left the deal no further on. Two ways to fix this:

• Report the critical skills separately. Show the skills that decide the deal (impact, decision process, next step) as their own line before any overall score.
• Weight by what the call type is for. On a discovery call, the questions that uncover the problem and its cost should outweigh delivery items. On a negotiation call, the commitment items should.

An overall score is useful for spotting trends. For coaching, the level on one skill with the quote behind it is what a rep can act on.

Setting up AI call scoring, in order

1. Choose one call type. Discovery is usually the one that matters most.
2. Write the scorecard in your method's terms, with a description for every level (what that takes is in the next section).
3. Check transcription. Speaker labels and complete calls come before anything else.
4. Run the ten-call check below against two managers.
5. Fix the scorecard where people disagree, then re-run.
6. Start with the 1:1. Use scores to pick which calls a manager hears, and widen use once managers trust them.

What the scorecard needs before a model can use it

The model can only be as precise as the scorecard it reads.

Skills a transcript can show. "Builds rapport" is hard to find in text. "Asks the buyer what the problem costs, and gets a figure or a consequence" can be found and quoted. Rewrite each skill as something the rep says or gets the buyer to say.

A description for every level. A five-level scale with only the top and bottom described leaves the model to guess the middle, and it will guess differently on different calls. Write what level 2, 3 and 4 look like, in one or two sentences each, with an example line where you can.

One scorecard per call type. A cold call, a first discovery call and a negotiation call ask for different things. Scoring a cold call on discovery skills marks every rep down for questions they had no reason to ask. Start with the call type that matters most to your pipeline, usually discovery, and build the others later. Our discovery call scorecard template is a starting point.

Why scores look wrong, and what to check

When reps or managers say the scores feel off, the cause is usually one of five things.

1. Everyone scores about the same. The level descriptions are too vague to separate a good call from an average one. Tighten the middle levels and re-score.
2. Half levels appear. A scale defined in whole levels should not produce a 3.5. In one production engine we tested in our call-scoring research, 37% of scores were fractional, and rounding alone added 16 points of exact agreement with the assessors. A half level is the model hedging; force a whole level.
3. Scores dropped across the team in one week. Check whether the vendor changed the underlying model. Newer models in our research marked reps harsher than the human assessors, so a model change can move every score with no change in the calls.
4. The quote does not match the score. If the feedback quotes a line that is not in the call, or quotes a line that supports a different level, the evidence step is failing. Check every quote against the transcript.
5. Some calls score far too low. Look at the transcript. Wrong speaker labels or a call that cut off early can make a good rep look silent. The design above held up on messy input, but a transcript missing most of the call cannot be scored fairly by anyone.

How to check a scorer on your own calls

You can test any AI scorer, ours included, in an afternoon with a set of calls your managers already know.

1. Pick ten calls of one type. Include a few strong, a few weak and a few in the middle.
2. Have two managers score them blind. Each uses the scorecard without seeing the other's marks or the AI's. Where they disagree, note it: that is the standard of agreement you can expect from people.
3. Rank the calls. Put the ten calls in order by the managers' average. Order matters more than any single number, because a manager deciding whom to coach is comparing calls.
4. Score the same ten with the AI. Compare its order with the managers'. Look at where it puts the strongest and weakest calls.
5. Score them again. Run the AI a second time on the same calls and count how many skill scores changed. A rep who can get a different score for the same call stops trusting all of them.
6. Check five quotes. Pick five pieces of feedback and find each quoted line in the transcript.

If the order is close, the re-run is stable and the quotes are real, the scorer is ready for the 1:1. If not, ask the vendor what they will change, and run the same ten calls again after they do. How to calibrate AI call scores covers this check in more depth.

The check, run on one team (illustrative)

The team and numbers here are fictional. Robin manages six account executives at a mid-sized software company and is about to roll out AI scoring on discovery calls. Robin and a peer, Sam, each score the same ten discovery calls blind.

They agree on most skills and split on impact for three calls: Sam gives credit when the rep asks about cost, and Robin only when the buyer gives a figure or a consequence. That disagreement is useful before any AI is involved. They settle it by rewriting the level 3 description for impact so it names the buyer's answer, and re-score those three calls.

Then the AI scores the ten. Its order is close to the managers' at the top and bottom, and it swaps two calls in the middle. Robin listens to both and sides with the AI on one of them. On the re-run, a handful of skill scores move, all by one level, all on calls the managers had also found hard to place. Four of the five quotes Robin checks are word for word in the transcript; the fifth is a paraphrase presented as a quote. Robin raises the paraphrase with the vendor as a fail on the quote check, and the rollout waits until it is fixed.

The afternoon cost two managers about three hours. It gave the team a tighter scorecard, a known standard of agreement, and one defect caught before reps saw it.

Running the design yourself

How to build an AI call scoring framework has templates for the level questions, the blind second judge and the quote check, and the AI call scoring guide covers how the scores feed coaching. To watch the level questions work on one of your own transcripts, talk to us.

How sales teams use AI call scoring

The scores are only worth having if they change what happens in the next meeting. These are the places they do.

The weekly 1:1. The manager opens the rep's scored calls from deals with a meeting coming up, picks the lowest skill that matters for that meeting, and plays the quote behind the score. One skill per week keeps the coaching specific.

Practice before the call. The rep runs a roleplay built from the live deal, scored on the same scorecard, so practice works on the skill the real call exposed.

The team view. Enablement looks at levels by skill and stage across the team to see where a method is slipping, for example impact questions in first discovery calls, and plans the next session around it.

In each case the scorer is checked against the team's own managers first. An AI sales expert built in your method, such as Keenan for Gap Selling or Val for the ValueSelling Framework®, scores against that method's own skills and talks the rep through the result on a call.

FAQ

What is AI call scoring software?

AI call scoring software transcribes recorded sales calls and scores each one against a scorecard, skill by skill, with quotes as evidence. Most tools also aggregate scores by rep and team. The useful ones let you check how closely the scores rank calls against your best managers and how much they change on a re-run.

Is AI call scoring accurate?

AI call scoring can be accurate, with the right design and a test on your own calls. Our final method ranked calls like trained assessors at 0.89, and its checklist model changed 1.7% of scores on a re-run. Single models with default prompts did worse, and the most expensive model we tested finished eighth.

Does AI call scoring replace managers listening to calls?

AI call scoring does not replace managers listening to calls. It covers the calls a manager cannot reach and points them at the ones worth hearing. The manager still decides what to coach, and their own scores are the standard the AI is checked against. The coaching happens before the next conversation on the deal.

Should AI call scoring use yes/no questions or a 1 to 5 scale?

Use both: a five-level scale, reached through yes/no steps. Asking "is this at least level 2? level 3?" in order gives the consistency of yes/no decisions and the detail of five levels. The most consistent method in our call-scoring research used that structure, 80 yes/no questions per call across 20 skills.

Can AI score calls in my sales methodology?

AI can score calls in a named methodology if the scorecard is written in that method's terms, with level descriptions for each skill. A generic scorecard labeled "MEDDPICC" will score generic things. Write each criterion the way your method defines it and check the scores against managers trained in it.