What a call scoring rubric is
A call scoring rubric is the set of skills a call type should show, with a written description of each level of each skill, so that different scorers can rate the same call the same way. Building one takes an enablement lead and two or three experienced managers about two weeks of short sessions, most of it spent scoring real calls and writing down why.
- 01Pick one call typeAnd write its job in one sentence
- 02Choose the skills from the jobOne behavior per skill
- 03Fix five whole levelsLevel 1 absent, level 5 expert
- 04Collect your managers' reasonsOn 10 to 15 real calls
- 05Write each level from the reasonsAnd define not applicable per skill
- 06Test, then versionOne master version and a change log
A rubric is the part of a scorecard that does the work. A list of skills with a 1 to 5 scale beside each is only a form; the level descriptions decide whether one manager's 3 means the same as another's. Without that consistency, a rep hears two different verdicts on the same call and stops trusting either. This piece is about how to get the consistency; the wider picture is in the AI call scoring guide.
The method draws on our research program (August to October 2026: more than 5,000 tests and more than 250,000 scoring decisions, with blind marking by PeerLab's trained assessors against a 20-skill scorecard). Several findings there were about rubric design, and they are built into the steps (field guide).
How to build a call scoring rubric in seven steps
1. Pick one call type and write its job
A rubric for "sales calls" fits none of them. Pick one: first discovery, demo, renewal, negotiation. Write one sentence on what that call type should produce for a deal. Good looks like: "A first discovery call should leave the rep with a confirmed problem, a sense of its size, the people in the decision, and an agreed next step."
2. Choose the skills from the job
List the behaviors that produce what the call should produce. Keep each skill to one behavior; "discovery and listening" is two skills and will get two half-scores. For a coaching scorecard reps will read, six to ten skills is enough; a twenty-item grade gets skimmed. A measurement instrument can carry more: the research rubric we used to test AI scorers has 20, listed in the discovery call scorecard template. Good looks like: every skill traces back to a part of the call's job.
3. Fix five whole levels
Level 1 is absent, level 5 is expert. Decide now that scores are whole numbers. In one production engine we tested in our call-scoring research, 37% of scores were fractional on a whole-level scale, and rounding alone added 16 points of exact agreement with the assessors. A half level is a scorer who has not decided. Good looks like: the scoring form has no way to enter 3.5.
4. Collect your managers' reasons on real calls
Take 10 to 15 recorded calls of the type. Have two or three managers score each skill independently and write a sentence on why, in their own words, for every score they had to think about. These sentences are marker reasons. In our call-scoring research, giving the AI scorer the trained assessors' marker reasons lifted average fit with human scores from 0.62 to 0.72, the largest single gain we found. Textbook descriptions of good behavior added nothing. Good looks like: a pile of short, specific sentences tied to real moments ("the buyer said 'sounds reasonable' and the rep moved on without a date").
5. Write each level from the reasons
Group the reasons by skill and level. Write each level description as what the transcript shows, using the managers' phrasing. Keep two or three of the original reasons under each level as examples. Where the managers disagreed, get them on a short call, settle it, and write down the reason they agree on; that sentence marks the boundary between two levels. Good looks like: each level reads like a description of a call someone on the team has heard.
6. Define not applicable for each skill
Write the trigger that makes a skill scorable: "an objection is raised", "price is discussed", "a second stakeholder is mentioned". Without a trigger there is no score. Leave N/A skills out of averages. Good looks like: no rep loses points on handling objections because the buyer had none.
7. Test, then version
Score a fresh set of calls with two managers and compare. Trained expert assessors in our call-scoring research still disagreed on 18% of scores, so expect some disagreement and look for the skills where it clusters. Rewrite those levels from the new reasons. Then freeze the rubric with a version number and a change log, and hold a calibration session whenever it changes. Good looks like: one master version for every team, and a record of what changed and why.
Worked example: one skill from scratch
This example is illustrative; the team and calls are invented.
An enablement lead builds a renewal-call rubric. One skill is gaining commitment. Two managers score twelve renewal calls and write reasons:
- "Rep asked if they were happy to renew. Customer said probably. Nothing scheduled." (Manager A, level 2)
- "Agreed to 'reconnect in a few weeks', no date." (Manager B, level 2)
- "Got a date for the renewal review. Didn't ask who from procurement attends." (Manager A, level 3)
- "Date, attendees and what the CFO needs to see, all agreed." (Manager B, level 4)
- "Customer proposed the meeting, named who they'd bring, and asked for the usage report in advance." (Manager A, level 5)
The managers split on one call: the customer agreed to a date but said "I'll check who else should come." Manager A gave it 3, Manager B gave it 4. On a ten-minute call they agree that level 4 needs named attendees, and write that as the boundary.
The finished skill:
| Level | Description | Example reason |
|---|---|---|
| 1 | No next step raised | "Call ended on pleasantries." |
| 2 | Next step proposed without a date or owner | "Agreed to 'reconnect in a few weeks', no date." |
| 3 | Date agreed; attendees or purpose left open | "Got a date. Didn't ask who from procurement attends." |
| 4 | Date, named attendees and purpose agreed | "Date, attendees and what the CFO needs to see, all agreed." |
| 5 | Customer drives the step and adds to it | "Customer proposed the meeting and asked for the usage report." |
N/A when: never. Every renewal call should end with a step.
For an AI scorer, the same skill becomes four yes/no questions: is this at least level 2? 3? 4? 5? Our most consistent research setup worked exactly this way, 80 yes/no questions per call across 20 skills, each carrying the assessors' reasons.
Where it goes wrong
Rewriting the levels as checklists. Turning descriptors into tick-boxes looks objective. When we rewrote the scorecard as "observable checks", ranking agreement with the assessors fell to 0.62, against 0.74 for the production engine the research started from. The checks were easy to apply and missed what the experts were judging.
Writing from a textbook. Descriptions borrowed from a methodology book read well and moved nothing in our tests. Your managers' sentences about your calls did.
Instructions in place of examples. Telling a scorer to "use the full range" made a model use less of it. Anchor each level with real cases from your own calls and leave the instructions out.
Editing in place. A rubric edited by each team drifts until scores across teams cannot be compared. Keep one master and a change log.
Changing the rubric without re-checking the AI. Any edit changes what an AI scorer does. A redesigned system prompt in our call-scoring research moved up to 40% of scores on a re-run. Re-run your calibration calls after every rubric change; the method is in How to calibrate AI call scores.
Using the rubric on every call
A finished rubric lets managers score consistently, but managers can only score a sample. Applied by a calibrated AI scorer, the same rubric covers every call: the final method in our call-scoring research reached 0.89 rank agreement with the assessors and changed 1.7% of scores on a re-run, at about $0.007 per call. The full template set for that loop is in How to build an AI call scoring framework.
The rubric is the standard, and coaching happens around it. Our AI sales experts carry the scorecard into the coaching call: the rep brings a live deal to Caty, or to Keenan or Val on a Gap Selling or ValueSelling Framework® team, talks through the last conversation, and the expert holds the rep to the rubric your managers wrote while working out the next step on that deal. How scorecards are configured is in the scorecard docs. The twelve-skill general version is the sales call scorecard. Talk to us.
FAQ
How do you build a call scoring rubric?
Pick one call type and write its job, choose six to ten skills that produce it (more only for a research instrument), and fix five whole levels. Have experienced managers score real calls and write why, then build each level description from their reasons. Define when each skill is not applicable, test with two scorers, and version the result.
How many levels should a sales call rubric have?
Five whole levels work well: enough to rank calls, few enough to describe each one. Our research scorecard used five, reached through yes/no questions ("at least level 2? 3?"). Avoid half levels; in one production engine from our call-scoring research, 37% of scores were fractional, and rounding them added 16 points of exact agreement.
What makes a good call scoring criterion?
A good criterion describes one behavior, with each level written from real calls in your managers' words. In our research, assessors' reasons lifted fit with human scores from 0.62 to 0.72, while textbook examples added nothing and a rewrite into "observable checks" dropped rank agreement to 0.62, against 0.74 for the production engine. Each criterion also needs a rule for when it does not apply.
Should a call scoring rubric be weighted?
Weighting helps only when you report a total, and totals hide the skill that matters on a given deal. Score and report each skill separately. If you need one number to compare calls of the same type, average the scorable skills and keep the skill scores beside it for coaching.
How often should a call scoring rubric be updated?
Update it when a calibration session settles a disagreement, when the sales motion changes, or when a skill comes back not applicable on most calls. Version every change and keep one master copy. If an AI scorer uses the rubric, re-run your calibration calls after each update.