The finding
Every instruction about how to score made an AI call scorer agree less with trained human assessors. The prompt as shipped agreed with human scores at 0.62. Each variant that told the model how to mark landed between 0.44 and 0.49, according to our field guide Can AI judge a sales call?.
Editing the prompt also made the scorer less stable. The unedited prompt moved 9% of its scores between identical runs. Edited prompts moved 27% to 40%. The field guide's summary is short: the bias is trained in, and a prompt cannot take it out.
This piece is one finding from the sales coaching research hub.
The problem we were trying to fix
The research ran from August to October 2026: more than 5,000 tests, more than 250,000 scoring decisions and 15 AI models. Trained assessors from PeerLab marked real enterprise discovery roleplays blind against a 20-skill capability scorecard and never saw a machine's answer.
Newer models marked reps harsher than those assessors, and the gap grew with each release, down to −1.1 for GPT-6 Sol on a shared scale (newer AI models mark reps harsher). The obvious fix for a harsh scorer is to tell it how to score. The guide describes what happened as a seesaw: pull the marks closer to the experts and everyone bunches in the middle; push for a spread and the marks get harsher and further off.
Five fixes and what they did
| Fix we tried | What happened | Figure |
|---|---|---|
| Tell it how to score | Every variant that explained the marking agreed less with the experts | 0.44 to 0.49 agreement, against 0.62 for the original prompt |
| State the expected spread of scores | Ranking collapsed | Rank agreement 0.33 |
| "Use the full range" | It used less of the range | |
| Redesign the system prompt | Lost the top marks, and scores became unstable | Up to 40% of scores moved on a re-run; the unedited prompt moved 9% |
| Show it another model's scores | It stopped spotting the best calls, even with a warning | Recall of top-band calls fell from 45% to 5% |
Two more changes outside the prompt failed in the same way.
Rewriting the scorecard as "observable checks." The idea was to turn judgment into checkable behaviors. Its rank agreement, how closely it put reps in the experts' order, fell to 0.62. That is a ranking figure, separate from the 0.62 fit of the original prompt above. The guide's advice is to keep the scorecard's own words.
Keeping one long scoring conversation going. A scoring account with history drifted harsher within 18 hours. A fresh account fixed it. Every scoring call should start a fresh conversation.
Across all of them, more direction produced less agreement. The model's severity comes from training, and an instruction pulls on the wrong lever.
The re-run problem
Instability never shows on a dashboard. When the same calls are scored twice with nothing changed, some scores move. With the original prompt, 9% did. After an edit, up to 40% did.
For a sales team that means a rep's score can change between Monday and Tuesday on an unchanged call, because someone tidied the prompt. Trend lines built on that score track the prompt history as much as the rep. Any team that edits a scoring prompt and does not re-run its marked calls twice has no way to see this.
For scale, the final method in the guide changed 1.7% of its scores on a re-run, and two trained human markers disagreed with each other on 18% of the scores they both gave. A scorer can repeat itself perfectly and still be wrong, so stability is necessary and never sufficient.
What worked
Three changes improved scoring, and none of them was a prompt instruction.
Match the scale to how the experts mark. The experts never used the bottom score. Removing that level from the scorecard brought the AI's marks closer, and the gain held on a re-run. Change the scorecard when the experts' own use of it says to.
Correct a measured bias after scoring. If a scorer ranks well but marks consistently low, measure the fixed skew against human marks and apply an offset to the output. Then re-test. The guide's rule: correct a measured bias with a fixed offset, or change the scorecard, and if neither moves it, change the method.
Change the method. The final setup splits each judgment into narrow yes/no questions. A checklist model, TypeSafe's Jev, answers 80 per call ("did the rep reach at least level 2? 3? 4? 5?" for 20 skills), each carrying the reasons trained assessors gave that level on other calls. A blind GPT-6 Luna pass scores the whole call separately, and the rankings are averaged. That reached 0.89 rank agreement with the experts. The reasons alone lifted average fit with human scores from 0.62 to 0.72, the biggest single gain in the study (assessors' reasons beat textbook examples).
The method mattered as much as the input. Given the same reasons, GPT-6 Sol ranked reps at 0.73, exactly where it started. The guide's templates are short on purpose. Its recommended blind Luna template is six lines: a role, the instruction to score the call against the scorecard, the scorecard, the transcript, whole numbers 1 to 5, JSON only.
How to change a scorer safely
Change a scorer safely by testing every edit against your own calls, marked blind by people, before it reaches a rep.
- Keep a blind-marked set of your own calls. People who never see an AI score mark them, with a one-line reason per mark.
- Score the set twice before any change. Record ranking against the human order and the share of scores that move between the two runs. That is your baseline.
- Make one change. A prompt edit, a scorecard edit or a model swap. One at a time, so you know which one moved the numbers.
- Score the set twice again. Compare ranking, re-run movement and bias against the human marks. Check the strongest and weakest calls separately.
- Keep the record. Store the transcript, scorecard, model version, prompt and date with every run, so a change in a rep's score can be traced to a change in the rep or a change in the instrument.
If the prompt edit lowers ranking or raises re-run movement, revert it, however sensible it reads.
A worked example
This example is illustrative; the team and figures are invented to show the procedure.
An enablement lead sees her scorer giving almost nobody a 5 on gaining commitment. She adds a line to the prompt: "Award 5 when the rep secures a dated next step with the decision maker." It reads like a sensible fix. Before rolling it out she scores her marked set twice with the old prompt and twice with the new one.
The new prompt does hand out more 5s. It also reorders the middle of the team, and a fifth of scores move between the two identical runs, against a handful before. Two reps who were clearly stronger than a colleague on the human marks now sit below him. She reverts the edit, measures how far below the human marks the scorer sits on that skill, and applies a fixed offset to the output. The 5s come back and the order holds.
Why this matters to a manager
A rep's score is the opening line of a coaching conversation. If it moved because a prompt changed, the conversation starts wrong, and a rep who notices will stop trusting the next one. When a rep talks a live deal through with one of our AI sales experts on a call (typing is the fallback), the expert holds them to the team's standard and works out the next conversation with the buyer. The experts are benchmarked against the leading AI models and retested whenever a model is released or retired.
So treat a prompt edit to a live scorer like a model swap: re-run it on blind-marked calls, several times, and keep it only if the ranking improves. Related: average error flatters a lazy scorer and AI picked the first call it read 62% of the time. The research is discussed in Episode 01 of Show Your Working.
FAQ
Can you fix AI call scoring bias with a better prompt?
In our tests, no. Every prompt variant that told the model how to mark agreed less with trained human assessors, landing between 0.44 and 0.49 against 0.62 for the original. Stating the expected spread pulled ranking to 0.33. A fixed offset after scoring, a scorecard change or a different method worked better.
Why do AI call scores change when I edit the prompt?
Prompt edits make scoring less stable. In our study the unedited prompt moved 9% of scores between identical runs, while edited prompts moved 27% to 40%. Re-score a blind-marked set twice after any edit and compare ranking and movement before trusting the new scores.
How should I prompt an AI to score sales calls?
Keep it short and keep the scorecard's own words. The guide's recommended blind scoring template gives a role, the scorecard, the transcript, and asks for whole numbers 1 to 5 as JSON. Use a fresh conversation for every call. Accuracy gains came from adding trained assessors' reasons to narrow yes/no questions.
Why does my AI scorer get harsher over time?
A long-running scoring conversation can drift. In our tests a scoring account with history drifted harsher within 18 hours, and a fresh account fixed it. Model upgrades also move severity: newer generations marked reps harsher than human assessors. Start every scoring call in a fresh conversation and re-test after any model change.
Should I show the AI examples of other scores?
Do not show an AI scorer another model's scores. When we showed a model another model's scores in our call-scoring research, its recall of top-band calls fell from 45% to 5%, even with a warning not to anchor on them. Keep scorers blind to each other. Reasons trained assessors gave on other calls helped; numbers from another scorer hurt.