Sales coaching research: call scoring and kickoff findings

Sales coaching research is evidence about whether coaching, training and the tools around them change what reps do on live deals. This hub collects the studies Replicate Labs has published under Show Your Working, with the numbers, the method behind each one and the practical point for a sales manager.

What the research covers

Show Your Working

Can AI judge a sales call?0.89 rank agreement
Newer models mark harder
Measure a scorer on ranking
Prompt fixes and position bias
The experts' reasonsFit from 0.62 to 0.72
What reps say after a kickoff
What the learning science says

Show Your Working

Can AI judge a sales call?0.89 rank agreement
Newer models mark harder
Measure a scorer on ranking
Prompt fixes and position bias
The experts' reasonsFit from 0.62 to 0.72
What reps say after a kickoff
What the learning science says

Two studies anchor it. The first, Can AI judge a sales call?, is a field guide from a research program that ran from August to October 2026: more than 5,000 tests, more than 250,000 scoring decisions and 15 AI models, checked against blind marking by trained assessors from PeerLab. The second, Game Winning Kickoffs, read 11,112 posts, reviews and pages from public sources in October 2026 to find out whether anyone could show a sales kickoff changed what people did afterward. A third, the benchmark we use to judge our own AI sales experts, is summarized below. The call-scoring study is discussed on camera in Show Your Working; the kickoff study follows on 11 November.

The term used throughout for a model that grades another model's output, or a person's, is LLM-as-a-judge.

Findings at a glance

Finding	Number	How it was measured	Source
Final scoring setup matches the experts' order of calls	0.89 rank agreement	Checklist model (yes/no questions per level) plus a blind second model, against blind PeerLab marks	Field guide
Scores that change when the same calls are scored again	1.7%	Checklist-model re-run on the same calls	Field guide
Trained assessors disagreeing with each other	18% of shared scores	Two experts marking the same calls	Field guide
The August "winner", run ten more times	0.55 average, single runs 0.41 to 0.75	Ten repeats on the same calls	Field guide
Model picks the first of two calls it reads	62% of the time	Head-to-head comparisons	Field guide
Average error and ranking	Correlated at 0.02	34 scoring configurations	Field guide
Fit with human scores after adding experts' reasons	0.62 to 0.72	Average across 20 skills	Field guide
First-hand kickoff accounts describing something used afterward	9 of about 4,800	Public and anonymous accounts, hand-checked	Kickoff study
Reps criticizing their own kickoff	0% in public, 38% anonymously	760 reps on LinkedIn; anonymous reviews	Kickoff study
Share of minutes given to learning content	29% in the guides' agenda templates, 11% in real agendas	22 templates and 79 real agendas	Kickoff study
Studies comparing kickoff attendees with non-attendees	0 of 21	Every effect claim the study found	Kickoff study

Can AI judge a sales call?

Yes, with the right setup, and most setups fall short. Our final method put calls in the same order as trained human assessors with 0.89 rank agreement and changed 1.7% of its checklist-model scores on a re-run, according to the field guide.

The study scored real enterprise discovery roleplays against a 20-skill capability scorecard, marked blind by assessors who never saw a machine's answer. Every setup was judged three ways: ranking (does it put calls in the experts' order), spread (does it tell a strong call from a weak one) and consistency (score the same call twice, get the same answer).

The headline changed. In August the best configuration gave the model the client's context, calibrated its scores and averaged two models. Run ten more times on the same calls, it averaged 0.55 on ranking, with single runs between 0.41 and 0.75. The production engine it was meant to beat already sat at 0.74. We retracted the headline.

Consistency is only part of the picture. Two trained assessors disagreed on 18% of the scores they both gave, and the checklist model disagreed with itself on 1.7%. A scorer can repeat an answer perfectly and still be wrong, so the guide also compares the blend with a second human assessor, from calls two experts marked independently: 0.89 against about 0.95 on ranking, 86% against about 81% of the strongest calls found, and 71% against about 86% of the weakest. The weakest-call gap matters most to a manager deciding who needs help first.

The calibrated results cover one scorecard and one cohort.

Further reading: Can AI score a sales call? and the practical guide, the AI call scoring guide.

Newer models mark harder

Each newer model generation we tested marked reps harder than the last, and from GPT-5.1 on every model marked below the human experts. The most expensive model in the study finished eighth.

The field guide plots bias against human marks by model, where negative means harsher: GPT-4.1 at +0.18, GPT-5.1 at −0.13, GPT-5.4 at −0.65, the GPT-5.6 family at −0.75 to −1.09, Claude Sonnet 5 at −1.01 and GPT-6 Sol at −1.1. On ranking, GPT-6 Sol scored 0.73, with three runs between 0.67 and 0.78, and 15% to 17% of its scores changed between identical runs. The production engine, GPT-5.1, scored 0.74. GPT-6 Luna, the cheapest current model tested, scored 0.77 to 0.82.

For a buyer the practical point is that a model upgrade changes the instrument. A harsher scorer redefines what a strong call looks like, and a team that swaps models without re-testing on the same marked calls loses its baseline. OpenAI's deprecations page, checked on 21 October 2026, lists GPT-5.1 as deprecated on 1 October 2026 and removed from the API on 1 April 2027, so this is a live problem for anyone building on it.

Further reading: Newer AI models mark sales reps harsher than human assessors.

Measure a scorer on ranking

Average error is the number most vendors quote, and on its own it tells a buyer very little. In our call-scoring research, a scorer that never read the transcript and gave every call the same score landed 0.41 from the experts on average, while the best real model landed 0.43.

Across 34 configurations in the field guide, average error and ranking correlated at 0.02. They measure different properties. Ranking asks whether the scorer puts calls in the experts' order, which is what a manager uses to decide whose call to review. Exact agreement can mislead too: in one production engine, 37% of scores were fractional, a 2.7 can never exactly match a human 3, and rounding alone added 16 points of exact agreement without changing a single judgment.

Further reading: Average error flatters a lazy call scorer.

Prompt fixes and position bias

Telling a harsh model how to score made it worse in our tests. Asking it to compare two calls head to head exposed a different bias: it picked the first call it read 62% of the time.

The field guide lists the fixes that failed. A redesigned system prompt lost the top marks, and up to 40% of scores moved on a re-run, where the unedited prompt moved 9%. Rewriting the scorecard as "observable checks" ranked reps at 0.62. "Use the full range" made it use less of the range. Showing it another model's scores stopped it spotting the best calls, even with a warning. One long scoring conversation drifted harsher within 18 hours, and a fresh account fixed it. The one change that held was matching the scale to how the experts mark: the experts never used the bottom score, and removing it brought the AI's marks closer on a re-run.

The guide's rule that follows: correct a measured bias with a fixed offset after scoring, or change the scorecard itself, and if neither moves it, change the method.

Further reading: Five prompt fixes that made AI call scoring worse and AI picked the first call it read 62% of the time.

What the experts' reasons changed

The biggest single gain in the study came from people. Giving the scorer the reasons trained assessors wrote for each level on other calls lifted average fit with human scores from 0.62 to 0.72.

A scorecard line such as "Explores trade-offs explicitly" does not tell a model what an assessor rewards. One expert's reason for that level read: "Surfaces budget source, CFO gate and decision timeline." The field guide reports the skills that moved most: strategic negotiation from 0.42 to 0.66, personas and drivers from 0.35 to 0.66, frames impactfully from 0.30 to 0.54, emotional control from 0.43 to 0.61 and gaining commitment from 0.64 to 0.83.

Written textbook examples added nothing. The method mattered too: given the same reasons, GPT-6 Sol ranked reps at 0.73, exactly where it started. The reasons helped the checklist model, which answers narrow yes/no questions.

The final setup puts both lessons together. The checklist model (TypeSafe's Jev) answers 80 yes/no questions per call ("at least level 2? 3? 4? 5?" for 20 skills, each carrying the experts' reasons). GPT-6 Luna scores the whole call blind and never sees Jev's answers. The rankings are averaged, then Luna writes the feedback from the final scorecard and every quote is checked against the call. The cost is about $0.007 per call for scores plus written feedback.

Further reading: Assessors' written reasons beat textbook examples.

What reps say after a kickoff

In public, almost nobody criticizes their own kickoff, and very few name anything they learned. Of about 4,800 first-hand accounts in the kickoff study, 9 people describe using something from the kickoff afterward and 3 tie it to a result.

The kickoff study read 4,057 LinkedIn posts about sales kickoffs. On their own company's event, 0% of 760 reps said anything negative, along with 1% of 172 front-line managers and 0% of 585 sales leaders. Energy and motivation came up in 44% of posts, strategy in 30%, meeting colleagues in 29% and awards in 25%; skills training appeared in 8% and one specific thing learned in 5%.

A learning ladder scored 3,582 public posts about the writer's own kickoff. Level 0, nothing about learning: 37%. Level 1, "learned so much" with no content: 25%. Level 2, names a topic: 32%. Level 3, names a change or was assessed: 5%. Level 4, used it afterward: 9 people by hand count. Of the 341 posts written weeks or months later, 5 describe using something.

Anonymous accounts tell a different story. Across 774 accounts from Glassdoor, RepVue, Reddit and Blind, 38% were negative about the kickoff itself and 33% positive; where the kickoff was the main point of the review, 59% were negative, and 92% said nothing about learning.

Further reading: What 4,057 LinkedIn posts say reps took from their kickoff, What 774 anonymous reviews say about sales kickoffs and the practical guide, the sales kickoff field study.

What the learning science says

Nobody has studied the sales kickoff directly, but decades of training research describe events shaped like one. Enjoying training correlates .02 with learning from it, and what happens back at work predicts whether a skill gets used.

The kickoff study checked each paper by its DOI. Usefulness ratings did better than enjoyment, at .26 (Alliger et al. 1997). Spaced practice beat massed practice across 63 studies, d 0.46 (Donovan & Radosevich 1999; Cepeda et al. 2008). A week later, people tested on material remembered 56% and people who re-read it 42% (Roediger & Karpicke 2006).

Skill loss reached d −1.4 after more than a year without use (Arthur et al. 1998). Across 117 studies, skills held when people worked on their own situations and their managers were trained too (Taylor, Russ-Eft & Chan 2005). Across 1,246 pharma reps, managers' coaching skill predicted goal attainment, and frequent coaching from less skilled managers was linked to worse results (Dahling et al. 2016). Saks & Belcourt (2006) and Blume et al. (2010) found the work environment before and after training more strongly tied to use than the room itself.

The papers agree on one thing a manager can use. The skill has to be used on the rep's own deals, soon and repeatedly, with someone who knows how to coach it. That is the gap we call performance drift, between what a rep learned and what they do on a call three weeks later (know-do gap vs performance drift).

Further reading: Seven learning-science findings every sales manager can use and the sales coaching framework.

The ROI nobody has measured

Nobody has published a comparison of kickoff attendees with non-attendees. The kickoff study found 21 studies, surveys and case studies claiming an effect, and none compared the two groups on win rate, quota attainment, revenue, ramp time or retention.

The only formal ROI calculation found for a sales-type conference, a 2005 franchise event measured with the Phillips ROI Methodology, came out at minus 53%, on a small sample with an author who expected a loss. Spending is better documented: median spend of $1,000 to $1,499 per rep (ATD 2023, 71 respondents), 24% of planners with ROI metrics in policy (Amex GBT 2026 forecast), and ServiceNow's 10-Q for Q1 2023 attributing a $19 million cost increase "primarily due to increased program costs and travel for our annual Sales Kickoff".

In the kickoff study, the 22 agenda templates in the guides give about 29% of kickoff minutes to learning content; real company agendas give it about 11%.

Further reading: Sales kickoff ROI: 21 studies claim an effect, none compare attendees.

How we judge our AI sales experts

We judge our experts against cases written as real requests, criteria set by methodology experts, and blind pairwise judging by more than one judge. We then tune the coach against that standard and re-test it.

The judging question is whether the rep or manager is likely to act on the answer. Each pair of answers is judged blind; the field guide showed a model judge favors whichever answer it reads first, so order matters. The criteria come from the people who wrote the method where we build with them (ASG for Gap Selling, ValueSelling Associates for the ValueSelling Framework®) and from clients' own playbooks, and they get sharper as those experts add cases. Models change every few weeks; the standard and the re-testing are what hold a coach to the bar when a new one ships.

The headline result: our flagship experts ranked first against the newest models from Anthropic, OpenAI and Google in 120 sales situations, across four jobs (call reviews and deal strategy, finished work a seller can send, deal and pipeline reports, manager coaching), winning 79% of 3,840 blind AI-judge votes (20 October 2026, the benchmark). Per-case scores stay private. A separate benchmark of language models writing sales emails is at the email LLM benchmark.

Further reading: How we benchmark AI sales coaches and the AI sales coaching guide.

Worked example: one deal, three findings

This example is illustrative. The company, rep and deal are invented to show how the findings apply.

One deal, three findings

01A score starts the conversation2 on gaining commitment
02The expert's reason carries the meaningA dated next step with the budget owner
03Use it on the live deal, this weekTwo questions planned on a call
04ThursdayThe buyer names the CFO and agrees a date

Priya and Halden Freight are invented to show how the findings apply.

Priya is an account executive at a mid-market software company. Her team came back from its January kickoff with a new discovery standard. Three weeks later she runs a discovery call with a fictional prospect, Halden Freight, a 40-seat opportunity.

Finding 1: a score starts the conversation. Her call is scored against the team's scorecard. She gets a 2 on gaining commitment. Her manager has already checked that the scorer ranks the team's calls in the order its own assessors would and holds on a re-run, so the 2 is worth talking about.

Finding 2: the expert's reason carries the meaning. "Gaining commitment, level 2" means little to Priya. The assessors' reason does: at level 3 on this team's scorecard, assessors reward a rep who agrees a dated next step with the person who owns the budget. Priya's call ended with "I'll send over some info."

Finding 3: use it on the live deal, this week. The kickoff taught the standard. The learning science says it fades unless used on her own situation, soon. So the debrief happens on a call: Priya talks the Halden deal through with Caty, who asks what the buyer said about budget, notes that nobody has named who signs, and works out with her the two questions for the follow-up call and the dated next step to ask for. If Priya cannot talk at that moment she types, and Caty picks up the same deal.

Later that week she asks both questions. The buyer names the CFO and agrees a date for a three-way call. That is the behavior the kickoff was meant to start, used on a live deal within a month.

What the research adds up to

The studies point the same way. A score measures a call and a kickoff starts a change, and neither alters what a rep says to a buyer on its own. That happens when someone works the rep's real deal with them, soon after the lesson and repeatedly, against a standard the team trusts.

Our AI sales experts are built for that job: Keenan (Gap Selling), Val (the ValueSelling Framework®) and Caty (any other method) work the rep's live pipeline, and Bryce does the same with the manager. To see what the Halden debrief feels like on one of your own deals, talk to us.

FAQ

Is there research on whether AI can score sales calls accurately?

Yes: our 2026 field guide tested 15 AI models in more than 5,000 tests against blind marking by trained PeerLab assessors. The final setup reached 0.89 rank agreement with the experts' order of calls and changed 1.7% of checklist-model scores on a re-run. Single models with default prompts did worse, and the calibrated results cover one scorecard and one cohort.

Do sales kickoffs change rep behavior?

Very few people can show that they do. Our kickoff study read about 4,800 first-hand accounts and found 9 people describing something they used afterward, and 3 who tied it to a result. No study we found compared attendees with non-attendees. The learning science suggests the change depends on what happens back at work in the weeks after.

What is the best way to measure an AI call scorer?

Measure ranking against blind human marks, and check both ends of the range. In our call-scoring study a scorer that never read the transcript beat the best real model on average error, 0.41 to 0.43, and average error correlated with ranking at 0.02 across 34 configurations. Also check how often it disagrees with itself on a re-run.

Are newer AI models better at scoring sales calls?

Newer AI models are not better at scoring sales calls by default. Each newer model generation we tested marked reps harsher than the last, moving from +0.18 for GPT-4.1 to −1.1 for GPT-6 Sol against human assessors, and the most expensive model finished eighth. GPT-6 Luna, the cheapest current model tested, ranked best on its own at 0.77 to 0.82. Re-test on the same marked calls before any upgrade.

What does learning research say about sales training?

It says enjoyment does not predict learning and that spacing, testing and use on real work do. Enjoyment correlated .02 with learning; spaced practice beat massed practice at d 0.46 across 63 studies; testing beat re-reading 56% to 42% a week later. Managers' coaching skill predicted results across 1,246 pharma reps, according to the papers our kickoff study checked.

How do you benchmark an AI sales coach?

We use cases written as real requests, criteria set with the method owners we build with (ASG, ValueSelling Associates) or from the client's own playbook, and blind pairwise judging by more than one judge. The test is whether the rep or manager would act on the answer. We tune the coach against that standard and re-test it.

Where can I watch the research discussed?

On Show Your Working, a monthly series where James Pursey and Greg McCallum of PeerLab test one claim about AI in sales in public. Episode 01, on call scoring, aired on 21 October 2026. Episode 02, Game Winning Kickoffs with Celeste Berke Knisely of A Sales Growth Company, goes live on Wednesday 11 November 2026.

Sales coaching research: what our call-scoring and kickoff studies found