The short answer
We judge our AI sales experts against cases written as real requests from reps and managers, using a written standard for each job, built with our partners where a partner owns the method. Answers are compared in pairs, blind and in both orders, by more than one judge, and the question each judge answers is whether the rep or manager would act on the answer. We benchmark the experts against the leading AI models, tune them on what we find, and retest whenever a model is released or retired.
This piece describes the method in principle. The headline result is on our homepage: our flagship experts ranked first against the newest models from Anthropic, OpenAI and Google in 120 sales situations, across four jobs (call reviews and deal strategy, finished work a seller can send, deal and pipeline reports, manager coaching), winning 79% of 3,840 blind AI-judge votes (20 October 2026). We publish one worked case on the homepage; per-case scores and the case library stay private. The public numbers on how we test come from our call-scoring field guide, Can AI judge a sales call?, and each rule below exists because that research caught a way for an AI judge to go wrong.
More of our research is collected in sales coaching research. The buyer's version, for teams evaluating a vendor, is how to benchmark an AI sales coach.
Why a feature grid cannot judge a coach
A feature grid cannot judge a coach, because two coaches can list the same features and give advice of very different quality. Most evaluation guides for sales tools compare platforms: integrations, content management, analytics, who to include in the buying committee. Those matter, and none of them says whether the coaching is good.
Models are commodities. Every vendor can buy the newest one the day it ships. What separates two coaches is the standard they are held to and how they are improved against it. Our call-scoring research showed how little a model swap guarantees: each newer generation we tested marked reps harder, and the most expensive model finished eighth (newer AI models mark reps harsher). A coach running on a new model needs the same re-testing.
Cases
A case is a request a rep or manager would make, written the way they would make it, with the context they would have. Three invented examples: "My champion went quiet after the pricing call and the quarter ends in three weeks." "Write the QBR summary for this account." "Which of my reps needs help on discovery this week, and on what?"
Cases cover the jobs the coach does: coaching a rep on a live deal, producing finished work, reporting to managers and coaching managers. Each is tagged with the situations that make sales hard, so the benchmark shows where a coach is weak as well as how it does on average. Some cases are written so the right answer backs the rep; a coach that criticizes by reflex fails them. Some carry planted errors the coach should catch.
A written standard for each job
Each job has a written standard: what a good answer does, stated plainly enough that a judge can check an answer against it. Where a partner owns the method, the standard is built with them, so a Gap Selling case is judged the way A Sales Growth Company coaches and a ValueSelling Framework® case the way ValueSelling Associates does. The two manager jobs, reports and manager coaching, have no method owner, so we write those standards ourselves.
The call-scoring research is why. A scorecard line on its own did not tell a model what the experts reward. Adding the reasons trained assessors gave for each mark lifted average fit with human scores from 0.62 to 0.72, the biggest single gain in that study, and written textbook examples added nothing (assessors' written reasons beat textbook examples). That was a finding about scoring calls, and we carry the lesson over to coaching answers: write down the reasons, in the method's own terms, because a label alone leaves the judge guessing.
Criteria are stated as they are. When an expert sharpens one, the new version is the standard from then on, and every candidate is judged against it.
Blind pairwise judging
There is no fixed scorecard for a good answer to an open coaching question, so we compare. Two answers to the same case go to a judge, which picks the better one against the criteria.
Three rules keep that comparison fair, and each traces to a measured failure.
| Rule | Why | Evidence from our research |
|---|---|---|
| Every pair is judged in both orders | Model judges favor whichever answer they read first | Asked which of two sales calls was stronger, a model picked the first one it read 62% of the time |
| Judges are blind to which coach wrote which answer | Context bends a judge's verdict | Shown another model's scores, a scorer's recall of top-band calls fell from 45% to 5% |
| More than one judge, and none judges its own model's answer | A single judge's tastes become the standard | Every model generation carried its own severity bias against the human marks. Keeping a model off its own answers is a design rule; we have not measured self-preference |
The evidence column comes from the field guide. The first rule is covered in depth in AI picked the first call it read 62% of the time. The general technique is LLM-as-a-judge.
The test: would they act on it?
Each judge answers one question: which answer is the rep, or the manager, more likely to act on? A coaching answer that is correct and ignored has not coached anyone.
That test rewards specific next steps on the real deal, in the rep's language, and it penalizes generic advice, lectures and answers that restate the question. It does not penalize depth. A manager report that goes deep on the right deals is judged on whether the manager would act on it, and length is no reason to mark it down.
Test, tune, retest
The benchmark is half the job. The other half is improving the experts against it. We benchmark them against the leading AI models, tune them on what we find, and retest whenever a model is released or retired.
- 01BenchmarkAgainst the leading AI models
- 02Tune on what we findOne change at a time
- 03Run it more than onceOne good run proves little
- 04RetestWhen a model is released or retired
- 01BenchmarkAgainst the leading AI models
- 02Tune on what we findOne change at a time
- 03Run it more than onceOne good run proves little
- 04RetestWhen a model is released or retired
A result has to hold up on more than one run. The call-scoring research retracted its August headline after the "winning" setup, run ten more times, averaged 0.55 on ranking with single runs from 0.41 to 0.75. One good run proves little.
The standard stays fixed while the models underneath change, and the experts are re-tuned to meet it.
What a benchmark cannot tell you
A benchmark like this answers one question well: across the cases in the library, which coach gives the answer a rep or manager is more likely to act on. Some things sit outside it, and anyone reading a result, ours included, should keep them in mind.
It ranks two coaches; it does not grade one. In our 20 October 2026 benchmark, winning 79% of 3,840 blind AI-judge votes means our experts' answers were picked more often in those comparisons. It is a statement about the pairs that were judged, so read it next to the answers themselves.
The judges are AI models. They work to the written standard for each job, and the rules above exist because AI judges fail in known ways. Reading a sample of the answers yourself is still a useful check on any judge.
A result describes a date. It covers the coaches and models as they were then. A new model release, ours or anyone else's, can move it, which is why we retest whenever a model is released or retired.
Answers are only part of coaching. A rep's experience also depends on the conversation: how the coach asks, listens and follows up when the rep talks a deal through on a call. Listen to recorded calls on real deals to judge that part.
Your deals are not our cases. The library is built around the methods our partners own. If your team sells a different way, run the smaller version below on your own pipeline before you trust anyone's headline, including ours.
What we publish and what we do not
We publish the method in principle, here, and the call-scoring numbers in the field guide: more than 5,000 tests, more than 250,000 scoring decisions and 15 models, with blind marking by PeerLab. Beyond that headline result and the one worked case on the homepage, we do not publish per-case scores or the case library. The criteria rest on methods our partners own, and the case library is built with them.
Running a smaller version yourself
A team evaluating any AI coach, ours included, can run a small version in a week. Keep the usual checks on integrations and security; this adds the step a feature grid misses, judging the advice itself.
- Write ten cases from your own pipeline. Real deals, real requests, in your reps' words. Include one where the rep did well, to see if the coach can say so.
- Get your method owner to write the criteria. One short paragraph per job: what would a good answer do?
- Collect two coaches' answers to each case. Strip any branding.
- Have two managers judge each pair in both orders, without knowing which coach wrote which. Ask one question: which would the rep act on?
- Count wins only where both orders agree. Treat a flip as a tie.
The rep's side of a coach is a conversation, so judge it that way too. On our platform the rep brings a live deal to the expert on a call and talks it through; typing is there for when they cannot talk. Ask any vendor for a recorded call on a real deal alongside the typed answers.
The research is discussed on Show Your Working. Our benchmark of models writing sales emails is at the email LLM benchmark. Related: average error flatters a lazy scorer. Talk to us.
FAQ
How do you benchmark an AI sales coach?
To benchmark an AI sales coach, write cases as real requests from reps and managers and have the people who own the sales method set the criteria. Then compare coaches' answers in pairs, blind and in both orders, with more than one judge. Ask each judge which answer the rep would act on.
Why judge AI coaching answers in pairs?
There is no fixed scorecard for a good answer to an open coaching question, so we ask judges a question they can answer: which of these two would the rep act on? Pairs must be judged in both orders: in our call-scoring research, a model asked to compare two calls picked the first one it read 62% of the time.
What does "likelihood to act" mean in AI coach evaluation?
Likelihood to act is the judging test we use: which of two answers is the rep or manager more likely to act on? It rewards specific next steps on the real deal and penalizes generic advice and lectures. It does not penalize depth when the depth is what the reader needs.
Can I trust an AI coach that runs on the newest model?
Only after it is re-tested. In our call-scoring research each newer model generation marked reps harsher than human assessors, and the most expensive model finished eighth. A new model is a change to the coach, and it should pass the same benchmark before it reaches reps.
Does Replicate Labs publish its AI coach benchmark scores?
Only the headline result and one worked case, both on our homepage. We publish the method in principle and the call-scoring research in full, including more than 5,000 tests and more than 250,000 scoring decisions. The benchmark's criteria rest on methods our partners own, so per-case scores and case libraries stay private.