The Replicate Labs benchmark · 20 October 2026

Ranked first against the leading AI models.

We put Replicate Labs against the newest models from Anthropic, OpenAI and Google in 120 sales situations. Judges from several AI labs compared each pair of answers blind.

  • 79%of judge votes won
  • 4 of 4jobs ranked first
  • 3,840blind judge votes
  • 120sales situations

By job

First in every job we tested.

Share of judge votes won against each of the five models. Pick a job to see the breakdown.

Seller coaching: 84% of judge votes won

The rep brings a live deal or a call. A good answer finds what the deal is missing, holds the line when the rep pushes back, and leaves one next step.

Share of votes won against each model
  • vs Claude Fable 5.181%
  • vs Claude Opus 5.585%
  • vs GPT-6.1 Sol81%
  • vs GPT-6 Astra79%
  • vs Gemini 4 Argon92%

34 situations. Elo 1742, against 1509 for GPT-6 Astra. The line marks 50%. Every bar clears it.

Read the answers

Same question. A fraction of the words.

The rep asked

“I've got Ardent Pet Clinics in my pipeline, corporate cards and bill pay, maybe 40 thousand a year. The VP Finance, Kofi, keeps taking my calls and says it's interesting, but every time I ask about timing he says 'let's keep talking'. My manager wants me to either close it this quarter or explain why. What do I do?”

18 of 20 judge votes on this situation, across all five models
Replicate Labs198 words

Stop asking Kofi for a date. His interest is not a next yes, and there is no buyer-confirmed reason to forecast this quarter from what you have told me. On the next call, ask, “What is happening with card spend or bill payments today that made this worth discussing, and what does it cost you if it stays that way?” If he cannot identify an impact worth acting on, tell your manager the forty thousand dollars a year is a possibility, not a quarter-close commitment. What has Kofi actually said is going wrong?

First reply shown: 93 of 198 words

Claude Fable 5.11,342 words

Kofi's "let's keep talking" is an answer [...] Interesting isn't painful. You have a relationship, not a deal, and your job this quarter is to find out which one it can become.

What's actually going on

A VP Finance who keeps taking calls but won't commit to timing usually falls into one of these:

1. There's no trigger. Nothing is broken enough to justify a migration, a procurement cycle, and training his team. Card programs and AP are sticky; "better" rarely beats "working."

2. There is a trigger but it's on his calendar, not yours [...] a renewal with their current card provider, a new controller starting, an audit finding, a clinic acquisition, a fiscal year rollover. He hasn't told you because you haven't asked the right way.

3. He's not the decision maker. At a multi-location vet group, there may be a CFO, a PE sponsor, or an owner-operator who actually signs. Kofi is gathering information for someone you've never met.

4. He likes you and finds it easier to keep saying yes to a call than to say no.

Any of these can be true at once. None of them get solved by another discovery call about features.

What to do this week

Have one honest conversation with Kofi, and make it different from the last five:

[...] 1,342 words across the conversation

Word counts cover each whole conversation; the first replies were 93 and 734 words. Votes from the 20 October run.

The judges

When the rep pushed back with no new buyer evidence, [Replicate Labs] held its position ('Push for Q4, yes. Forecast it as Q4-ready, no.') and gave the rep one sharp pre-call question for Robin about what exactly was approved.

Claude Opus 5.5, judging blind against GPT-6.1 Sol · Seller coaching

In the judges' own words.

Each judge saw two answers marked A and B. The name in brackets is ours.

  • [Replicate Labs] refused the rep's 'just run all 15' push with a flat 'No' and a reason, then delivered a finished opener plus four questions and a buyer-owned next step the rep can use tomorrow.

    Claude Fable 5.1, judging blindagainst GPT-6 Astra
  • [Replicate Labs] holds firm that a conditional agreement is not a booked meeting and gives one concrete next step: establish when payroll will return the numbers.

    GPT-6.1 Sol, judging blindagainst Claude Fable 5.1
  • [Replicate Labs] challenges the rep's overstated $36,000 impact as unconfirmed attribution rather than established cost, protecting credibility with the buyer.

    Kimi K2 Thinking, judging blindagainst GPT-6.1 Sol
  • [Replicate Labs] holds the distinction between Marta’s estimate and verified losses under pushback, then gives one usable question about payment records.

    GPT-6 Astra, judging blindagainst Claude Opus 5.5
  • [Replicate Labs] gives the manager a usable reset: review one call, ask Jordan to assess the buyer evidence, and agree one question to practice and review on the next call.

    GPT-6.1 Sol, judging blindagainst Claude Opus 5.5
  • [Replicate Labs] makes the correct $136,000 commit call, corrects the total to $297,000, and assigns concrete actions before Friday.

    GPT-6 Astra, judging blindagainst Claude Opus 5.5

How it was judged

Every vote was cast blind.

  1. 1

    120 situations

    Real sales moments across four jobs, each with a written standard of what a good answer does.

  2. Two answers, no names

    Replicate Labs and one model answer the same situation. The judge sees them as A and B.

  3. 3

    Seven judges

    From several AI labs. No judge's vote counted on its own lab's answers.

  4. 4

    3,840 votes

    Each vote picks the better answer of the pair, or calls it a tie.

  5. 5

    Win rate and Elo

    Win rate is the share of votes won. Elo turns every vote into one ranking.

Each model ran with its own defaults. Replicate Labs ran as one player: our seller expert on the seller jobs, our manager expert on the manager jobs.

Measured on 20 October 2026. The results describe the versions tested on that date. Models change, so we run it again whenever a leading model is released or retired.

Data and transparency

We ran this benchmark, and we recognise the conflict of interest in comparing our own experts with the models they are built on. That is why the judges were blind, came from several labs and never judged their own family.

For how well AI can score a recorded sales call, see the call scoring study.

Test it on your hardest situations.

Talk to us