How was the benchmark judged?

Blind. Each of the six models judged the field in turn, playing three buyer personas (a busy CRO, a skeptical VP of Sales, and a founder burned by AI tools) and picking one winner with written reasoning. Email labels and order were randomised for every judge, so no judge knew which model wrote what. That is 18 independent judgements.

Isn’t it biased if the LLMs judged each other?

The opposite. Judging was blind and anonymised, so models had no way to favour their own output. GPT judged Keenan’s email against its own without knowing which was which, and still picked Keenan.

What makes a cold email good, according to the buyers?

Across all 18 verdicts, buyers rewarded one behaviour: diagnose the problem before pitching. The winning email led with a specific, uncomfortable observation backed by real numbers, never named the product, and closed with "no pitch." The losers opened with "saw your post," pitched features, and asked for 20 minutes.

Why do frontier LLMs write weak cold email?

Because they complete the request instead of earning the reply. Ask a top model for a cold email and it produces fluent, confident, product-first copy: the "saw your post, here is what we built, worth 20 minutes" pattern that buyers delete on sight.

Keenan is Replicate Labs’ AI sales coach, built on the Gap Selling and Gap Prospecting methodology. It coaches reps on real work: sharpening cold email, scoring discovery calls, and reviewing live deals.

The Email LLM Benchmark

We asked six AIs to write a cold email. Then we let the buyers decide.

Name: The Email LLM Benchmark: which AI writes the best cold email
Creator: Replicate Labs
Published: 2026-05-06

0% Keenan won 17 of 18 blind, buyer-judged head-to-heads against the best models on the planet.

Six models wrote, then judged one another blind as real buyers. GPT-5.5, Gemini 3.1, and Claude Sonnet 4.6 won zero votes between them.

Keenan Replicate Labs 0 94.4% win rate

GPT-5.5 OpenAI 0

GPT-5.2 OpenAI 0

Gemini 3.1 Google 0

Claude Opus 4.7 Anthropic 1

Claude Sonnet 4.6 Anthropic 0

Keenan won 17 of 18 blind cold-email head-to-heads. Busy CROSkeptical VPBurned Founder

See how we ran it, below

Which AI writes the best cold email?

The short answer

In a blind, buyer-judged benchmark run by Replicate Labs, six AI models each wrote one cold email from an identical brief, then judged one another blind while role-playing three skeptical B2B buyers. Across 18 blind judgements, Keenan, Replicate Labs’ AI sales coach, won 17 (94.4%). The frontier models, GPT-5.5, GPT-5.2, Gemini 3.1, and Claude Sonnet 4.6, won zero between them. Claude Opus 4.7 took the single remaining vote.

The method

How did we test it?

We gave six AI models one identical brief: write a cold email to Sarah, a skeptical CRO at Lumina, a 120-rep SaaS company, who evaluates every vendor on hard 90-day ROI. Same buyer, same context, same constraints, for all six. Then we turned the models into a jury. Each model in turn played three buyers: a busy CRO, a skeptical VP of Sales, and a founder who has been burned by AI tools before. Every judge read the emails with the labels and order randomised, picked the one that would actually earn a reply, and wrote down why. No judge knew which model wrote which email. No model could favour its own work, because it could not see its own name. That is 18 blind judgements in total, buyers choosing on merit alone.

6 modelsIdentical briefBlind + anonymisedRandomised order18 verdicts

6models wrote

6models judged, 3 personas each

18blind verdicts

The result

What were the results?

It was not close. Keenan took 17 of the 18 votes. The frontier models, the ones that top every public leaderboard, split a single vote between all four of them, and three of them were shut out entirely.

Keenan Replicate Labs

17 94.4%

Claude Opus 4.7 Anthropic

1 5.6%

GPT-5.5 OpenAI

0 0%

GPT-5.2 OpenAI

0 0%

Gemini 3.1 Google

0 0%

Claude Sonnet 4.6 Anthropic

0 0%

Keenan won across all three buyers: the busy CRO, the skeptical VP, and the burned founder. The only vote it dropped went to Claude Opus 4.7, from a single founder persona.

The emails

What did the winning email look like?

Here is the email that won, in full, exactly as Keenan wrote it. Read it before you read the analysis. Notice what it never does: it never names the product, never references a LinkedIn post, and never asks for a meeting.

Winner Keenan · Replicate Labs 17 / 18

Subject: What newer reps lose

Sarah, your newer reps probably sound sharper than ever with better prompts and cleaner call prep.

But if you reviewed ten slipped deals, I’d bet the decay shows up somewhere else: weak problem definition, shallow impact, and next steps the buyer never owned.

We saw this inside a 120-rep SaaS org. 68 percent of reviewed calls had no quantified gap. 31 late-stage deals had one stakeholder. Managers gave feedback on talk track, not diagnosis.

The reps were not getting worse. Their coaching was grading fluency instead of buyer clarity.

That matters if ROI has to show in 90 days.

I can send the review notes if useful. No pitch.

Why it won

Opens on the buyer’s problem, not the sender.
Runs a diagnostic ("if you reviewed ten slipped deals") that makes the buyer inspect her own org.
Uses forensic numbers (68 percent, 31 deals), not marketing ones.
Reframes the root cause ("coaching was grading fluency instead of buyer clarity").
Never pitches: no product name, no meeting ask, closes with "No pitch."

Every email, unedited. Read the whole field yourself.

GPT-5.5 A brochure with a calendar ask

Subject: You were right about AI co-pilots

Sarah — saw your post about skill-decay in newer reps and the frustration with AI co-pilots that don’t actually make reps better.

That’s exactly why we built Replicate Labs.

Instead of generic “say this next” prompts, reps submit real emails, calls, and cadences to AI sales coaches modeled on top methodologies — e.g. a Gap Selling coach that flags where the rep missed current state, root cause, business impact, or compelling reason to change.

For Lumina, I’d prove it in 90 days like this:

- 20–30 newer reps across US + EMEA
- Baseline their recent calls/emails/cadences
- Give them weekly coach feedback tied to specific deal moments
- Measure lift in discovery-to-next-meeting, stage conversion, outbound reply quality, and manager coaching time saved

If there’s no measurable improvement in 90 days, you shouldn’t buy it.

Worth a look, or should I send you the pilot design first?

Why it lost

Opens with "saw your post."
Pivots to the product by line three.
Hands the buyer a pilot table before proving it understands the problem.
Fluent, confident, and interchangeable with a hundred other vendor emails.

GPT-5.2 The template, fully assembled

Subject: Sarah — prove “rep skill lift” ROI in 90 days (not more AI copilot)

Sarah —

Saw your post on skill-decay with newer reps + the frustration with “AI co-pilot” tools that don’t actually make people better. That’s exactly the problem we built Replicate Labs around.

We give reps access to AI coaches that emulate specific sales coaches/methodologies (e.g., Keenan / Gap Selling). Reps paste in emails, call notes, recordings, cadences — and the coach grades and rewrites with methodology-grounded feedback (problem, impact, gap, next-step), so managers aren’t stuck giving generic “ask better questions” advice.

Given Lumina’s scale (~120 reps across US/EMEA), would you be open to a 90-day ROI pilot structured like this?

- Cohort: 20 newer reps + 5 managers
- Baseline: pick 3 skills you care about (discovery depth, multithreading, next-step discipline)
- Measurement: pre/post scoring on real artifacts + leading indicators (meeting-to-oppty, stage conversion, cycle time)
- Output: weekly skill lift report you can show your CEO/board

If I send a 1-page pilot plan + the exact metrics template, will you tell me if it’s a fit for how you evaluate vendors?

— [Your Name]
Replicate Labs
[Phone] | [Email]

Why it lost

Opens "Saw your post on skill-decay... that’s exactly the problem we built Replicate Labs around." Personalisation straight into the product.
Hands over a four-line pilot plan and asks to be evaluated as a vendor, before naming a single problem Sarah actually has.

Gemini 3.1 Anti-co-pilot, same co-pilot pitch

Subject: AI co-pilots & skill decay

Sarah,

Read your post about AI co-pilots causing skill decay among newer reps. You're spot on—when a tool does the thinking, reps stop getting better.

We built Replicate Labs to be the anti-co-pilot.

Instead of writing generic emails *for* your 120 reps, it acts as an elite coach. Reps submit their calls, cadences, or emails and get them torn down by AI trained exclusively on top methodologies (like Keenan’s Gap Selling). They get specific, methodology-grounded feedback that forces actual skill development.

I know from your recent podcast interview that you evaluate vendors on a strict 90-day ROI. Because we're fixing the underlying skill gap instead of masking it with AI-generated fluff, we routinely hit that metric.

Open to me sending over a 2-minute video showing how the AI would critique a real cold email?

Why it lost

Mirrors the post back ("You’re spot on"), brands itself "the anti-co-pilot," then pitches the product anyway.
Name-drops her "recent podcast interview" and closes by asking to send a 2-minute demo video.

Claude Opus 4.7 The best of the frontier, still a demo trap

Subject: Your LinkedIn post on AI co-pilots

Sarah,

Your post three weeks ago hit a nerve — “AI co-pilots that surface next-best-action but never make the rep better next quarter.” That’s the gap we built for.

Replicate Labs gives your reps access to AI coaches trained on specific methodologies — Keenan on Gap Selling, others on MEDDPICC, command-of-the-message, etc. Reps drop in their actual emails, call recordings, and sequences. They get back the same feedback Keenan would give in a 1:1 — what they missed in discovery, where the problem identification was thin, why the email didn’t earn a reply. Methodology-grounded, rep-specific, repeatable.

For your skill-decay problem on newer reps: the coach is available 24/7, doesn’t get tired of repeating itself, and every interaction is logged so your enablement team can see what’s actually moving and what’s not.

On the 90-day ROI bar you mentioned on the podcast — we measure cohort skill lift (discovery depth, problem articulation, email reply rates) at 30/60/90. Two of our mid-market customers around your size hit measurable reply-rate and meeting-set lift on newer-rep cohorts inside that window. Happy to send the methodology and the raw numbers.

Worth 20 minutes to see if the Gap coach handles a real Lumina deal better than your current stack?

James
Replicate Labs

Why it lost

The best of the frontier attempts. It quotes the post precisely and ties frustration to a mechanism, and one founder rated it top.
But it still leads with the product, still asks for 20 minutes, and still reads like a vendor explaining itself.
Good enough to win once in eighteen. Not good enough to win.

Claude Sonnet 4.6 Confident, and interchangeable

Subject: You said it better than we could

Sarah,

Your LinkedIn post three weeks ago — the one about skill-decay and AI co-pilots that don't actually help reps improve — that's the exact reason Replicate Labs exists.

We're not another co-pilot. We give your reps direct access to AI coaches built on the methodologies of coaches like Keenan (Gap Selling). Rep shares a call or email, gets back specific, methodology-grounded feedback — not generic suggestions, not a summary. Actual coaching on what went wrong and how to fix it.

With 120 reps across US and EMEA, the skill variance is probably costing you more than you can measure. We can show you what it looks like to close that gap — and we can show you ROI in 90 days or we don't deserve your time.

Worth 20 minutes?

James Pursey
Replicate Labs

Why it lost

Flattery open ("You said it better than we could"), product reveal by line two ("that’s the exact reason Replicate Labs exists").
Bold "ROI in 90 days or we don’t deserve your time," then the 20-minute ask.

The pattern

Why do frontier LLMs write bad cold email?

We read all 18 verdicts and the pattern was almost eerily consistent. The buyers did not reject the frontier emails for being badly written. They rejected them for being written at all: for sounding like exactly what they were, a language model completing a request. Here is what the losers kept doing, in the buyers’ own words.

They opened with "saw your post."

Every buyer flagged it. It is the tell that screams automation.

I deleted the others instantly because they all used the lazy "I noticed you posted about" hook and bloated their emails with product pitches and heavy asks.

Gemini 3.1, playing the Busy CRO

They led with the product.

The pivot from personalisation to pitch happened in the first three lines, every time.

Every other email opens with some version of "saw your post, we built this for you," which is fine once but feels like a mail-merge by the fifth time.

Claude Sonnet 4.6, playing the Skeptical VP

They asked for 20 minutes before earning it.

The meeting ask, the pilot table, the "worth a look." A demand for time made before any relevance was proven.

The rest are variations of the same vendor-script... fine, but indistinguishable from the other 149 emails.

GPT-5.2, playing the Skeptical VP

They were interchangeable.

The sharpest indictment. Not that any single email was bad, but that all of them were the same.

They all lean on the same recycled LinkedIn post plus 90-day ROI scaffolding, interchangeable AI-slop with pilot tables I’ve seen 50 times this quarter.

Claude Opus 4.7, playing the Busy CRO

What did the winning email do differently?

Keenan did the one thing none of the frontier models did. It diagnosed before it pitched. Then it stopped. The buyers noticed immediately, and they described the same five moves over and over: it ran a mental exercise that made them inspect their own org, it used specific forensic numbers instead of marketing claims, it reframed the root cause into something genuinely useful, it never mentioned the product, and it closed with "no pitch" instead of a meeting ask.

It’s the only one that didn’t immediately feel like a vendor trying to drag me into a demo. The slipped-deal pattern was specific and believable... The others leaned on fake personalization, "AI coach" jargon, inflated ROI claims, or asked for 20 minutes.

GPT-5.5, playing the Skeptical VP

It earns the reply because it doesn’t pitch. It shows evidence and stops.

Claude Sonnet 4.6, playing the Skeptical VP

The frontier models wrote emails about the seller. Keenan wrote an email about the buyer. That is the entire difference, and buyers can feel it in the first sentence.

This is not a knock on the frontier models. They did exactly what they were asked. That is the problem. A general model completes the prompt, and "write a cold email" produces fluent, product-first copy by default. Keenan was built on a method, Gap Prospecting, whose entire discipline is to resist that instinct: lead with the buyer’s problem, quantify it, and earn the reply before you ask for anything. The method is the difference, and the method is learnable.

Start free

Get Keenan on every deal, in every moment

Coaching and doing, for free. On your real emails, your real calls, your live pipeline. 90 seconds to start, no card.

Get Keenan free

We asked six AIs to write a cold email. Then we let the buyers decide.

Which AI writes the best cold email?

They opened with "saw your post."

They led with the product.

They asked for 20 minutes before earning it.

They were interchangeable.

What did the winning email do differently?

Get Keenan on every deal, in every moment.

Get Keenan on every deal, in every moment