LLM-as-a-judge is the practice of using a large language model to grade an output, or pick the better of two, against written criteria in place of a human reviewer.
Judging calls and coaching answers
In sales, the judge is usually scoring a call against a scorecard, or comparing two pieces of work such as coaching answers or email drafts. It is fast and cheap, and it carries biases a team has to measure.
In our field guide Can AI judge a sales call?, a model asked which of two sales calls was stronger picked the first one it read 62% of the time, and each newer model generation marked reps harsher than trained human assessors. A judge is trustworthy only after it has been checked against blind human marks on ranking (average error flatters a lazy scorer), and re-checked whenever its model changes. The overview of the research is the sales coaching research hub.
Checking a judge before trusting it
This example is illustrative. An enablement team uses an AI judge to score discovery calls. Before trusting it, they have assessors mark a set of calls blind, then compare the judge's order of calls with theirs and score the set twice to see how many marks move.
- 01Assessors mark calls blind
- 02Compare the judge's orderAgainst the assessors' order of calls
- 03Score the set twiceCount how many marks move
- 04Re-check when the model changes
What passing looks like: our final setup for this job uses two judges kept blind to each other, one answering narrow yes/no questions about the call and one giving a whole-call score, with their rankings averaged. It reached 0.89 rank agreement with the experts' order. For comparing coaching answers, we judge every pair in both orders, so position bias cannot decide the winner. The research is discussed on Show Your Working, and a separate benchmark of models writing sales emails is at the email LLM benchmark.
Related terms
FAQ
Is LLM-as-a-judge reliable?
It can be, once measured. An unchecked judge carries biases that move scores for reasons unrelated to the call, and a checked setup can rank calls close to trained assessors, as the call-scoring guide shows. Check any judge against blind human marks first.
What are the biases of an LLM judge?
Known biases include preferring the first item it reads, severity that shifts between model versions, and anchoring on other scores in its context. Shown another model's scores, one scorer's recall of top-band calls fell from 45% to 5% in our call-scoring research.
How do you reduce LLM judge bias?
Run comparisons in both orders and keep judges blind to each other and to labels. Use written criteria from the people who own the standard. Correct a measured severity bias with a fixed offset after scoring, and re-test whenever the model changes.