Eval: does an AI grader agree with people?
Before you trust an AI to grade answers, measure it. This harness asks an AI judge to pick the better of two answers and compares it with human experts.
Data: 40 cases from MT-Bench human judgments, a public dataset (CC BY 4.0). The answers being judged are from 2023 models. Judge: gemini-3.8-flash. Real run, nothing edited.
Result
- Agreed with the humans on 32 of 40 (80%).
- Each pair is judged twice with the answers swapped. It changed its pick 1 time.
- Most cases have one or two human votes, so the labels are noisy too.
Why it matters: the same harness runs on your own cases, so you know how far to trust an AI grader or feature before you ship it.
All 40 cases
Disagreements first. Downloads: results.json, cases.json.
| Question | Humans chose | Judge chose | Votes | Match |
|---|---|---|---|---|
| Write a simple website in HTML. When a user clicks the button, it shows a random joke from a list of 4 jokes. | claude-v1 | gpt-4 | 1 | no |
| A binary tree is full if all of its vertices have either zero or two children. Let B_n denote the number of fu... | vicuna-13b-v1.2 | flipped when swapped | 1 | no |
| Analyze the following customer reviews from different sources for three different smartphones - the latest iPh... | llama-13b | claude-v1 | 1 | no |
| When a number is divided by 10, the remainder is 4. What is the remainder when twice the number is divided by ... | vicuna-13b-v1.2 | gpt-4 | 1 | no |
| Write a C++ program to find the nth Fibonacci number using recursion. | alpaca-13b | vicuna-13b-v1.2 | 1 | no |
| Which methods did Socrates employ to challenge the prevailing thoughts of his time? | alpaca-13b | vicuna-13b-v1.2 | 1 | no |
| Imagine you are writing a blog post comparing two popular smartphone models. Develop an outline for the blog p... | claude-v1 | gpt-4 | 2 | no |
| Help me construct a catchy, yet scientifically accurate, headline for an article on the latest discovery in re... | gpt-4 | gpt-3.5-turbo | 1 | no |
| Picture yourself as a 100-years-old tree in a lush forest, minding your own business, when suddenly, a bunch o... | gpt-4 | gpt-4 | 2 | yes |
| Extract the following information from the presented texts: The name of the book, the author, the main charact... | vicuna-13b-v1.2 | vicuna-13b-v1.2 | 2 | yes |
| Could you write a captivating short story beginning with the sentence: The old abandoned house at the end of t... | gpt-3.5-turbo | gpt-3.5-turbo | 1 | yes |
| Edit the following paragraph to correct any grammatical errors: She didn't remembre where is her purse, so I t... | vicuna-13b-v1.2 | vicuna-13b-v1.2 | 2 | yes |
| Provide insights into the correlation between economic indicators such as GDP, inflation, and unemployment rat... | gpt-3.5-turbo | gpt-3.5-turbo | 2 | yes |
| Imagine yourself as a doctor tasked with devising innovative remedies for various ailments and maladies. Your ... | gpt-4 | gpt-4 | 1 | yes |
| Craft an intriguing opening paragraph for a fictional short story. The story should involve a character who wa... | gpt-4 | gpt-4 | 1 | yes |
| The city of Vega intends to build a bridge that will span the Vegona River, covering a distance of 1.8 kilomet... | gpt-4 | gpt-4 | 2 | yes |
| One morning after sunrise, Suresh was standing facing a pole. The shadow of the pole fell exactly to his right... | vicuna-13b-v1.2 | vicuna-13b-v1.2 | 4 | yes |
| Describe a vivid and unique character, using strong imagery and creative language. Please answer in fewer than... | alpaca-13b | alpaca-13b | 1 | yes |
| Embrace the role of Sheldon from "The Big Bang Theory" as we delve into our conversation. Don’t start with phr... | gpt-3.5-turbo | gpt-3.5-turbo | 3 | yes |
| Please read the paragraph below and count how many times the words "Amazon", "river", and "you" appear. Please... | claude-v1 | claude-v1 | 1 | yes |
| Please describe the concept of machine learning. Could you elaborate on the differences between supervised, un... | claude-v1 | claude-v1 | 1 | yes |
| Now you are a machine learning engineer. Your task is to explain complex machine learning concepts in a simpli... | gpt-4 | gpt-4 | 1 | yes |
| Write a descriptive paragraph about a bustling marketplace, incorporating sensory details such as smells, soun... | claude-v1 | claude-v1 | 1 | yes |
| How many integers are in the solution of the inequality |x + 5| < 10 | gpt-4 | gpt-4 | 4 | yes |
| How have the Alps and Rhine River influenced settlement and agriculture in Western Europe? List three impacts. | alpaca-13b | alpaca-13b | 2 | yes |
| Write a persuasive email to convince your introverted friend, who dislikes public speaking, to volunteer as a ... | alpaca-13b | alpaca-13b | 1 | yes |
| Identify the countries, their capitals, and the languages spoken in the following sentences. Output in JSON fo... | gpt-3.5-turbo | gpt-3.5-turbo | 3 | yes |
| How do the stages of life shape our understanding of time and mortality? | gpt-4 | gpt-4 | 1 | yes |
| Please assume the role of an English translator, tasked with correcting and enhancing spelling and language. R... | gpt-3.5-turbo | gpt-3.5-turbo | 1 | yes |
| Given a set of complex equations, extract all unique variable names from each equation. Return the results as ... | gpt-4 | gpt-4 | 2 | yes |
| Benjamin went to a bookstore and purchased a variety of books. He bought 5 copies of a sci-fi novel, each pric... | gpt-4 | gpt-4 | 2 | yes |
| Given the following records of stock prices, extract the highest and lowest closing prices for each month in t... | gpt-4 | gpt-4 | 1 | yes |
| Share ideas for adapting art masterpieces into interactive experiences for children. List 5 specific artworks ... | llama-13b | llama-13b | 1 | yes |
| A is the father of B. B is the father of C. What is the relationship between A and C? | llama-13b | llama-13b | 1 | yes |
| Evaluate the following movie reviews on a scale of 1 to 5, with 1 being very negative, 3 being neutral, and 5 ... | gpt-3.5-turbo | gpt-3.5-turbo | 1 | yes |
| Please explain the differences between exothermic and endothermic reactions, and include the criteria you used... | gpt-4 | gpt-4 | 1 | yes |
| Consider a satellite that is in a circular orbit around the Earth. The speed of the satellite decreases. What ... | gpt-3.5-turbo | gpt-3.5-turbo | 2 | yes |
| Suppose you are a mathematician and poet. You always write your proofs as short poets with less than 10 lines ... | claude-v1 | claude-v1 | 2 | yes |
| Which word does not belong with the others? tyre, steering wheel, car, engine | gpt-4 | gpt-4 | 2 | yes |
| Explain what's base rate fallacy and list five specific examples of how politicians use it for campaigns. | gpt-4 | gpt-4 | 2 | yes |