Your AI eval set is your taste, made measurable

An evaluation set is a list of real inputs to your AI feature, each paired with the output you would consider good. That is the whole definition. It is also the closest thing AI product development has to written-down taste — and the discipline most solo founders skip. I have been arguing that design taste is the new moat, but taste needs somewhere to live. An eval set is the storage.

On this page

• What is an eval set?
• Why do teams skip evals?
• How do I build an eval set?
• Why an eval set is written-down taste

What is an eval set?

An eval set is a test for a probabilistic system. You give the model a real input, compare its output against what you decided is good, and score it. Anthropic's guide to building evals for agents describes it the same way: give the AI an input, then apply grading logic to its output to measure success.

Every eval has four parts. An input prompt. The model's output. A golden answer — the thing you decided was good. And a score.

The important word is "real." Not invented. Not optimistic. Real inputs pulled from your logs, your support inbox, and your own hands testing the product.

Why do teams skip evals?

Because they feel like homework at the exact moment everything got fast. You can regenerate a prompt or a screen in seconds. Writing down thirty real inputs and what good looks like for each takes an afternoon and produces nothing you can demo. So it feels like the most expensive thing you could do.

It is the cheapest insurance you will buy. OpenAI's evaluation best practices name the failure mode directly: vibe-based evals, where "it seems like it's working" is your strategy. You swap a model, tweak a prompt, and judge by feel. The problem with judging by feel is that feel has no memory. You will not notice the regression next week. You definitely will not notice the one that shipped quietly.

How do I build an eval set?

• Collect twenty to fifty real inputs. Pull from support tickets, error logs, real usage. Capture the awkward ones — typos, empty states, requests that abuse your product.
• Write a golden answer for each. What does a good output look like? Be specific enough that another person, or a model, could grade against it.
• Pick a grading method. Exact match for structured output. A rubric with an LLM-as-judge for open-ended answers. Human review for the subjective top of the funnel.
• Run it on every change. New prompt, new model, new retrieval tweak — the eval runs first. Ninety-two percent yesterday and eighty-nine today is a story you need to read.
• Add to it forever. Every new failure mode your users find becomes a new test.

The Claude cookbook walkthrough of building evals runs the same loop: write inputs and golden answers once, grade cheaply, and let the eval become the thing you improve rather than the prompt.

Why an eval set is written-down taste

Here is the part that connects to design. Taste, in practice, is a pattern of noticing — this spacing is off, that copy overpromises, this empty state is embarrassing. An eval set forces you to write that noticing down in a form a machine can grade. It is taste with a pass rate.

This is why I keep writing that AI has no taste. The model averages what it has seen. Your eval set is the counterweight — a fixed record of what you specifically consider good, applied on every change instead of whenever you happen to remember. It will not make your taste better. It makes it consistent, which is the part taste alone cannot do.