// Before you start
This guide builds on these:
You do not need special tooling to evaluate an AI feature. You need a test set, a rubric and the discipline to run them.
Build the test set#
Start with twenty real or realistic inputs. For each, write what a good answer contains. Keep a few hidden for the final check.
Choose the rubric#
Three to five yes/no questions: correct, complete, in format, in scope, safe. Binary questions are easier to agree on than scores.
Automate the boring part#
A script that runs every example and prints "passed / total" is enough. Store the results in git so you can compare releases.
Read the failures#
The number tells you whether something changed. The failures tell you why. Read every one, and add the pattern to your test set.
// Try it
Build your first test set
Pick one feature of yours that uses an LLM. Write ten realistic inputs for it.
- For each input, write one sentence: what a good answer contains.
- Turn those sentences into three yes/no rubric questions.
Show a solutionHide the solution
// Solution
A good first set mixes easy, typical and awkward inputs. For a ticket summariser:
- An easy one: a clear, short complaint.
- A typical one: a long ticket with some noise.
- An awkward one: a ticket in two languages, or with no real question.
Rubric1. Does the summary say what the customer wants?2. Is it in three bullets or fewer?3. Does it avoid guessing at causes?