// Before you start
This guide builds on this one:
An agent that works once proves almost nothing. Models are not deterministic, so the only honest question is: how often does it do the right thing on the questions that matter? A small test set answers that, and it is the single habit that separates demos from things people can rely on.
Write down what "good" looks like#
Before any code, write five to ten questions you would really ask, each with the one fact a good answer must contain. Include at least one question the notes cannot answer, because the failure you most want to catch is a confident invention.
- Typical: "When is standup?" must mention 9:30.
- Needs two facts: "What do I do before launch?" must mention tests and the sitemap.
- Awkward: a question with a typo, or one that uses different words from the note.
- Unanswerable: "What is the office wifi password?" must not produce a password.
Turn it into a test file#
Keep the checks simple. You are looking for the right fact, not the exact wording, so test whether the answer contains the text you expect.
import { ask } from './agent.ts'type Case = { question: string; mustInclude?: string[]; mustNotInclude?: string[] }const cases: Case[] = [ { question: 'When is standup?', mustInclude: ['9:30'] }, { question: 'Who runs standup?', mustInclude: ['Priya'] }, { question: 'What do I need to do before launch?', mustInclude: ['tests', 'sitemap'] }, { question: 'Which book should I read first?', mustInclude: ['Pragmatic'] }, // A question the notes cannot answer: the agent must not invent something. { question: 'What is the office wifi password?', mustNotInclude: ['password is'] },]let failed = 0for (const { question, mustInclude = [], mustNotInclude = [] } of cases) { const answer = (await ask(question)).toLowerCase() const missing = mustInclude.filter((text) => !answer.includes(text.toLowerCase())) const banned = mustNotInclude.filter((text) => answer.includes(text.toLowerCase())) const passed = missing.length === 0 && banned.length === 0 if (!passed) failed += 1 console.log(`${passed ? 'PASS' : 'FAIL'} ${question}`) if (!passed) { console.log(' got:', answer.slice(0, 200)) if (missing.length) console.log(' missing:', missing.join(', ')) if (banned.length) console.log(' should not say:', banned.join(', ')) }}console.log(`\n${cases.length - failed}/${cases.length} passed`)process.exitCode = failed ? 1 : 0Run it with npx tsx evals.ts. Each case prints PASS or FAIL, and failures show what the agent actually said. Run it two or three times: if a case passes once and fails once, that is information too.
Heads up
Do not only test the questions you already know it handles. A test set made of easy wins feels good and catches nothing.
Read the failures, then change one thing#
Every failure falls into one of a few families. Name the family before you touch anything:
- It searched badly: the query was a whole sentence, or the wrong words. Fix the tool description.
- It did not search at all: it answered from memory. Fix the system prompt: "search before you answer".
- It searched well but answered badly: it ignored or misread the result. Tighten the answer rules in the prompt.
- It invented an answer: add the rule "if the notes do not contain it, say so", and keep that unanswerable test case forever.
Change one thing at a time and re-run everything. If you change three things and the score goes up, you will not know which one helped, and the next change might quietly undo it.
Use the questions it gives you as new cases, after you have checked they are realistic.
// Try it
Write five cases of your own
Add three more notes to the list (your own, real ones), then write five new test cases for them.
- Two typical questions with a clear expected fact.
- One question that needs information from two notes.
- One question phrased with different words from the note.
- One question the notes cannot answer.
Run the set. Pick the failure that surprises you most and decide which family it belongs to.
Show a solutionHide the solution
// Solution
There is no single right answer, but a good set looks like this: the expectations are short facts, not whole sentences, and at least one case is designed to make the agent fail.
{ question: 'Wen is standup?', mustInclude: ['9:30'] }, // typo{ question: 'Who is in charge of the Monday meeting?', mustInclude: ['Priya'] }, // different words{ question: 'Who runs standup and what is on the launch list?', mustInclude: ['Priya', 'sitemap'] }, // two notes{ question: 'When is my passport renewal due?', mustNotInclude: ['due on', 'expires'] }, // not in the notes