"It seems to work" is how most AI features ship, and how most of them break. A basic evaluation takes an afternoon and pays for itself the first time you change a prompt.
Collect real examples#
Pull twenty real inputs from your product (or the ones you expect). Write what a good answer looks like for each, in a sentence. Put them in a spreadsheet.
Write a rubric you can apply#
Pick three to five yes/no questions: Is it correct? Does it follow the format? Does it stay in scope? Yes/no is easier to agree on than a 1-to-10 score.
const results = await Promise.all( examples.map(async (example) => { const output = await generate(example.input) return { id: example.id, output, passed: rubric(example, output) } }))console.log(`${results.filter((r) => r.passed).length}/${results.length} passed`)Run it on every change#
The number itself matters less than the trend. When it drops after a change, you have found the problem before your users did.
Note
Add a new example every time something fails in production. Your test set slowly becomes a record of everything you have learned.