Every model or prompt change is a gamble against invisible regressions — users notice breakage days later, one degraded output at a time.
← Back to patterns
Trust & Verification
EmergingPersonal Eval Set
Your own test cases, run against every change
Every model or prompt change is a gamble against invisible regressions — users notice breakage days later, one degraded output at a time, with no record of what "good" looked like. Personal Eval Set turns your own representative cases into a pinned test bench: run the suite in one click — or automatically when the model changes — and see per-case verdicts with diffs where behavior moved. The regression suite, brought to everyone who depends on a model's behavior.
Metaphor
Test Bench
Interaction
Pin-as-Case, One-Click Suite Run, Per-Case Verdicts
Use case
Prompt Changes, Model Updates, Workflow Stability
Failure mode
Invisible Regression
Control mechanisms
Pinned Cases With Expected Qualities · Suite Runner · Regression Diff · Case Pruning
Reference examples
promptfoo · LangSmith datasets & evals · Braintrust
Interactive Example
The live side-by-side is the comparison.
This prototype already contains the contrast between chat and the pattern response. It stands as the primary comparison surface for this page.
You would notice, eventually. The bench notices in 4.2 seconds.
Thursday, 11:20: the ticket summary comes back as hedging prose, and the only record of what good looked like is an output from June 12, buried three weeks up the scrollback. You find it, paste it back, the model promises to keep the format. The fix lasts exactly one conversation — and nothing is watching for the next drift.
On a bench, good is written down before it moves: four pinned cases, each carrying the qualities you named — ≤ 5 bullets, names the customer, no hedging. One click runs all four against the 3.2 → 3.3 vendor update and returns 3 pass · 1 regression · suite ran in 4.2s, with the diff on the failure showing bullets collapsed into prose, 24 words to 61, three hedges inserted.
The honesty lives in the assertions. Switch off no hedging and the same run turns green — the bench measures only what you claimed to want — and a case you prune stops being measured at all. Engineers have had this as promptfoo and LangSmith datasets for years; the version that starts from an output you liked yesterday is still missing.
Write down what good looks like, or every regression becomes an argument you hold from memory.
DOPE Lens
How this pattern improves AI interaction quality.
Observability
A suite run renders per-case verdicts in one view — what held, what regressed, what improved — instead of anecdotes
Directability
The user defines what good means — pin a case from any output, name its expected qualities, prune it when the job changes
Predictability
Changes are measured against the same fixed cases every time, so "did it get worse?" has a stable answer
Explainability
Each failed case shows the diff against its expected qualities — the regression is demonstrated, not suspected
Fit
Where it earns its place.
Use it
when
when
Work depends on stable model behavior and changes (model, prompt, config) need a regression check before they ship.
Avoid
when
when
Exploratory work with no stable definition of correct.
In the wild
Where this pattern already ships.
The pattern is shipping but still maturing across products. Entries below show how different teams have approached it; expect variation as the space settles.
promptfoo
Open-source eval harness: declare test cases with assertions, run them against prompt or model changes, see a pass/fail matrix — the pattern as a developer CLI/CI tool.
Visit promptfoo ↗
LangSmith · Datasets & evals
LangChain
Pin real production cases into datasets and score new versions against them — regression testing for LLM behavior at platform scale, developer-facing.
Visit LangChain ↗
Braintrust
Eval-first LLM engineering: side-by-side diffs of eval runs across model/prompt versions. The end-user version of this — pin-from-output, one-click suite — remains unshipped.
Visit Braintrust ↗
Research Frame
Why this pattern exists, and where it breaks.
Pin your own representative cases as a test bench with expected qualities, and run the suite — one click or automatically — whenever the model, prompt, or config changes.
Chat keeps no record of what good looked like — each conversation re-litigates quality from scratch, from memory.
A stale eval set calcifies yesterday's requirements — cases need pruning and refreshing as the job evolves, or green checks mean nothing.
Common questions
What people ask about this pattern.
When should I use the Personal Eval Set pattern?
+
Work depends on stable model behavior and changes (model, prompt, config) need a regression check before they ship.
When should I avoid the Personal Eval Set pattern?
+
Exploratory work with no stable definition of correct.
What problem does Personal Eval Set solve?
+
Every model or prompt change is a gamble against invisible regressions — users notice breakage days later, one degraded output at a time.
Why is chat the wrong fit for this?
+
Chat keeps no record of what good looked like — each conversation re-litigates quality from scratch, from memory.
Related Patterns
Curated connections, not just neighbors.
Hand-picked relationships — what this pattern pairs with in practice, what it replaces, what it builds on, and what builds on it.
Often paired with
Model Changelog
Model Changelog announces the model update in-product; Personal Eval Set re-runs your pinned cases and shows which held.
Often paired with
Verifiable Task Completion
Verifiable Task Completion proves one task met its declared criteria; Personal Eval Set checks the system still meets yours.