Stage 9 — Prompt Quality Evaluation
Without quality measurement, prompt engineering turns into subjective debate.
Stage topics
- Response determinism
- Repeatability
- Quality metrics
- Prompt versioning
Determinism
You need controlled variance, not perfect identical outputs.
Some tasks require strict stability; others allow bounded variability.
Repeatability
Evaluation must be reproducible:
- fixed case set,
- fixed acceptance criteria,
- compare prompt versions, not intuition.
Metrics
Minimal useful set:
- accuracy/task success,
- format validity,
- policy compliance,
- latency/cost.
Metric weights depend on product constraints.
Prompt versioning
Every prompt update should include:
- version id,
- hypothesis,
- before/after results,
- rollout or rollback decision.
Building an evaluation loop
Evaluation starts before rewriting the prompt. First, collect representative cases: easy cases, edge cases, known failures, and high-value user scenarios. Then define what a correct answer means for each case. Only after that should you change the prompt and compare versions. If you edit first and define success later, the evaluation will usually justify the change instead of testing it.
A useful benchmark is small but stable. It should be large enough to catch regressions, but not so large that nobody runs it. Human review can be part of the loop, but the criteria must still be written down. For structured outputs, automate format checks. For factual outputs, check grounding. For support or teaching content, check usefulness against learner goals.
| Evaluation item | What it answers | Example signal |
|---|---|---|
| Regression cases | Did old behavior break? | Same pass/fail criteria |
| Edge cases | Does prompt handle limits? | Safe refusal or valid fallback |
| Format tests | Is output machine usable? | Parser/schema success |
| Cost and latency | Is quality affordable? | Token and response-time budget |
Stage takeaway
A good prompt is not “pretty”; it is measurably better on a controlled dataset.
Beginner explanation
Prompt evaluation is necessary because one good answer proves nothing. The model can answer a demo case well and fail on a nearby real request. Therefore a prompt should be tested on a set of cases: normal cases, hard cases, known failures, empty data, conflicting instructions, and valuable user scenarios.
An evaluation dataset is a set of inputs and expected criteria. You do not always need one perfect answer. Sometimes criteria are enough: JSON is valid, all required fields exist, sources are not invented, the answer helps a beginner, and length limits are respected. For each criterion, decide who checks it: automated test, human reviewer, or model judge.
Iteration should test a hypothesis. Weak approach: “I rewrote the prompt and it looks nicer.” Strong approach: “I added examples and expect format validity to rise from 82% to 95% without lowering factuality.” Then compare baseline and new version. If the new version improves one demo but breaks the regression set, it is not an improvement.
Mini scenarios from real projects
- Prompt was “improved” by intuition and broke other cases: no regression test set exists.
- Metrics go up but user value does not: surrogate metrics are tracked instead of business quality.
- Prompt version changed, but team cannot explain why results improved: versioning discipline is missing.
Fast decision rules
- Validate every prompt change against a fixed benchmark/regression case set.
- Tie quality metrics to product outcomes, not only measurement convenience.
- Attach changelog and expected impact to every prompt version.
Self-check questions
- Why is objective prompt improvement impossible without a regression set?
- Which metrics actually reflect quality for your use case?
- What should prompt versioning track besides prompt text?