One good answer is a demo
An evaluation is a repeatable way to compare behavior against requirements. Start with examples that represent the real task, define what counts as acceptable and retain failure cases. The OpenAI evaluation guide explains structured evaluation workflows. Our recommendation: choose a rubric before you see the outputs, so a charming answer cannot move the goalposts.
For extraction, check exact field values and whether the system correctly declines unreadable input. For summarization, check coverage, factual support and omitted caveats. For a coding change, review correctness, tests and security implications. Some criteria can be automated; others need human judgment. If you use another model as a grader, inspect its decisions rather than assuming a second model has become an impartial judge.
Build a set that can disappoint you
Include ordinary examples, edge cases and cases where the right answer is “not enough evidence.” Keep model version, prompt, tool configuration and data version with each run. Do not quietly change several variables and then attribute the result to a new model. Separate examples used to tune the prompt from examples used to check whether the change generalizes. These are evaluation-design recommendations grounded in the guide, not results from tests we have run.
An evaluation set is not a magic seal. It samples behavior under particular conditions. Keep reviewing production failures and update the set when the job changes. Avoid publishing a percentage without the case count, rubric and limitations.
Price the successful outcome
Cost per successful task is total workflow spend divided by accepted outcomes. Include input, output, tools, retries and any relevant infrastructure. The API rate card illustrates how these line items can differ. Gateway and logging services can add their own rules; see Cloudflare’s gateway pricing.
Here is a hypothetical example, not a platform measurement: 20 attempts cost $4 in total and 16 meet your rubric. Cost per accepted result is $4 ÷ 16 = $0.25. If you need a person to repair every answer, include that review effort in a separate operational metric. Do not report only the cheapest individual call while ignoring failures.
Decide what changes the verdict
Exercise: A new prompt halves token cost but causes unsupported claims in three cases that previously passed. Is it an improvement?
Answer: Not automatically. Apply your predeclared quality threshold, inspect severity and compare total accepted outcomes. A cheaper failure is still a failure. Whether the trade-off is acceptable depends on the job, not on the prettiness of the cost chart.
The takeaway: keep the report card and the receipt together. A premium AI workflow should be explainable in terms of supported output and total effort, not just a spectacular screenshot.