0%
Forward Deployed Engineering
The Role
Discovery and Scoping
Customer Data and Systems
Running Deployments
Evaluations and Acceptance Tests
An evaluation, often shortened to eval, is a repeatable way to measure how well a system performs a task on a fixed set of cases. For customer deployments of AI and data systems, evals do three jobs. They tell the team whether a change made things better or worse. They give the customer evidence that the system meets what was agreed. And they turn arguments about whether the model is good enough into a shared, measurable question. This lesson covers why evals matter, how to build them with the customer, how to score outputs, how to use evals as release gates and how to keep evaluating once the system is live.
Demos are not evidence
A demo shows that the system can work on a few chosen examples. It does not show how often it works on the cases that arrive every day. Teams that ship on the strength of a good demo usually discover the failure modes from users, which is the most expensive way to find them. An eval with a hundred real cases, scored the same way every time, reveals the weak spots before launch.
Evals make change safe
AI systems change constantly: prompts are edited, retrieval is tuned, models are upgraded and customer documents are updated. Each change can improve one kind of case and break another. Without an eval, nobody can tell. With one, the team runs the set before every release, compares the results with the current version and sees exactly which cases moved. Evals are to AI features what automated tests are to ordinary code.
Evals as a shared contract
When the customer helps build the eval set and agrees the scoring, the eval becomes part of the contract between vendor and customer. Acceptance means passing the agreed set at the agreed level, not winning an argument in a meeting. This protects both sides: the customer knows what they are getting, and the vendor knows when the work is done.
What evals cannot do
An eval measures performance on the cases it contains. It cannot promise performance on cases it does not cover, such as a new product line or a change in how customers write. It also measures what the scoring checks, which may not be everything users care about. Treat eval results as strong evidence within their scope, keep growing the set and combine evals with monitoring in production.
Evals beyond LLMs
Evals are not only for language models. The same approach works for forecasts, matching, extraction, classification and search: a fixed set of cases with known correct answers, a clear scoring rule and results tracked over time. FDEs who build eval habits early can apply them to every part of a deployment where output quality is not simply right or wrong.
Starting early
Build the first eval during the prototype or pilot, not at the end of the project. Even twenty cases with known answers give the team a way to compare approaches. The set grows as the work goes on, and by acceptance it covers the main categories with enough cases to trust the numbers. An eval built at the last minute is usually too small, too easy and too late to shape the design.
Who owns the eval
Give the eval an owner on each side. The FDE team owns the tooling, runs and reports. A customer subject matter expert owns the correctness of the expected answers and agrees changes to the set. Without a customer owner, the team ends up deciding what correct means on its own, which undermines the eval as a shared contract.
Measuring what users value
Choose measures that reflect what users care about, not only what is easy to compute. For a drafting feature, users care whether the draft is correct, complete, in the right tone and ready to send with few edits. For extraction, they care about each field being right, especially the ones that drive payments or deadlines. Ask users what a bad output looks like and make sure the eval catches it.
Reporting results clearly
Report eval results in a way the customer can read: overall score, score by category, change from the last release and examples of failures. A single number hides too much, while a full spreadsheet overwhelms. A one page summary with a few concrete failure examples helps sponsors understand where the system stands and where it needs work.
Costs of running evals
Evals cost time and money, especially when graded by people or by a model. Keep a small fast set that runs on every change and a larger full set that runs before releases. Budget for the expert time needed to review new cases and disputed scores. The cost is small compared with discovering a regression from a customer complaint.
Evals and trust
Customers trust systems they can measure. When a sponsor can see that the extraction is right on ninety six percent of invoice totals and that the remaining errors are flagged for review, they can explain the system to their auditors and their leadership. Without numbers, every visible mistake becomes a story that spreads, and the sponsor has nothing to answer it with. Evals give the sponsor a way to defend the deployment.
Comparing approaches
Evals settle design choices quickly. Should the team use a larger model, better retrieval or more examples in the prompt? Run each option on the same set and compare scores, cost and latency. The answer is often surprising: a retrieval fix may beat a model upgrade at a fraction of the cost. Without an eval, these choices are made by preference or by whatever was tried last.
The habit of looking at failures
The most valuable part of an eval run is often not the score but the failures. Read them. Group them by cause: missing information, wrong passage retrieved, misread table, rule ignored, ambiguous question. Each group points to a different fix. Teams that read failures improve quickly, while teams that only watch the score tend to tune prompts at random.
Evals and accountability
When something goes wrong after launch, the eval history shows what was tested, what passed and what was known. This protects the team from blame for cases nobody could have predicted, and it shows honestly when a known weakness was accepted. Keep the results of every release run, with the version of the set, the prompts and the model used.
Explaining evals to non specialists
Many stakeholders have never heard of evals. A simple comparison helps: an eval is like an exam for the system, written with the customer, with answers checked by their experts, taken again after every change. Most people understand immediately why you would want that before trusting a system with real work.
Evals for the product team
Share eval results and failure groups with the product team. Patterns across customers, such as every deployment struggling with scanned tables, are strong evidence for product investment. A common eval format across deployments also lets the product team test a platform change against many customers at once before releasing it, which protects every deployment from the same regression.
When evals are skipped
Under deadline pressure, evals are often the first thing cut. The result is predictable: a release that looks fine in a quick check breaks a category nobody tried, and the team spends more time on the incident than the eval would have cost. Protect the eval run in the release plan as firmly as the security review.
Keep the eval run on the release checklist.
Report the result with the release notes.
Use evals rather than demos as evidence, run them on every change and treat them as the shared acceptance contract with the customer. Start during the pilot, give the eval an owner on each side, measure what users value, report by category with failure examples and keep a fast set and a full set.