Building a Model Evaluation Practice for Production AI

Most teams pick an AI model the same way they pick a coffee machine - the demo looks good or they hear about it from someone, so they assume it works. But in production, evaluation decisions directly impact cost, latency, accuracy, and user trust. Without a structured eval practice, you're flying blind when a model regresses, or a new release claims to be better.

This session gives you a framework for evaluating LL

Ms against *your* workloads and operationalizing evaluation as a continuous practice, not a one-time exercise.

**What we'll cover:**

* Why standard benchmarks are often misleading for enterprise use cases

* Building a golden dataset: how to sample, label, and version evaluation sets from real production traffic

* Key metrics beyond accuracy: hallucination rates, precision/recall tradeoffs, cost-per-correct-answer, and latency percentiles

* Running evals at scale: open-source tools (Ragas, Prompt

Foo, Lang

Smith) and using Amazon Bedrock Model Evaluation

* Setting a model eval