Day 2 • 10:50 R0
Learn Japanese with Manga! Building a Multimodal Search Engine with Gemini Embedding 2
Machine LearningChinese (English slides)
@x0mg
Eval-Driven Development for GenAI Systems
Kyle Naranjo
Day 1 • Sat, Oct 17
14:00 - 14:15
Location
R2
Language
English
Category • Level
Machine Learning • Medium
Running agents in production introduces a class of failures that traditional software testing doesn't cover well. Outputs are non-deterministic, behavior depends on model versions and prompt wording, and regressions are often invisible until they surface in production. This talk covers how to build a structured eval framework in Python that treats agent behavior as testable and measurable. We'll define what a proper agent test case looks like, walk through deterministic assertions for tool calls and structured outputs, and build a model-graded evaluation layer using a judge model for open-ended responses.
Description
Most teams running agents in production test by feel. You change a prompt, manually check a few outputs, and ship. This works until a model update silently degrades retrieval quality, a tool refactor breaks calling behavior, or an edge case surfaces that was never tested.
Agents are non-deterministic, so conventional unit testing doesn't directly apply. But agent behavior is still testable and measurable.
Core concepts covered:
- What an eval actually is: A test case is (input, expected behavior). For agents, expected behavior has two layers: did the agent take the right actions (tool calls, retrieval), and was the final output correct or high quality?
- Deterministic assertions: Checking tool call correctness, argument structure, retrieval recall, and output schema. Fast, cheap, and should cover the majority of your test suite.
- Model-graded assertions: For open-ended outputs where exact match isn't meaningful, a judge model scores responses against a rubric.
Relevant references:
Kyle Naranjo is a Data Engineering Consultant at Thinking Machines Data Science, where he builds data platforms and AI-powered data products for enterprises. His work spans large-scale data pipelines, probabilistic record linkage, and GenAI/agentic systems. He holds 10 professional certifications across Google Cloud, Microsoft Azure, AWS, Databricks, and OpenAI.
