The Agent Failure Playbook: Production Gone Wrong

Kushal Vijay

Kushal Vijay

Day 2 • Sun, Oct 18
13:20 - 13:50
Location
R1
Language
English
Category • Level
Machine Learning • Expert

Agents are breaking in production in ways traditional software never did - infinite tool loops, context explosions, hallucinated API calls, silent cost runaway, and agents that confidently do the wrong thing.

This talk is a failure-first guide to production Python agents. Real postmortems, real failure patterns, and the Python techniques that prevent them. Drawing from building AI systems at Microsoft, teaching 700K+ AI enthusiats, and collecting failure stories from the community, I'll give you the playbook nobody writes until something breaks badly.

You'll leave knowing the ways Python agents fail in production - and how to defend against each one.

For developers who'd rather learn from others' pain than their own.

Description

Production AI agent failures don't look like software bugs. They're subtle, expensive, and often invisible until the damage is done.

An agent loops a tool 47 times and burns $200 in a single request. An agent confidently calls an API with hallucinated parameters and corrupts downstream data. A context window silently overflows and the agent starts ignoring its own instructions mid-conversation.

These aren't hypothetical. They're patterns from building production AI at Microsoft, teaching agents to thousands of developers, and collecting community war stories from 600K+ followers asking "why did my agent do this?"

This talk is the failure playbook nobody writes until something goes badly wrong.

The failure taxonomy 5 production failure modes across three categories:

Reasoning failures - An agent overrides its own system prompt mid-conversation because a tool response contained conflicting instructions. No exception raised. No error log. Just wrong behavior.

Infrastructure failures - A retry loop with no exit condition hits an external API 47 times in 4 seconds. The agent reports success. The bill arrives later.

Integration failures - An agent constructs a valid-looking but hallucinated API payload. Pydantic doesn't catch it because the types are correct. The downstream database does.

For each failure: what it looks like in production, a real example, Python patterns that prevent it, and detection strategies.

Behind every failure is an architectural decision that seemed reasonable at the time. This talk gives you the production hindsight to make better decisions upfront

Kushal Vijay
Kushal Vijay

Kushal Vijay is a Software Engineer 2 at Microsoft, where he works on authentication, reliability, and AI-powered experiences used by millions of users. Beyond engineering, Kushal is a content creator with a community of 700,000+ developers across YouTube, LinkedIn, X, and Instagram, where he shares insights on AI, careers, and software engineering. He has spoken at international conferences including PyCon Japan 2025, PyCon Hong Kong 2022, and Xtreme Python Conference, and has delivered 80+ sessions and webinars across universities and industry. He has also hosted AI workshops and hackathons, helping developers and students build real-world AI and software systems. Kushal is an ACM ICPC’19 Regionalist and has mentored students through Google Code-In under the TensorFlow organization.

Related Speeches