hen we shipped an AI agent powered by Cosmos DB's MCP server internally at Microsoft, it passed every test we threw at it—until real users found creative ways to break it. The agent would silently degrade: returning plausible-but-wrong query results, calling tools in inefficient loops, or burning through token budgets without completing tasks. Traditional monitoring showed green dashboards while users filed complaints. We needed observability designed for agents.
This talk presents the observability and evaluation stack I built to make agent failures visible, measurable, and catchable before users notice—combining OpenTelemetry instrumentation, LLM-as-judge evaluation, and automated regression gates in CI/CD.
Manik Khandelwal is an Engineer at Microsoft and a core maintainer of the Azure Cosmos DB Node.js SDK. He builds tools that make modern apps more scalable and intelligent—using AI, real-time data, and JavaScript.
