hen we shipped an AI agent powered by Cosmos DB's MCP server internally at Microsoft, it passed every test we threw at it—until real users found creative ways to break it. The agent would silently degrade: returning plausible-but-wrong query results, calling tools in inefficient loops, or burning through token budgets without completing tasks. Traditional monitoring showed green dashboards while users filed complaints. We needed observability designed for agents.
This talk presents the observability and evaluation stack I built to make agent failures visible, measurable, and catchable before users notice—combining OpenTelemetry instrumentation, LLM-as-judge evaluation, and automated regression gates in CI/CD.
Manik Khandelwal is an Engineer at Microsoft and a core maintainer of the Azure Cosmos DB Node.js SDK. He builds tools that make modern apps more scalable and intelligent—using AI, real-time data, and JavaScript.
Running 4 to 8 parallel agent coding sessions sounds like a superpower. It nearly broke me.
I was spending more time switching context to check what each agent had done than I would have spent writing the code myself.
Manual testing, staging deployments, code reviews just to understand what the agent had built: I had become the bottleneck. The agents were fast. I wasn't.
This talk is about how our team redesigned the workflow around one principle: the agent should be able to prove its own work. That means agents that deploy themselves, run their own test suites, watch CI and fix failures, and produce PRs with enough evidence that a reviewer can approve with confidence, without reading every line. I'll share what it concretely took to get there, and where humans still need to stay in the loop and why.
I'm a Senior Software Engineer at Datadog, working on AI-powered developer tooling. Before that, four years leading R&D at IQVIA France, and two decades building platforms at startups across fintech and insurtech. I've been obsessed with developer workflows since long before AI made... Read More →
