How New Relic Made an AI Travel Agent Production-Ready
Building an AI demo is easy. Making it trustworthy in production is a completely different challenge. New Relic’s Principal Developer Relations Engineer Harry Kimpel published a detailed walkthrough on achieving production-ready AI agent observability. The guide shows how WanderAI, a fictional AI travel planning startup, moved from a polished demo to a fully monitored production service. The stack uses OpenTelemetry, the Microsoft Agent Framework, and New Relic.
Every AI demo eventually faces one critical question: how do you know it is actually working? Not just “is the server up”, but whether agents make good recommendations, respond fast enough, and produce genuinely useful output. A demo does not have to answer that. A production AI agent must.
WanderAI starts as a Flask web app. A user submits travel preferences, and a ChatAgent from the Microsoft Agent Framework generates an itinerary. It works well in testing. However, the moment an AI agent faces real users, the stakes change completely. Latency can come from many sources, the LLM, tool calls, cold starts, or network hops. Output is also non-deterministic. Failures hide silently, with the agent cheerfully building a plan on bad data.
Fortunately, getting baseline observability required just two lines of code. The Microsoft Agent Framework already emits traces, logs, and metrics that follow OpenTelemetry GenAI semantic conventions. Pointing an OTLP exporter at New Relic surfaced the full agent journey inside the APM and Services view, orchestration spans, tool calls, LLM round trips, and stitched logs all visible together.
Even so, auto-instrumentation only covers about 60 percent of the picture. The remaining 40 percent lives in custom code. Therefore, the team added custom spans around each tool function and the main route. They attached business-specific attributes like travel location, weather condition, and trip duration. Those attributes then become the dimensions for filtering, grouping, and alerting later.
Next, the team built the full operational layer. They created a performance dashboard tracking request rate, error rate, p95 response time, tool usage breakdown, and token-cost rollup. Alerts fire on an error rate above 5 events in 5 minutes and p95 latency over 25 seconds. Two SLOs define reliability promises, 99.5 percent availability and 95 percent of requests under 10 seconds, with a fast-burn alert at ten times the normal budget consumption rate. In addition, deployment markers from New Relic’s Change Tracking API link regressions directly to specific deploys, so the team knows exactly what changed.
Quality evaluation runs across three distinct layers. First, New Relic’s AI Monitoring captures every LLM interaction as a structured event. This unlocks model inventory, model comparison, and automated quality detection across all responses. Second, rule-based checks run inline on every itinerary, testing for structure, weather mentions, and required sections. Third, the system collects user feedback as a custom event joined to the original trace ID. As a result, a single NRQL query can compare model quality directly against real user satisfaction scores.
Security comes last but carries equal importance. Microsoft Foundry Guardrails handle platform-level detection of prompt injection, jailbreak attempts, and harmful content. Inside the app, a custom domain-aware detector runs before the agent starts, catching travel-specific abuse patterns that generic guardrails miss. Crucially, every security decision stays fully observable. Risk scores, matched patterns, and decisions all attach as span attributes in New Relic. Together, both layers reached over 90 percent detection on adversarial prompts with fewer than 10 percent false positives on legitimate travel requests.
Ultimately, production-ready AI agent observability means every interaction is traced, every output is evaluated, every model is comparable, and every security decision is visible. The entire WanderAI setup, instrumentation, dashboards, AI monitoring events, and security layers, is available as a free, open-source that runs in GitHub Codespaces in roughly three to five hours.
