From demo to production: 3 key lessons from developing a voice-to-voice AI agent
In this article...
- Moving from demo to production grade solution introduces real constraints: latency, orchestration, and reliability.
- Voice-to-voice interaction improves user experience, but only if backed by careful system design.
- Production systems require guardrails, asynchronous orchestration, and strong data governance.
- Voice is considered as personal data under GDPR, which shapes how the system can be built and improved
Voice-to-voice AI agents are emerging as a promising evolution for hotlines and helpdesks, enabling more natural interactions than traditional touch-tone or chat-based experiences.
In demos, this looks deceptively simple. In production, it becomes a multi-tasking system that must be well orchestrated to deliver a superior user experience and take full advantage of the technology.
At Amadeus, we recently developed a production grade AI call center voice agent in partnership with Microsoft. Unlike traditional Q&A assistants, this system is designed to handle travelers' requests end to end, with the ability to autonomously take actions, manage, and modify orders.
So, what did it take to move this voice-to-voice AI agent from demo to production?
AI voice agents that go beyond Q&A
To understand how it all works, let’s start with a typical voice-to-voice AI agent interaction, which can be broken down into five main steps:
- Intent detection and checking if it is in the AI agent scope
- Gathering the required information to retrieve the order
- Understanding the caller's request and proposing alternatives
- Capturing the caller's choice, confirming it, and performing the change
- Redirecting the caller to a human agent at any point in the process, either at the caller’s request or when the AI agent cannot handle the task
Because the agent has edit rights and can modify orders, it needs to be both autonomous and tightly controlled. This is where strong guardrails become essential.
To support that balance, we developed a complete QA framework and harness to validate the agent's behavior. The framework itself has been detailed by our colleagues at Microsoft in a separate post, so we won't go deep into it here.
What changes when you move from demo to production?
Building a production-grade voice AI system introduces several critical challenges. These are not only technical. They are also deeply tied to user experience and system reliability.
In this blog post, I will share three key lessons learned from the technical challenges we encountered while taking a voice-to-voice AI agent from pilot to production, and how we addressed them.
Lesson 1: Latency and asynchronous orchestration shape the user experience
In a live phone conversation without any visual support, even limited latency can break the conversational flow.
Human responders typically begin replying within around 300 milliseconds of a person finishing a sentence. If the system is slower, the experience feels unnatural.
To deliver low-latency and high-quality interactions, we relied on a voice-to-voice model instead of a traditional speech-to-text and text-to-speech pipeline. This reduces processing steps and improves responsiveness.
But voice-to-voice alone isn't enough. The latency introduced by backend systems also needs to be managed to ensure a smooth experience. To achieve this, all backend and API calls are handled asynchronously.
API calls can be slow, and in a voice interaction, long periods of silence aren't acceptable.
To avoid this, tool calls are executed asynchronously, and the call is triggered as soon as all the elements required to do it are gathered.
While waiting for the tool to complete the task, the agent continues the conversation, keeping the caller informed with updates by using set phrases such as: "I'm retrieving your booking" or "I’m working on your request, and I’ll have an answer soon."
When the task is completed and the outcome is ready, a callback is sent and the agent smoothly starts streaming the answer to the caller. This allows the conversation to continue without interruption.
We also introduced thresholds to monitor waiting times and avoid endless filler loops. In the end, if delays get too long, or if the task is too complex for an AI agent to handle, then the call can be automatically forwarded to a human agent.
Lesson 2: Voice UX must be concise and curated
Unlike on a website, presenting a list of alternatives to a traveler without visual support is a different kind of challenge for an AI voice experience.
Our agent required us to stick to a limited and concise list of options. First, we restricted the list based on the traveler's wishes, such as morning or afternoon flights, or direct flights only. Then, we kept the maximum at three options.
Each proposed option is then curated to include only the essential elements influencing the decision making. The rest of the details can be provided on request.
Therefore, this required extensive collaboration with UX experts to define how information should be structured and delivered.
In parallel, we introduced a parametric ranking system. It enabled service providers or the system owner to influence the proposed options through configurable parameters, while still satisfying caller expectations.
Lesson 3: Governance and privacy need to be designed in from the start
Dealing with voice in the context of AI is tricky because voice itself is personal data under GDPR rules. Limiting its usage to comply with the requirements became a must.
- Voice recordings, names, PNRs, and any personal data in the text transcripts were treated as PII (Personal Identifiable Information) data with restricted and controlled access.
- We developed an anonymization engine to remove PII data from both transcripts and voice recordings.
- For debugging and solution improvement, we relied primarily on the anonymized transcript of the call.
- When the tone of voice mattered, we used the anonymized voice stream instead.
Responsible AI considerations, including privacy, transparency, and human oversight, were built into the solution from the outset.
Ultimately, the original data stream remains accessible only for legitimate cases such as disputes, conflicts, or customer complaints. This ensures compliance while maintaining the ability to improve the system.
What production taught us about voice-to-voice AI agents
Moving from demo to production isn't about adding a conversational layer. It requires rethinking how systems interact in real time.
Latency management and asynchronous orchestration, user experience design, and data governance are all critical to delivering a smooth interaction.
Without a doubt, voice-to-voice is an important enabler of AI adoption. It offers a superior experience with low latency, higher quality interactions, and better empathy. But to make it work at scale in real-life use cases, the proposed solutions must overcome several challenges and adapt the ecosystem around them.
In the end, the quality of the conversation depends as much on system architecture as it does on the AI model itself.
In other words, a great demo proves the model works. A great production system proves the engineering does.
With our foundation as a system of record, our deep integration across the industry, and our ability to run at production scale, supported by strong governance through our AI Office and strategic partnerships, we are helping the industry embrace AI with integrity and trust.
That is the role Amadeus is built to play: orchestrating AI for the benefit of the travel industry and making the experience of travel better.
TO TOP