ServiceNow and Hugging Face have introduced EVA, an open-source evaluation framework built to benchmark multi-turn voice AI agents across 50 realistic enterprise scenarios. By evaluating 20 commercial and open-source audio architectures, researchers discovered that voice agents optimized for task completion consistently delivered inferior user experiences. The framework fills a critical gap in current testing by jointly evaluating spoken dialogue quality alongside task resolution.
Dual Evaluation Architecture: EVA-A and EVA-X
Traditional voice benchmarks isolate specific capabilities like transcription or turn-taking, leaving end-to-end user interactions unexamined. EVA resolves this by simulating complete multi-turn audio calls using a bot-to-bot architecture built on the open-source Pipecat Python framework. A synthetic caller simulator interacts with the voice agent over live audio while a tool execution engine validates changes against scenario databases.
To capture the full scope of conversational performance, EVA divides scoring into two core dimensions:
- EVA-A (Accuracy): Combines deterministic checks for task completion with LLM-based faithfulness judges and Large Audio Language Model (LALM) checks for spoken speech fidelity.
- EVA-X (Experience): Assesses user experience using LLM judges that evaluate response conciseness, conversation progression, and timing.
Key Findings Across 20 Voice AI Systems
Testing across cascade architectures (speech-to-text to LLM to speech synthesis) and audio-native models revealed distinct systemic weaknesses in current voice platforms.
- Agents that perform well on task completion tend to deliver worse user experiences, and vice versa.
- Named entity transcription remains a primary point of failure, where mishearing a single character in a confirmation code causes catastrophic authentication dropouts.
- Multi-step enterprise workflows involving policy adherence—such as rebooking flights while keeping seat selections and baggage allocations—consistently cause agent execution failures.
- Behavioral consistency remains low across all tested models, marked by a substantial gap between peak single-run performance (pass@3) and repeatable multi-run success (pass^3).
Benchmarking Limitations
While EVA provides deterministic end-state validation, its qualitative evaluation relies heavily on LLM-as-judge scoring models. According to the publication by ServiceNow and Hugging Face, LLM judges introduce inherent stylistic biases and risk systematic evaluation bias when evaluating models developed by the same provider.