ServiceNow AI has open-sourced EVA-Bench 2.0, expanding its voice agent benchmark to include 121 enterprise tools and 213 evaluation scenarios across three distinct industries. EVA-Bench 2.0 provides a 4x increase in scenario coverage across 121 tools and 213 scenarios, open-sourced under the MIT license. Enterprise developers can now evaluate agent tool orchestration using standardized datasets covering airline, IT, and healthcare domains.
Synthetic Generation via SyGra
ServiceNow engineered the benchmark using SyGra, a graph-based synthetic generation pipeline powered by OpenAI's GPT-5.4. SyGra jointly constructs the user goal as a decision tree, the initial scenario database state, and the expected ground-truth terminal database state. This joint approach prevents silent inconsistencies that typically corrupt evaluation signals when components are generated separately.
Every scenario incorporates realistic enterprise constraints such as multi-factor authentication, non-satisfiable user goals, and adversarial inputs. The dataset also requires exact action completion proofs, like confirmation numbers or case IDs, rather than simple verbal commitments from the agent.
Multi-Domain Workflows and Hardened Validation
To assess model versatility, ServiceNow expanded the benchmark from a single enterprise domain into three distinct sectors:
- Airline Customer Service (CSM): Features 50 scenarios testing flight adjustments, standby lists, and booking workflows.
- Enterprise IT Service Management (ITSM): Contains 80 scenarios covering account lockouts, system access, and technical support.
- Healthcare HR Service Delivery (HRSD): Comprises 83 scenarios grounded in US health administration, FMLA policies, and NPI identifiers.
ServiceNow subjected every scenario to automated schema checks, LLM-based consistency validation, and manual expert reviews. To ensure fairness, the team verified that every scenario was solvable by at least one of three frontier models: OpenAI GPT-5.4, Google Gemini 3.1 Pro, or Anthropic Claude Opus 4.6. However, because tests were conducted on text transcripts directly, the benchmark currently evaluates text-based tool orchestration rather than full end-to-end audio pipeline performance.
What's Next
ServiceNow has made all three datasets publicly available on Hugging Face alongside an evaluation framework and public leaderboard. The team is also developing a multilingual extension featuring localized names, addresses, and phone numbers to evaluate voice agents in non-English enterprise deployments. As enterprise adoption of AI agents accelerates, standardized benchmarks like EVA-Bench 2.0 offer crucial performance baselines for production deployments.