ServiceNow AI Research has published an evaluation benchmark revealing that leading automatic speech recognition (ASR) models experience severe performance drops when handling bilingual code-switched speech. Evaluated across four enterprise language pairs in HR and IT service scenarios, model Word Error Rates (WER) jumped significantly compared to monolingual baselines. The findings highlight critical operational vulnerabilities for global enterprise voice agents handling bilingual customer support and helpdesk requests.
Enterprise Voice Agents Struggle With Bilingual Input
Over half of the world's population speaks multiple languages, frequently switching between them mid-sentence during everyday conversation. To measure how automated systems handle this behavior, ServiceNow AI Research developed a dataset across four language pairs: Spanish-English, French-English, Canadian French-English, and German-English. Using an evaluation harness named AU-Harness, researchers evaluated utterances ranging from 12 to 40 words drawn from real-world IT helpdesk and human resources support scenarios.
Evaluating Top Speech Recognition Models
The study benchmarked seven prominent speech recognition engines using exact transcription accuracy (WER), semantic word error rate (SWER, judged by Gemma-4-31B), and downstream task comprehension (Answer Error Rate, or AER):
- ElevenLabs Scribe V2: Delivered the highest overall transcription accuracy and lowest error rates across both semantic and downstream metric tests.
- Google Gemini 3 Flash: Followed closely in transcription accuracy while outperforming competitors on semantic and answer error rates due to its Large Audio Language Model (LALM) reasoning.
- AssemblyAI Universal 3-Pro: Achieved top-tier raw transcription accuracy, though its relative standing shifted downward under semantic evaluation.
- Deepgram Nova 3 Multilang & Mistral Voxtral Small 24B-2507: Maintained mid-tier performance, though Deepgram registered elevated error rates on specific key details.
- Nvidia Parakeet TDT 0.6b V3: Placed lower overall but narrowed the gap on German-English audio.
- OpenAI Whisper Large V3 Turbo: Ranked lowest, with WER spanning 0.16 to 0.61, largely because unprompted code-switched audio triggered automatic translation rather than verbatim transcription.
How Language Switching Causes Downstream Errors
In enterprise environments, transcription errors propagate directly into downstream automation, leading to misrouted helpdesk tickets or incorrect policy answers. The study found that Large Audio Language Models like Gemini 3 Flash leverage language understanding to preserve essential context even when exact word transcription fails. Traditional models, by contrast, degraded substantially when forced to process mid-sentence language switches.