OpenAI has detailed the infrastructure behind GPT-4o's voice mode, which the company says achieves an average response time of 320 milliseconds, with a fastest observed response of 232 milliseconds — a range comparable to human conversation pace, according to a technical blog post from the company.
From a Three-Model Pipeline to One
Previously, voice assistants relied on a chained pipeline: a speech-to-text model like Whisper transcribed audio into text, a language model like GPT-4 generated a text response, and a separate text-to-speech model converted that response back into audio. OpenAI says this approach introduced latency at each step and discarded non-verbal information such as tone, emotion, and interruptions.
GPT-4o replaces this pipeline with a single end-to-end multimodal model trained natively on text, vision, and audio, processing audio input and generating audio output directly. OpenAI attributes the bulk of the latency reduction to this architectural change.
The Hardware Powering Real-Time Speech
OpenAI says it runs these workloads in its own data centers using NVIDIA GPUs, and describes several engineering challenges in scaling the system:
- Dynamic load — voice conversations require sustained, high-throughput processing rather than the bursty patterns typical of text interactions, requiring efficient management across the GPU fleet.
- Model colocation — GPT-4o's size requires splitting the model across multiple GPUs, which OpenAI says required custom networking and inference code to minimize inter-GPU latency.
- Resilience — the system is designed to reroute requests around hardware failures or network degradation without interrupting an ongoing conversation.
OpenAI frames its hardware, software, and model co-design as a core part of its strategy for scaling real-time AI systems.