NVIDIA VoiceChat 11B: Full-Duplex Speech AI with 448 ms Turn-Taking
NVIDIA released NemotronLabs VoiceChat 11B, an open full-duplex speech model with 448 ms turn-taking latency and live tool calling. Weights are public but research-only.
NVIDIA has released NemotronLabs VoiceChat 11B, an open 11-billion-parameter model that handles real-time voice conversation in a single network instead of chaining three separate systems. It listens while it speaks, supports user interruptions, and can call external tools mid-conversation without going silent. Measured turn-taking latency sits at 448 ms on Full-Duplex-Bench 1.0. Weights and a container are publicly available under a permissive license, though NVIDIA itself labels the checkpoint research-only and documents several known failure modes.
What happened
| Detail | Value |
|---|---|
| Model name | NemotronLabs VoiceChat 11B |
| Parameters | 11 billion |
| Smooth turn-taking latency (TOR 0.82) | 448 ms on Full-Duplex-Bench 1.0 |
| User-interruption take-over rate | 1.00 at 480 ms |
| Training data | ~550,000 hours of audio |
| GPU requirement | One GPU with at least 80 GB VRAM (A100, H100, RTX 6000 Pro, B200) |
| License | OpenMDW-1.1 (permissive) |
| Readiness status | Research only (per NVIDIA) |
Most production voice assistants today are cascaded pipelines: audio goes into an automatic speech recognition (ASR) model, text goes into a large language model, and the output text goes into a text-to-speech (TTS) synthesizer. Each hand-off adds latency and a new failure point. VoiceChat 11B collapses all three steps into one hybrid Mamba/Transformer network that processes a continuous 16 kHz audio stream and emits 22.05 kHz speech directly.
The architecture pulls from three existing NVIDIA components. A Fast Conformer encoder from Nemotron-Speech-Streaming-En-0.6b handles the incoming stream. The Nemotron Nano v2 LLM backbone converts audio tokens to text tokens. An NVIDIA TTS decoder renders those into audio codes. A fourth, new output channel handles tool calls separately from the audio stream.
How tool calling works without dead air
When the model decides to call an external tool, it emits a <TOOLCALL> block on the side channel. Your backend returns results in a <TOOL_RESPONSE> block. The practical problem with this pattern is silence: the user hears nothing while an API runs. VoiceChat 11B addresses this with operator-defined on-hold messages. You assign a spoken line to each tool in advance; the model reads it aloud the instant it generates the trigger text, bridging the gap.
NVIDIA sets clear limits here. The model supports a maximum of five tools per session, cannot reliably call multiple tools at the same time, and does not allow user interruptions during tool execution. System prompts and tool responses must be ASCII-only and phrased so the TTS decoder can read them naturally.
Benchmark results on spoken tool calling (AU Harness BFCL-v3) show 58.5% accuracy on simple calls, 62.5% on multiple, 42.5% on parallel, and 27.5% on parallel-multiple tasks. The model reached 89.6% on irrelevance detection (knowing when not to call a tool) and a 56.1% average. On Full-Duplex-Bench v3, tool selection accuracy is 82.5%, argument accuracy 44.2%, and pass@1 33%. NVIDIA reports the model ranks second among open full-duplex models on both VoiceBench and Full-Duplex-Bench 1.0.
What are the known failure modes?
NVIDIA documents these problems explicitly in the repository:
- Audio context is capped at two minutes per session.
- After several turns, the model can degrade into non-recoverable gibberish.
- The model sometimes continues speaking after a turn should have ended (runaway self-talk).
- User speech can be dropped or mis-transcribed.
These are not speculative risks; they are stated by the team releasing the model. This is why NVIDIA is explicit that the checkpoint is for research, not production.
Who can run it and who can’t
The weights and a container image are both public. To run the model you need one x86_64 Linux machine with a GPU carrying at least 80 GB of VRAM: an A100, H100, RTX 6000 Pro, or B200. There is no hosted API and no inference provider currently serving the model, so teams without direct GPU access have no way to evaluate it today.
According to NVIDIA, the target use cases span contact centers, automotive in-cabin assistants, retail and drive-thru ordering, telecom IVR (interactive voice response) modernisation, game NPC dialogue, and accessibility tooling.
Why it matters
Cascaded voice stacks have a well-known problem: latency compounds across three separate models, and if any one of them hiccups, the conversation breaks. A single unified model can, in principle, be faster and simpler to operate. At 448 ms for turn-taking, VoiceChat 11B is meaningfully faster than most production cascaded pipelines, and the barge-in support (1.00 take-over rate) means users are not stuck waiting for the agent to finish its sentence before they can speak.
The tool-calling addition is significant because it is the first time an open full-duplex model has done this at all. For businesses building voice front-ends over internal APIs, the pattern of emitting calls on a side channel while speaking a hold message is directly practical. Teams exploring AI integration for their products will want to watch this architecture carefully as it matures.
Our take
The honest framing here is: this is an impressive research artifact with real production ambitions, but it is not production-ready yet. The two-minute context ceiling and gibberish degradation are not edge cases you can engineer around; they are fundamental session constraints. For a contact center call that might run eight minutes, that is a blocker.
What is worth taking seriously is the architecture direction. Collapsing ASR, LLM, and TTS into one model is the right long-term path. The latency numbers are genuine improvements, and the tool-calling design is pragmatic. We have seen similar “research-only” releases from labs mature into production-ready systems within 12 to 18 months. The benchmark scores on argument accuracy (44.2%) and pass@1 (33%) also signal that tool use is early and will need significant work.
If you have GPU access and a voice AI pilot planned for late 2025 or 2026, it is worth cloning the repo and running the benchmarking harness now. If you are evaluating a customer-facing voice deployment today, keep your cascaded stack and watch for v2.
We cover developments like this regularly on the Lumien AI news desk. If you are scoping a voice or automation build, get in touch and we can help you separate what is ready from what is still research.
Frequently asked questions
What GPU do I need to run NVIDIA VoiceChat 11B?
You need a single GPU with at least 80 GB of VRAM on x86_64 Linux. Compatible options listed by NVIDIA include the A100, H100, RTX 6000 Pro, and B200. There is currently no hosted API or cloud inference provider for this model.
Is NVIDIA VoiceChat 11B ready for production use?
No. NVIDIA labels the checkpoint 'research purposes only' and documents known failure modes including a two-minute audio context ceiling, degradation into gibberish after several turns, runaway self-talk, and dropped user words.
How does VoiceChat 11B handle tool calling?
The model emits tool calls on a separate output channel as a TOOLCALL block. Operators define an on-hold message per tool that the agent speaks immediately while waiting for the API response, so the conversation does not go silent. The model supports up to five tools per session.
What is the latency of NVIDIA VoiceChat 11B?
On Full-Duplex-Bench 1.0, the model achieves smooth turn-taking at 448 ms with a take-over rate of 0.82. For user interruptions, the take-over rate is 1.00 at 480 ms, meaning the agent reliably yields when a user speaks over it.
