Real-time voice AI: talk to RafayGen with the voice orb

The difference between a voice assistant that feels like a conversation and one that feels like a walkie-talkie is almost entirely latency, and latency in a voice AI system is a budget spent across four stages that each want more of it than you have.

This article explains what those stages are, where the delay actually comes from, why interrupting the assistant is technically harder than it looks, and what you can do at your end to make voice AI work better.

The four stages, and the budget

A spoken exchange with an AI runs through: voice activity detection, speech-to-text, the language model, and text-to-speech. Human conversation tolerates roughly 200 to 300 milliseconds of gap before a pause starts to feel like hesitation. Two seconds feels broken. So the whole chain has well under a second of real budget.

Voice activity detection decides when you have stopped talking. This is the stage nobody thinks about and it frequently dominates perceived latency, because a naive implementation waits a fixed silence window — often 800 milliseconds or more — before deciding your sentence ended. That delay is spent before any AI has been asked anything. Tune it too short and the system interrupts you mid-thought; too long and every reply feels sluggish. There is no setting that is right for all speakers, which is why some systems feel snappy for one person and slow for another.

Speech-to-text is the fastest stage on modern infrastructure — a short utterance transcribes in a couple of hundred milliseconds on a specialised inference provider. Streaming transcription, which emits words as you speak instead of waiting for the end, hides most of this cost entirely.

The language model is the expensive stage, and the number that matters is not total generation time but time-to-first-token. If the first word can be spoken while the rest is still being generated, the user experiences the first-token latency, not the full one. This is why streaming is not a nicety in voice systems; it is the architecture.

Text-to-speech adds its own delay, and the same trick applies: synthesise and start playing the first sentence while the model is still writing the second. Waiting for a complete answer before speaking any of it roughly doubles the perceived wait.

Why barge-in is the hard part

Interrupting the assistant mid-sentence — barge-in — is what separates a real conversation from taking turns at a microphone. It is also where naive implementations break, and the reason is acoustic rather than conceptual.

The simple design is half-duplex: while the assistant is speaking, the microphone is closed. This is easy, it guarantees the system never hears itself, and it makes interruption structurally impossible. You have to wait for it to finish, which is exactly the walkie-talkie feeling.

The alternative is to keep the microphone open while audio is playing, which means the system now hears its own voice through your speakers and must distinguish that from you. On headphones this is manageable. On a laptop or phone speaker it requires echo cancellation, and the residual leak still looks like speech to a voice detector. Get it wrong and the assistant interrupts itself, hears the interruption, and spirals.

The workable middle is a talk-over detector: keep listening during playback, but require a higher confidence threshold and a longer sustained-speech window before treating incoming audio as a genuine interruption. Then stop playback immediately, discard the rest of the pending answer, and resume listening. Done well, you can cut the assistant off mid-word the way you would cut off a person.

Why Urdu voice is harder than English voice

Every stage above is trained predominantly on English audio, and the degradation in other languages is not uniform.

Transcription is the weakest link. Urdu speech recognition has far less training data than English, and Pakistani English — which is what most bilingual speakers actually use for technical vocabulary — is itself an under-represented accent. A specific and common failure is a model that transcribes Urdu into Devanagari instead of the Urdu script, because Hindi and Urdu are close enough acoustically that a multilingual model can pick the wrong output script entirely.

Code-switching compounds it. "Mujhe iska summary bana do in bullet points" contains two languages and a script decision, and systems that identify one language per utterance handle it badly.

Synthesis has the opposite problem: Urdu text-to-speech is intelligible but the prosody is often flat, and it mispronounces English loanwords embedded in Urdu sentences — which is most sentences.

Getting better results at your end

• Use headphones or earbuds. This single change removes the echo-cancellation problem entirely and makes interruption reliable.
• Speak in complete phrases rather than trailing off. Voice detection segments on silence, so a sentence that fades out gets cut early.
• Do not spell things out letter by letter. Recognition is worse at letters than at words; say the word and correct it afterwards if needed.
• Give numbers and names slowly and check them. This is where transcription errors are both most likely and most consequential.
• For a long question, say it as two short ones. Long single utterances accumulate transcription errors and give the model a noisier input.
• If the assistant consistently mishears a particular term, type it once in the chat. Establishing it in context makes the model far more likely to interpret the garbled transcription correctly next time.

What voice is genuinely better for

Voice is not a universally superior interface and it is worth being honest about where it wins. It is better for thinking out loud, for practising a spoken answer before an interview or a viva, for questions asked while your hands are busy, and for anyone who finds typing — particularly typing Urdu — slow or painful. For an older relative who speaks Urdu fluently and never learned to type it, voice is not a convenience, it is the entire difference between having access to this technology and not.

It is worse for anything you need to keep, edit or verify. You cannot skim spoken output, you cannot copy a code block out of it, and re-listening to find one detail is far slower than re-reading. The useful pattern is mixed: talk to explore, read to work.

How RafayGen's voice orb is built

The orb runs the pipeline described above in the browser with no app to install: streaming speech-to-text, a low-latency chat lane, and text-to-speech that begins speaking the first sentence while the model is still writing the rest.

Barge-in is implemented as a talk-over detector rather than a half-duplex gate, for exactly the reason given earlier — the half-duplex version made interruption impossible, which was the single biggest complaint about the first version of it. It shares your account and history with the chat workspace, so a topic you were typing about is still there when you switch to speaking, and it follows you between Urdu and English mid-conversation.