TTS
Text-to-Speech (TTS) is a speech synthesis technology that automatically converts written text into spoken audio output. Core capabilities include linguistic analysis and text processing, generation of natural speech waveforms via speech synthesis engines, and support for multiple languages and voice types. TTS is broadly applied in accessibility assistance, intelligent assistants, navigation systems, and educational software, and constitutes a foundational technology for human-computer interaction.
Core Facts
GPT-4o's end-to-end voice architecture incorporates audio as discrete tokens into a unified Transformer processing framework, handling the full listen-think-speak pipeline within a single model.
There are approximately 7,000 languages in the world, and the vast majority are low-resource languages.
The new generation of voice interaction has shifted to an end-to-end multimodal model architecture that understands and generates directly at the speech signal level, compressing latency and preserving paralinguistic information.
Timeline (last 90 days)
By wrapping target-language vocabulary in lightweight XML tags (such as a target tag) during streaming output, and having the client split by tag and route to the TTS session for the corresponding locale, in-sentence accent switching can be achieved, solving the problem of mixed-script vocabulary being mispronounced by English TTS.
The traditional voice pipeline is a serial "listen—think—speak" process and cannot listen while speaking; implementing barge-in requires low-latency streaming ASR and interruptible TTS.
When selecting components, the metrics Reimer focuses on include: the audio framework's input and output latency, STT's confidence time, the LLM's time to first sentence (a combination of time to first token and tokens per second), and TTS's time to first audio (which must be measured up to the point where the user actually hears sound).
AstroBeam's parallelization approach is: without waiting for STT to fully confirm, start the LLM early as soon as there is some confidence that the user has probably finished speaking and that the transcription is roughly correct; on the LLM side, send the first speakable chunk to TTS immediately; and have TTS stream it out to the user.
Speech-to-speech models are less controllable than pipeline architectures on safety review, because LLM output can be intercepted and validated before being fed to TTS, whereas speech-to-speech models are hard to verify before output; voice style and intelligence level are also locked into the model's presets.
Per-character billing directly corresponds to the TTS model's actual computational load (input text length); when usage fluctuates, cost floats with actual calls, making it better suited for agent scenarios with highly variable conversation lengths.
ASR (automatic speech recognition) and TTS (text-to-speech) form the ears and mouth of a voice agent, while the LLM is the brain.
ASR requires streaming processing to reduce first-character latency, while TTS requires streaming synthesis to reduce first-packet latency.
AssemblyAI's main business is on the speech recognition (ASR) side.
Some TTS vendors bill separately for concurrent connections and API call counts.
40 more timeline events
All Facts (20)
GPT-4o's end-to-end voice architecture incorporates audio as discrete tokens into a unified Transformer processing framework, handling the full listen-think-speak pipeline within a single model.
90%VerifiedThere are approximately 7,000 languages in the world, and the vast majority are low-resource languages.
90%VerifiedThe new generation of voice interaction has shifted to an end-to-end multimodal model architecture that understands and generates directly at the speech signal level, compressing latency and preserving paralinguistic information.
80%VerifiedTraditional voice AI systems adopt a cascaded architecture that splits voice processing into independent modules such as ASR, NLU, dialogue management, and TTS, with information loss and latency accumulation between each module.
80%VerifiedThe three-stage architecture of traditional voice interaction (ASR+LLM+TTS) typically has a total latency of 1–3 seconds.
80%VerifiedModern voice cloning solutions typically use a speaker encoder to map a voice into a high-dimensional embedding vector, which is then injected into the acoustic model during the TTS decoding stage to control the output timbre.
80%VerifiedThe training data of most speech models is primarily monolingual, and mixed-language corpora are relatively scarce.
75%VerifiedTime to First Audio (TTFA) refers to the time elapsed from text input to the output of the first frame of playable audio, and is a core metric for measuring the response speed of streaming TTS systems.
75%VerifiedA complete voice AI interaction involves three serial processing stages: STT, LLM inference, and TTS.
75%VerifiedTraditional voice dialogue systems consist of three independent modules connected in series—ASR (Automatic Speech Recognition), LLM (Large Language Model), and TTS (Text-to-Speech)—each trained separately and executed sequentially.
75%VerifiedFor an AI voice assistant to achieve full duplex, four modules must run simultaneously—automatic speech recognition (ASR), natural language understanding (NLU), language model inference, and text-to-speech (TTS)—while also handling voice activity detection (VAD) in real time.
75%Verified传统语音AI普遍采用级联式架构，先用ASR将语音转文字，再交给语言模型处理，最后用TTS合成输出
75%VerifiedTraditional speech-to-speech translation uses a three-module cascaded architecture of ASR → MT → TTS, which suffers from error accumulation and compounded latency.
75%VerifiedTraditional AI voice interaction systems use a pipeline architecture of four separate modules—ASR, NLU, NLG, and TTS—connected in series, with a combined total latency typically between 1 and 3 seconds.
75%VerifiedEnd-to-end native architectures directly process audio input and output audio without relying on intermediate conversion steps, preserving richer acoustic features.
70%VerifiedAI real-time voice translation systems simulate the simultaneous interpretation process through a pipeline collaboration of three modules: Automatic Speech Recognition (ASR), Neural Machine Translation (NMT), and Text-to-Speech (TTS).
70%VerifiedODS integrates voice interaction capabilities, supporting speech-to-text (STT) and text-to-speech (TTS)
65%VerifiedTraditional cascaded architectures generate video before adding audio, which leads to problems such as audio-video desynchronization, lip-sync mismatches, and ambient sound being out of step with on-screen actions.
65%VerifiedAchieving an interruptible real-time conversational experience involves a technical pipeline covering voice activity detection (VAD), automatic speech recognition (ASR), intent understanding and context management, retrieval and generation, TTS synthesis, and seamless continuity.
65%Verified800毫秒是语音代理端到端延迟的业界公认阈值，人类面对面对话回应延迟通常在200-500毫秒之间
65%