The Three-Way Trade-Off of Voice Agents: Balancing Latency, Intelligence, and Cost

In real-time voice interaction, developers often find themselves caught in a seemingly unsolvable triangle: to get low latency, you have to sacrifice intelligence or cost; to go cheap, you have to accept slower responses. Devin Reimer, founder and CEO of AstroBeam, drew on hands-on experience with two fully voice-driven games (Stellar Cafe and Robo Detective) to break down the trade-off logic between latency, intelligence, and cloud economics for virtual voice agents in a technical talk. His core conclusion: faster isn't always better when it comes to latency, costs must be accounted for from day one, and speech-to-speech models are not cost-effective for game scenarios at this stage.

Latency: 1.5 Seconds Is the Comfort Line, but Consistency Matters More Than Peak Speed

Reimer defines end-to-end latency as the time from "the user's last phoneme ends to hearing the returned audio." After extensive player testing, they found that about 1.5 seconds is the threshold at which users stop actively complaining about speed; and when pushed further down to around 1.2 seconds, the conversation takes on a more natural "supernormal" rhythm. But going below 1 second can actually feel eerie—too fast, and users feel it's unrealistic.

The more critical finding is that perceived latency depends heavily on context. In testing, the same response duration could be rated as "fast" by one user and "slow" by another, because users have completely different expectations for the response time to "what's your name" versus "what's the meaning of life." The first turn of a conversation is usually expected to be faster, while a series of short back-and-forth exchanges raises expectations for subsequent turns. Therefore, there is no universal millisecond number; the engineering goal should target a comfortable range of around 1.5 seconds.

Once you're in this range, the real battleground shifts to consistency. Reimer emphasizes that if 90% of responses are 1.5 seconds but 10% jump to 3 seconds, the user experience doesn't just "get worse"—it "collapses." Therefore, when selecting components, focus on P99 latency rather than peak speed: consistent speed matters far more than occasional bursts of extreme speed.

P99 latency is a percentile metric in the latency distribution: sort all requests' end-to-end latencies from lowest to highest, and the value at the 99th percentile is P99—meaning 99% of requests have latency no greater than that value, and the remaining 1% are slower. Related metrics include P50 (the median, representing typical experience) and P95. In real-time voice interaction, averages or best-case values can be skewed by a few extremely fast requests, masking long-tail degradation; and users are extremely sensitive to occasional multi-second stalls—one bad experience can destroy trust in a product's "real-time" nature. Therefore, engineering practice typically uses P99 as the constraint line for the latency budget, paired with P50 to assess normal performance, together characterizing "consistently fast" rather than "occasionally extremely fast." Monitoring P99 also helps pinpoint sources of tail latency, such as cold starts, model queuing, or network jitter.

Pipeline Architecture vs. Speech-to-Speech: Cost Is the Deciding Factor

The standard voice agent pipeline consists of three stages: speech-to-text (STT), large language model (LLM), and text-to-speech (TTS). Serial execution accumulates latency, so people often ask: why not just use a speech-to-speech model? After all, it only runs one stage and is theoretically faster.

Reimer's measured answer: speech-to-speech models are typically 3 to 5 times more expensive than parallelized pipelines, which is a "deal-breaker" for cost-sensitive scenarios like games. In addition, pipeline architectures offer more controllable safety moderation—LLM output can be intercepted and validated before being sent to TTS; whereas speech-to-speech models make it difficult to verify content before output. Voice style and intelligence level are also locked into the model's presets and cannot be switched per use case. The only clear advantage of speech-to-speech models is understanding user tone (e.g., sarcasm), but Reimer believes this gap will narrow as STT capabilities evolve.

Key Techniques for Parallelization

AstroBeam's approach is to break the pipeline apart and parallelize as much as possible: rather than waiting for STT to fully confirm, as soon as there's sufficient confidence that "the user may have finished speaking" and "the transcription is roughly correct," the LLM is started early; if the judgment was wrong, the cost of canceling and re-running is low because the context is usually already cached and few tokens have been generated. On the LLM side, rather than waiting for the complete response, as soon as the "first speakable chunk" (usually one sentence) is available, it's immediately sent to TTS; TTS then streams it to the user. Since audio playback almost always takes longer than subsequent generation, the second chunk is ready while the user is listening to the first sentence—invisible to the user.

When selecting components, Reimer focuses on these metrics: the audio framework's ingress/egress latency, STT's "confidence time" (the slower of transcription confidence and turn-end confidence), LLM's "first-sentence time" (a combination of time-to-first-token and tokens per second), and TTS's "first-audio time" (must be measured to the point where the user actually hears sound, excluding leading silence).

Speech-to-speech models (also called end-to-end speech models) refer to a single model that directly takes user audio as input and produces synthesized speech as output, without explicitly going through separate text transcription and LLM inference stages. Representative work includes GPT-4o's native voice capabilities, Moshi, and others. Their appeal lies in eliminating the accumulated latency of chaining multiple components and preserving paralinguistic information such as tone, pauses, and emotion. But the costs are: inference is typically significantly more expensive than text LLMs (audio tokens have greater representation density and generation length), output content is difficult to review word-by-word before playback, and voice timbre, style, and "intelligence level" are bound inside the model, making it hard to swap or fine-tune per scenario. Therefore, in game scenarios with high demands for cost, safety, and character controllability, most teams still choose decomposable, swappable pipeline solutions.

Time to First Token (TTFT) and Tokens Per Second (TPS) are two core performance metrics for LLM inference. TTFT measures the time from when a request is sent to when the model produces its first output token, primarily affected by the prefill phase—the time the model needs to process the entire prompt and build context representations; longer prompts mean slower prefill. TPS measures the rate of token-by-token generation once in the decode phase, determining the total duration of the complete response. For voice agents, TTFT directly determines "first-sentence time," because TTS can only be started once the first sentence is available; TPS affects whether subsequent sentences can be ready before the user finishes listening to the previous one. Together they determine whether the streaming experience is seamless, and they should be measured separately during component selection to avoid looking only at total latency.

Intelligence: Not Just Good Enough for Users, but Good Enough for Developers

Reimer specifically points out an easily overlooked dimension: model intelligence must satisfy both end users and the development team. When using early LLaMA models, although users could barely get through, the development process was extremely fragile—the model frequently produced unreasonable responses, and the team had to invest significant engineering effort patching things up. For products that need rapid iteration, this hidden cost is enough to derail progress.

Intelligence isn't just about the LLM. On the STT side, word error rate matters, but what matters more is the "closeness" of errors—if the error is wildly off, even with context the LLM struggles to correct it; if the error is close, the LLM can often guess the intent correctly. Platform characteristics also affect component selection: XR devices have microphones close to the mouth, yielding high audio quality; while mobile apps used on noisy streets will show completely different model performance.

On the TTS side, intelligence manifests in naturalness, expressiveness, and speed. Reimer's testing found that the more expressive the agent, the more intelligent users perceive it to be; and response speed is also directly associated by users as a signal of intelligence—fast replies sound smarter.

The LLM side is the most complex: it needs speech understanding, world knowledge grounding, strong rule-following, low hallucination, character consistency, and the ability to handle prompt bloat. In a game world, "there's an apple on the table" when there actually isn't directly breaks the experience; and as development progresses and play history accumulates, prompts grow longer and longer, and some models degrade noticeably in quality with long prompts—this must be validated during component selection.

Prompt bloat refers to the continuous lengthening of system prompts and context fed to the LLM as the product iterates and conversation history accumulates, leading to increased inference costs, slower time-to-first-token, and potentially causing the model to "forget the middle" of long contexts—key instructions get buried in large amounts of text, and rule-following and character consistency decline. In game voice agents, world-building settings, character personas, quest states, and play history all continuously pile into the prompt, making bloat almost inevitable. Common mitigation techniques include: summarizing and compressing history, retrieving and injecting key memories by relevance, managing stable rules and dynamic state in layers, and stress-testing models with prompts close to production length during component selection to validate the quality degradation curve under long contexts.

Cost: Do the Math First, Then Build

Reimer uses concrete numbers from two games to demonstrate the cost model. Take Stellar Cafe: average playtime is about 2 hours, voice agent operating cost is about $1.20/hour/user, the game is 100% voice-driven, and users interact throughout. At a one-time purchase price of $10, after store fees, revenue per user is about $4.59, and voice cost is about $2.40—the economic model works. He mentions that the price was previously $15 and was reduced to $10 by driving down server costs, and he explicitly states they won't do subscriptions because "it has to work under a buy-to-play model, otherwise it can't scale."

As a comparison, a narrative game of 5 to 9 hours (average 7 hours) with 60% voice usage, priced at $20, has voice costs of about $5 and profit per user of about $9—also viable. But Reimer also gives a counterexample: if you make a free-to-play game where users play an average of 10 hours, 100% voice-driven, with revenue of $3 per user, then you lose $10 per user—the economic model completely falls apart. His advice is direct: don't wait until you're done to do the economics.

Local Models: The Theoretical Cost Cure, Today's Engineering Challenge

Running models locally can completely eliminate server bills, but Reimer lists several hurdles: computationally, running multiple models simultaneously (four in his case) without maxing out the GPU or affecting game rendering; speed-wise, local models are often slow enough to feel more like "hints" than real-time conversation; quality-wise, locally you can typically only run smaller LLMs and more basic TTS, which may be neither good enough for users nor for developers; portability-wise, optimizations tuned for one GPU are hard to migrate to new platforms. Hybrid approaches (local when possible, cloud otherwise) aren't as cost-effective as imagined either: even small models on the cloud cost only a few cents, so unless the entire stack is localized, the benefit is limited; and running different models locally versus in the cloud significantly increases development and min-spec adaptation costs. His judgment: local models will "very likely eventually win" for price-sensitive applications, but today they remain limited to specific use cases (such as command-style interaction).

Open Weights vs. Closed Weights: Stability and Lifecycle

On the LLM side, Reimer leans toward open-weight models, with controllability as the core reason. Closed frontier models are more intelligent, but have several structural problems: first, speed may degrade over their lifecycle (due to growing user numbers, hardware migrations, etc.); second, closed services are often optimized for batch processing and asynchronous tasks, not necessarily for P99 latency, whereas on the open side you can choose hosting providers specifically optimized for this metric; third, closed models are often retired after about a year, and if a product is expected to be in production for three years, it may go through three or more model changes, each migration potentially costing weeks or even months—and "a stronger model" isn't necessarily better for your use case; fourth, new versions of closed models often come with higher pricing, potentially invalidating the economics that worked at launch after migration.

AstroBeam's current production stack is: LiveKit as the audio agent framework, DeepGram for STT, Gemma 4 31B as the primary LLM (hosted on LiveKit), in-world AI text-to-speech 2 for TTS, plus a custom agent harness. Reimer emphasizes that Gemma 4 31B is the best combination of speed, intelligence, and cost, and a major unlock for development efficiency, but the team still mixes multiple open and closed models for tasks like memory generation, safety moderation, and customization—"choosing the most suitable model for each specific task."

The handling of safety models is also worth noting: the traditional approach embeds safety moderation into the LLM call, adding latency. AstroBeam instead triggers an independent safety model in parallel when the first speakable chunk is generated; since the safety model is smaller and faster, it almost always completes before audio playback, achieving zero additional latency; a full moderation pass runs again after the complete response is generated. Because the moderation target is a whole sentence rather than scattered tokens, they can use a larger model with stronger context awareness—for example, allowing lines like "off with their heads" in a medieval game that fit the world-building.

Conclusion

Reimer's framework can be condensed into one sentence: aim for stable latency rather than fast latency, intelligence must serve both users and developers, and costs must be calculated before writing the first line of code. The engineering difficulty of voice agents is amplified in game scenarios, but conversely, solutions that work in games often hold up when migrated to other voice-first applications. This field is still very early, but the economic models are starting to work—the remaining question is whether more teams can find their own balance point among latency, intelligence, and cost.

One-line Summary

Voice agents need to aim for stability across 1.5-second latency, usable intelligence, and calculable costs; speech-to-speech models aren't cost-effective at this stage.

Paragraph Summary

AstroBeam founder Devin Reimer, drawing on hands-on experience with two fully voice-driven games, breaks down the trade-offs of virtual voice agents across latency, intelligence, and cost. On latency: about 1.5 seconds is the comfort line where users stop complaining, 1.2 seconds feels more natural, and below 1 second feels eerie; perceived latency depends on context, and once in range, consistency (P99) matters more than peak speed. On architecture: speech-to-speech models are faster but cost 3 to 5 times more than parallel pipelines, with poorer safety moderation and style controllability—not cost-effective for games. Parallelization techniques include starting the LLM before STT is fully confident, sending the first speakable chunk to TTS immediately, and running safety models in parallel for zero additional latency. Intelligence must satisfy both users and developers, and prompt bloat must be guarded against. Costs must be calculated before development; buy-to-play works, but free with long play sessions may lose money; local models are the long-term direction but are currently constrained by compute, speed, quality, and portability.