ElevenLabs Lands on Vercel AI Gateway: 14 Voice Models in One Place

Vercel has partnered with ElevenLabs to bring 14 ElevenLabs audio models to AI Gateway. Developers can now use a single gateway API key to access the full range of audio capabilities—from speech-to-text and streaming speech synthesis to real-time voice agents, music generation, sound effects, and multi-speaker dialogue. For teams building voice-interactive applications, this means no more integrating with multiple audio providers separately.

One Gateway, Six Categories of Audio Capabilities

The ElevenLabs models supported by AI Gateway cover six major categories: speech-to-text (transcribing recordings or adding real-time captions to spoken content), streaming speech synthesis (audio starts playing while the language model is still generating its response, so users don't have to wait for the complete answer), real-time voice agents (users converse with an application in a live session and hear spoken responses, ideal for customer service hotlines, tutoring tools, and in-app assistants), music generation (turn text descriptions into finished songs or instrumental tracks), sound effects generation (create game and UI sound effects from short descriptions, or seamlessly looping ambient audio), and multi-speaker dialogue (voice an entire conversation in a single call, with a different voice for each speaker).

These models are accessible through a single gateway API key. An application or agent can first transcribe what the user said, run inference with any of hundreds of models, then respond using an ElevenLabs voice—and can also pair it with generated music and sound effects for a soundtrack. All requests flow into the same unified usage, cost, and tracing view as other AI Gateway traffic.

The specific model IDs are grouped by type as follows:

• Speech-to-text: elevenlabs/scribe-v2, elevenlabs/scribe-v2-realtime
• Streaming speech synthesis: elevenlabs/eleven-flash-v2.5, elevenlabs/eleven-flash-v2, elevenlabs/eleven-multilingual-v2, elevenlabs/eleven-v3-conversational
• Real-time voice agents: elevenlabs/eleven-realtime-v1-max, elevenlabs/eleven-realtime-v1-mini
• Music: elevenlabs/music-v2.5, elevenlabs/music-v2
• Sound effects: elevenlabs/sound-effects-v2
• Multi-speaker dialogue: elevenlabs/eleven-v4, elevenlabs/eleven-v4-turbo, elevenlabs/eleven-v3

This type-based model split is one of the more practical aspects of the whole setup. Because each stage is an independent model, developers can swap out the transcription, language, or voice model without changing the rest of the pipeline. For example, you could use Scribe v2 Realtime to transcribe text while the user is speaking, pass it to a language model to generate a response, and then have a low-latency model like Eleven Flash v2.5 speak it aloud as the answer streams in—so users hear the beginning of the response before the model has finished writing it.

Two Approaches to Real-Time Voice

There are two technical paths for real-time voice interaction, suited to different scenarios.

The first is pipeline-based: transcription, inference, and synthesis are three independent steps. In the official example, the code feeds a 16kHz microphone stream to Scribe v2 Realtime for transcription, sends the completed transcript to a language model (the example uses anthropic/claude-haiku-5.5), and then passes the model's text output directly into speech synthesis. This approach offers high flexibility but requires developers to wire the stages together themselves.

The second is the all-in-one real-time agent: Eleven Realtime v1 Max and v1 Mini handle the entire voice conversation within a single model, with user audio input and spoken response output both completed in a single WebSocket session—no separate wiring for transcription or voice steps. It's suited to scenarios where users expect a natural back-and-forth conversation, such as customer service calls or in-app assistants. The server's only job is to issue a short-lived token for the chosen model, and the browser opens the WebSocket session itself, keeping the API key out of the client. On the browser side, the useRealtime hook from @ai-sdk/react can handle connections, microphone capture, and audio playback. Note that these two models support semantic VAD (semantic-vad) to determine when the user has finished speaking, but do not support server-vad.

The Difference Between Semantic VAD and Server VAD (server-vad)

VAD (Voice Activity Detection) is used to determine when a user starts speaking and when they've finished. Traditional VAD relies on acoustic features such as volume and silence duration, which can trigger falsely when a user pauses to think or when there's ambient noise. Semantic VAD builds on this by incorporating contextual semantic understanding, more accurately identifying whether a sentence is truly complete—reducing interruptions and premature cutoffs, and making conversation turns feel more natural.

server-vad refers to performing VAD decisions on the server side: the client continuously uploads audio, and the server decides when to segment speech and trigger subsequent processing. This approach places low demands on the client but adds end-to-end latency and consumes extra bandwidth. Eleven Realtime models do not support server-vad, meaning developers cannot leave sentence-boundary decisions entirely to the server—they need to handle speech boundaries on the client or within the model. This is one reason for using a client-side hook like useRealtime to manage the microphone stream.

Music, Sound Effects, and Multi-Speaker Dialogue

Music and sound effects models are used similarly to speech synthesis—they also call generateSpeech, except the text field describes the desired sound rather than the words to be read aloud. Music v2.5 and Music v2 can create original music from text descriptions, specifying genre, mood, or instruments, and return a finished track of the specified length, with or without vocals. When finer control is needed, you can also compose a song section by section, setting the style and length for each part. Sound Effects v2 targets shorter sounds, such as footsteps, a door closing, or rain that loops in the background of a game level.

Multi-speaker dialogue is supported by Eleven v4, Eleven v4 Turbo, and Eleven v3, which can perform a multi-speaker script in a single request. Assign a voice to each line of text, and each voice switch begins a new turn of dialogue. Two-person podcasts or full-cast scenarios return a single continuous recording, with no need to stitch together separate clips. Each voice is an ElevenLabs voice ID, and script segments can also be streamed in by a language model.

The Reuse Design of generateSpeech

generateSpeech was originally a text-to-speech call interface, with the semantics of "given text, return audio." In music and sound effect scenarios, this interface is reused as "given a description, return audio": the text is no longer content to be read aloud but a natural-language description of the target sound (for example, "a lighthearted piano intro" or "rain sounds, seamlessly loopable").

The benefit of this design is that callers don't need to learn a new request format—authentication, billing, retry, and tracing logic remain identical; only the model ID and prompt text need to change. The trade-off is that different models have varying ability to understand descriptions, and prompt wording significantly affects output quality. In practice, this usually means tuning prompt templates separately for music and sound effects.

The Engineering Value of the Gateway Layer

AI Gateway provides a unified model invocation API, usage and cost tracing, and configuration for retries, failover, and performance optimization—aiming to deliver higher production availability than a single provider. It also includes custom reports, Zero Data Retention support, per-API-key budgets, routing rules, and more. On pricing, AI Gateway passes through provider pricing as-is with no markup and charges no platform fee for inference; bring-your-own-key (BYOK) requests are covered as well.

For teams building voice applications, the real significance of this partnership is that every stage of the voice pipeline can be managed under the same billing, tracing, and fault-tolerance system—and model choices can be swapped at any time without rewriting integration logic.

All ElevenLabs models are currently 50% off on AI Gateway for the first 14 days, running through November 9. The AI Gateway audio documentation includes complete examples for each model.