Why This Integration Matters
If you've ever wired up audio in an AI app, you know the pain: one SDK for transcription, another for TTS, a third for realtime voice, plus separate billing dashboards and trace logs scattered across providers. Vercel's AI Gateway now folds 14 ElevenLabs models into a single unified API surface — the same key, the same usage/cost/trace views, the same retry and failover config you already use for text models.
The lineup covers the full audio stack:
- Speech to text — transcribe recordings or caption live audio
- Streaming TTS — start playback before the LLM finishes writing
- Realtime voice agents — full-duplex conversation over a single WebSocket
- Music — describe a track in text, get a finished song
- Sound effects — game/UI audio from a short description
- Multi-voice dialogue — a full cast in one request
To celebrate the launch, all ElevenLabs models are 50% off on AI Gateway for 14 days, through November 9th. Pricing otherwise reflects provider rates with no markup and no platform fee on inference, including BYOK requests.

Hands-On: Streaming Voice Pipeline in ~10 Lines
The killer feature here is composability. Because transcription, reasoning, and speech are each their own model, you can swap any stage without touching the rest of the pipeline. Here's a full transcribe → reason → speak loop using the AI SDK:
import {
experimental_streamSpeech as streamSpeech,
experimental_streamTranscribe as streamTranscribe,
streamText,
} from 'ai';
// 1. Transcribe the user's microphone stream (16 kHz PCM)
const heard = streamTranscribe({
model: 'elevenlabs/scribe-v2-realtime',
audio: microphoneStream,
inputAudioFormat: { type: 'audio/pcm', rate: 16000 },
});
// 2. Reason over the transcript with any text model
const { textStream } = streamText({
model: 'anthropic/claude-haiku-5.5',
prompt: await heard.text,
});
// 3. Stream the reply straight into ElevenLabs TTS
const { audioStream } = streamSpeech({
model: 'elevenlabs/eleven-flash-v2.5',
text: textStream,
voice: 'hpp4J3VqNfWAUOO0d1Us',
});
Notice that textStream is piped directly into streamSpeech — the user hears the answer begin before the language model finishes writing it. That's the whole point of streaming TTS: perceived latency drops from seconds to milliseconds.
Realtime Voice Agents
For natural back-and-forth (support lines, in-app assistants), the Eleven Realtime v1 Max / Mini models run the whole conversation in one model over a WebSocket. Your server's only job is minting a short-lived token so your API key never touches the client:
import { gateway } from 'ai';
const { token, url } = await gateway.experimental_realtime.getToken({
model: 'elevenlabs/eleven-realtime-v1-max',
});
On the browser side, use the useRealtime hook from @ai-sdk/react. In sessionConfig, set an ElevenLabs voice ID and enable turnDetection: { type: 'semantic-vad' } — these models support semantic VAD but not server-vad, so plan your turn-taking logic accordingly.
Music, SFX, and Multi-Voice Dialogue
Music v2.5 and Sound Effects v2 both use generateSpeech, but you pass a description instead of words to read:
import { generateSpeech } from 'ai';
const { audio } = await generateSpeech({
model: 'elevenlabs/music-v2.5',
text: 'A mellow lo-fi beat with soft piano',
providerOptions: { elevenlabs: { musicLengthMs: 30000 } },
});
For podcasts or scripted scenes, Eleven v4 / v4 Turbo / v3 voice multiple speakers in a single request — each voice change starts a new turn, and you get one continuous recording with no clip stitching.

Model Comparison at a Glance
| Use Case | Model | Latency Profile | Notes |
|---|---|---|---|
| Realtime transcription | scribe-v2-realtime | Very low | Streams text while user speaks |
| Low-latency TTS | eleven-flash-v2.5 | Low | Best for streaming pipelines |
| Full-duplex voice agent | eleven-realtime-v1-max / -mini | Low (WebSocket) | Semantic VAD only, no server VAD |
| High-quality narration | eleven-v4 / v4-turbo | Medium | Multi-voice dialogue support |
| Legacy multi-voice | eleven-v3 | Medium | Still supported |
| Music generation | music-v2.5 / music-v2 | Higher | Section-by-section control available |
| Sound effects | sound-effects-v2 | Medium | Loops cleanly for game ambience |
Trade-offs to keep in mind:
- Realtime agents do not support
server-vad— you must usesemantic-vad, which changes how you tune interruption handling. - Music generation is the slowest of the batch; don't call it in a user-facing hot path.
- Token-based realtime auth requires a server round-trip before the browser can open the WebSocket — budget for that in your first-token latency.
Limitations and Caveats
- Not a free lunch on cost: the 50% discount is a 14-day launch promo ending November 9th. After that, you're back to provider rates (still no markup, but ElevenLabs audio is not cheap at scale — budget for it).
- Vendor coupling risk: routing everything through AI Gateway is convenient, but if you later need provider-specific features (custom voice cloning APIs, fine-tuning), you'll be reaching past the abstraction.
- Realtime is browser-heavy: the
useRealtimehook handles mic capture and playback, but you still own permission handling, echo cancellation, and mobile Safari quirks. - Semantic VAD ≠ perfect turn-taking: in noisy environments it can cut users off mid-sentence. Test with real audio, not clean studio recordings.
Where to Go Next
- Read the AI Gateway audio docs — full examples across all 14 models.
- Prototype a voice agent with
eleven-realtime-v1-maxbefore the discount ends; the cost delta on a real workload is the best benchmark you'll get. - Instrument your traces early — the unified trace view is the main operational win here; use it to compare audio model latency against your text pipeline.
- If you're optimizing AI spend broadly, pair this with the cost-management playbook in Beyond the Hype: A Strategic Blueprint for Maximizing AI ROI.

Bottom Line
Vercel AI Gateway + ElevenLabs removes a real chunk of glue code: one key, one billing view, one trace, and composable models you can swap per-stage. The 50%-off window through November 9th is the practical moment to benchmark whether streaming TTS and realtime agents actually move your product metrics — because after the promo, audio inference is a line item you'll want to have measured, not guessed.
If you're also benchmarking the hardware side of AI workloads, the NVbandwidth guide to measuring GPU interconnect and memory performance is a solid companion read.