Why This Integration Matters

If you've ever wired up audio in an AI app, you know the pain: one SDK for transcription, another for TTS, a third for realtime voice, plus separate billing dashboards and trace logs scattered across providers. Vercel's AI Gateway now folds 14 ElevenLabs models into a single unified API surface — the same key, the same usage/cost/trace views, the same retry and failover config you already use for text models.

The lineup covers the full audio stack:

  • Speech to text — transcribe recordings or caption live audio
  • Streaming TTS — start playback before the LLM finishes writing
  • Realtime voice agents — full-duplex conversation over a single WebSocket
  • Music — describe a track in text, get a finished song
  • Sound effects — game/UI audio from a short description
  • Multi-voice dialogue — a full cast in one request

To celebrate the launch, all ElevenLabs models are 50% off on AI Gateway for 14 days, through November 9th. Pricing otherwise reflects provider rates with no markup and no platform fee on inference, including BYOK requests.

Developer integrating ElevenLabs audio models via Vercel AI Gateway for realtime voice agents Technical Structure Concept

Hands-On: Streaming Voice Pipeline in ~10 Lines

The killer feature here is composability. Because transcription, reasoning, and speech are each their own model, you can swap any stage without touching the rest of the pipeline. Here's a full transcribe → reason → speak loop using the AI SDK:

import {
  experimental_streamSpeech as streamSpeech,
  experimental_streamTranscribe as streamTranscribe,
  streamText,
} from 'ai';

// 1. Transcribe the user's microphone stream (16 kHz PCM)
const heard = streamTranscribe({
  model: 'elevenlabs/scribe-v2-realtime',
  audio: microphoneStream,
  inputAudioFormat: { type: 'audio/pcm', rate: 16000 },
});

// 2. Reason over the transcript with any text model
const { textStream } = streamText({
  model: 'anthropic/claude-haiku-5.5',
  prompt: await heard.text,
});

// 3. Stream the reply straight into ElevenLabs TTS
const { audioStream } = streamSpeech({
  model: 'elevenlabs/eleven-flash-v2.5',
  text: textStream,
  voice: 'hpp4J3VqNfWAUOO0d1Us',
});

Notice that textStream is piped directly into streamSpeech — the user hears the answer begin before the language model finishes writing it. That's the whole point of streaming TTS: perceived latency drops from seconds to milliseconds.

Realtime Voice Agents

For natural back-and-forth (support lines, in-app assistants), the Eleven Realtime v1 Max / Mini models run the whole conversation in one model over a WebSocket. Your server's only job is minting a short-lived token so your API key never touches the client:

import { gateway } from 'ai';

const { token, url } = await gateway.experimental_realtime.getToken({
  model: 'elevenlabs/eleven-realtime-v1-max',
});

On the browser side, use the useRealtime hook from @ai-sdk/react. In sessionConfig, set an ElevenLabs voice ID and enable turnDetection: { type: 'semantic-vad' } — these models support semantic VAD but not server-vad, so plan your turn-taking logic accordingly.

Music, SFX, and Multi-Voice Dialogue

Music v2.5 and Sound Effects v2 both use generateSpeech, but you pass a description instead of words to read:

import { generateSpeech } from 'ai';

const { audio } = await generateSpeech({
  model: 'elevenlabs/music-v2.5',
  text: 'A mellow lo-fi beat with soft piano',
  providerOptions: { elevenlabs: { musicLengthMs: 30000 } },
});

For podcasts or scripted scenes, Eleven v4 / v4 Turbo / v3 voice multiple speakers in a single request — each voice change starts a new turn, and you get one continuous recording with no clip stitching.

AI Gateway dashboard showing ElevenLabs speech-to-text and music model usage with cost tracking Developer Related Image

Model Comparison at a Glance

Use CaseModelLatency ProfileNotes
Realtime transcriptionscribe-v2-realtimeVery lowStreams text while user speaks
Low-latency TTSeleven-flash-v2.5LowBest for streaming pipelines
Full-duplex voice agenteleven-realtime-v1-max / -miniLow (WebSocket)Semantic VAD only, no server VAD
High-quality narrationeleven-v4 / v4-turboMediumMulti-voice dialogue support
Legacy multi-voiceeleven-v3MediumStill supported
Music generationmusic-v2.5 / music-v2HigherSection-by-section control available
Sound effectssound-effects-v2MediumLoops cleanly for game ambience

Trade-offs to keep in mind:

  • Realtime agents do not support server-vad — you must use semantic-vad, which changes how you tune interruption handling.
  • Music generation is the slowest of the batch; don't call it in a user-facing hot path.
  • Token-based realtime auth requires a server round-trip before the browser can open the WebSocket — budget for that in your first-token latency.

Limitations and Caveats

  • Not a free lunch on cost: the 50% discount is a 14-day launch promo ending November 9th. After that, you're back to provider rates (still no markup, but ElevenLabs audio is not cheap at scale — budget for it).
  • Vendor coupling risk: routing everything through AI Gateway is convenient, but if you later need provider-specific features (custom voice cloning APIs, fine-tuning), you'll be reaching past the abstraction.
  • Realtime is browser-heavy: the useRealtime hook handles mic capture and playback, but you still own permission handling, echo cancellation, and mobile Safari quirks.
  • Semantic VAD ≠ perfect turn-taking: in noisy environments it can cut users off mid-sentence. Test with real audio, not clean studio recordings.

Where to Go Next

  1. Read the AI Gateway audio docs — full examples across all 14 models.
  2. Prototype a voice agent with eleven-realtime-v1-max before the discount ends; the cost delta on a real workload is the best benchmark you'll get.
  3. Instrument your traces early — the unified trace view is the main operational win here; use it to compare audio model latency against your text pipeline.
  4. If you're optimizing AI spend broadly, pair this with the cost-management playbook in Beyond the Hype: A Strategic Blueprint for Maximizing AI ROI.

Cloud architecture diagram of AI Gateway routing audio model requests to ElevenLabs Algorithm Concept Visual

Bottom Line

Vercel AI Gateway + ElevenLabs removes a real chunk of glue code: one key, one billing view, one trace, and composable models you can swap per-stage. The 50%-off window through November 9th is the practical moment to benchmark whether streaming TTS and realtime agents actually move your product metrics — because after the promo, audio inference is a line item you'll want to have measured, not guessed.

If you're also benchmarking the hardware side of AI workloads, the NVbandwidth guide to measuring GPU interconnect and memory performance is a solid companion read.

Source: Vercel blog — ElevenLabs on AI Gateway

This content was drafted using AI tools based on reliable sources, and has been reviewed by our editorial team before publication. It is not intended to replace professional advice.