# Speech and language vendors

Three vendors sit behind every spoken turn: one that hears the caller, one that decides what to say, and one that says it. Sphoro Voice speaks a lot of them, and you can pin the ones you want or let a chain choose.

## A chain, not a vendor

Each of the three jobs is an ordered list, and the call uses the first vendor that works. When one refuses, the call falls through to the next rather than failing.

```
transcription   deepgram ──▶ assemblyai ──▶ gladia
      model     anthropic ──▶ gemini ──▶ openai
      voice     deepgram-aura ──▶ elevenlabs ──▶ cartesia
```

| Job | Falls through when |
| --- | --- |
| **Model** | Before its first token — a stall of about a second and a half is enough. After the first token the answer is already being spoken and switching would change speaker mid-sentence. |
| **Voice** | At open *and* mid-session. Vendors do accept a session and then refuse to synthesise into it, which sounds exactly like a dead line; that case re-routes mid-turn. |
| **Transcription** | At open only. A stream that drops mid-call is rebuilt against the same vendor, up to three times. |

> **The head of the chain is what almost every call uses.** So the fastest vendor belongs first, not the best one — the rest of the chain is insurance, and insurance you pay for on every call is just a slow chain. See [latency](https://voice.sphoro.com/docs/latency).

## Pinning a vendor

Per language, because the right transcriber for Tamil is not necessarily the right one for English.

`PATCH /v1/agents/{id}`

**Shell**

```bash
curl -s -X PATCH https://voice.sphoro.com/v1/agents/$AGENT_ID \
  -H "Authorization: Bearer $SPHORO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "languages": [
      { "code": "en-IN", "voice_id": "…", "tts_provider": "elevenlabs", "stt_provider": "deepgram" },
      { "code": "ta",    "voice_id": "…", "tts_provider": "sarvam",     "stt_provider": "gladia" }
    ],
    "models": { "anthropic": "claude-sonnet-5", "gemini": "gemini-2.0-flash" }
  }'
```

`models` names the model to use *on each vendor*, which is what keeps a fall-through sensible: the second vendor in the chain does not have the first one's model names.

> **Voice ids are not portable between vendors.** An id that means a warm Indian-English voice on one vendor means nothing on another, and asking for it returns an error that silences the call. That is what the per-language `voices` map is for — one id per vendor, so a fall-through still sounds like your agent: `{"voices": {"elevenlabs": "…", "cartesia": "…"}}`.

## What this deployment can reach

Generated from the running server, so it is what is actually here rather than what was true when somebody last edited a page. Whether *your account* has credentials for a given vendor is a separate question — the API refuses an unknown one with a `validation_error` naming the field, which is the authoritative answer.

### Transcription

What hears the caller. The differences between these are real and audible: accent coverage, how quickly each decides a sentence has ended, and whether it can follow a caller switching language mid-sentence.

| Name | What it is for |
| --- | --- |
| `assemblyai`AssemblyAI | Also authenticates on the socket. A close second on latency. |
| `aws`Amazon Transcribe | The widest Indian-language coverage here — Odia, Assamese and Punjabi as well as the usual ten — and one of the few recognizers that follows a caller between languages and reports which it heard. Linear PCM only. |
| `azure`Microsoft Azure Speech | Broad Indic coverage, a phrase list for the names a caller says, and µ-law natively, so a phone call needs no conversion. Its realtime protocol is the Speech SDK's rather than a published REST contract — see the package comment. |
| `cartesia`Cartesia | Ink. Takes µ-law natively, so a phone call goes up exactly as it arrives, and it authenticates on the socket, so it costs no round trip. The same key serves Cartesia for speech out. |
| `deepgram`Deepgram | Authenticates on the socket itself, so there is no negotiation round trip for a caller to wait through. The same key serves Deepgram Aura for speech out. |
| `elevenlabs`ElevenLabs | Scribe v2 Realtime — ninety-odd languages on the key already held for speech out. Breadth rather than depth: a specialist beats it on that specialist's languages. |
| `gemini`Google Gemini | The Live API used as an ear. Takes linear PCM only, so it declines a carrier call and serves the browser. Same key as the language model. |
| `gladia`Gladia | The strongest at code-switching, which matters on Hinglish calls — but it negotiates before it listens, so it sits last. |
| `gnani`Gnani | Vachana. Trained on Indian languages first rather than adapted to them — the second such recognizer here beside Sarvam, which is what keeps an Indic call from depending on one vendor. Sends no partial transcripts, and takes linear PCM, so a carrier call is expanded before it goes up. The same key serves Gnani for speech out. |
| `google`Google Cloud Speech | The broadest language coverage available, phrase hints for the names a caller says, and µ-law natively. It is the one recognizer here that is not streaming — Google offers that over gRPC only — so it transcribes each utterance after it ends, which costs a round trip per turn and gives no partial results. A late fallback rather than a primary. |
| `openai`OpenAI | gpt-live-transcribe, on a transcription-only Realtime session. Accepts G.711, so it can serve a phone call. Same key as the language model. |
| `reverie`Reverie | Its code-mixed models — Hinglish, Tamlish and four more — transcribe an English product name inside an Indian-language sentence in English letters, which is what a knowledge base written in English is searched by. Linear PCM only. The same credential serves Reverie for speech out. |
| `sarvam`Sarvam | saaras:v3-realtime. Trained on Indian languages first rather than adapted to them, and it authenticates on the socket, so it costs no round trip. It declines a language it does not have instead of guessing, so the call moves to the next recognizer rather than being transcribed into nonsense. The same key serves Sarvam for speech out and for the model. |
| `smallest`Smallest | Pulse. Covers most of the Indic set on the key already held for speech out. Linear PCM only, so it declines a carrier call. |
| `soniox`Soniox | Independent of every other recognizer here, which is what makes it a real fallback rather than a second door to the same outage. Returns revisable tokens rather than transcripts, so a turn is assembled from them. |
| `speechmatics`Speechmatics | Strong on accents and code-switching. |

### Voices

What speaks. Every one is asked for μ-law at 8 kHz where it supports it, because that is what the carrier wants and resampling in the media path is latency nobody can afford.

| Name | What it is for |
| --- | --- |
| `azure-tts`Azure TTS | Azure Cognitive Services. Set AZURE_SPEECH_REGION. Broad language coverage, and the same pair serves Azure for recognition. |
| `cartesia`Cartesia | Built for telephony latency. Also a socket vendor, so it belongs near the front. |
| `deepgram-aura`Deepgram Aura | Fast and inexpensive; shares the Deepgram key with recognition. Each voice speaks one language, named in the model id — see Speaks. |
| `elevenlabs`ElevenLabs | Streams over a socket, so it starts speaking while the model is still writing. Library and cloned voices need a paid plan; premade voices do not. |
| `gnani`Gnani | Gnani Timbre. Forty-two voices across eleven Indian languages, and it returns raw µ-law natively — so a phone call needs no conversion at all. Shares the Gnani key with recognition. |
| `google-tts`Google TTS | Google Cloud TTS. Very broad language coverage. Returns base64 rather than a stream, so it cannot start speaking early — a late fallback, not a primary. |
| `groq-tts`Groq TTS | PlayAI voices on Groq's hardware. Shares the Groq key. |
| `hume`Hume | Hume Octave. Emotionally expressive; a good fit where tone carries meaning. |
| `lmnt`Lmnt | LMNT. Low latency, μ-law native. |
| `murf`Murf | Murf Falcon. Streams as it synthesizes and returns raw µ-law at 8 kHz, so a phone call needs no conversion. One voice speaks every language Murf has — Hindi, Tamil and the other Indic locales included — chosen per phrase by locale. |
| `neuphonic`Neuphonic | Neuphonic. Built for low-latency conversational use. |
| `openai-tts`Openai TTS | Shares the OpenAI key. Good quality, moderate latency. Speaks at 24 kHz whatever it is asked for, which is exactly the browser profile and is converted for a carrier — see Returns. |
| `playht`Playht | PlayHT. Set PLAYHT_USER_ID as well — it authenticates with both. |
| `polly`Amazon Polly | Returns raw µ-law at 8 kHz, so a phone call needs no conversion. Kajal and Aditi each speak Hindi as well as Indian English. It offers no 24 kHz linear PCM, so it declines a browser call and serves the carrier. |
| `resemble`Resemble | Resemble. Set RESEMBLE_VOICE and RESEMBLE_PROJECT. |
| `reverie`Reverie | Forty-seven voices across twelve Indian languages, each named for the language it speaks. The same credential serves Reverie for recognition. |
| `rime`Rime | Built for conversational telephony; very low latency, μ-law native. |
| `sarvam`Sarvam | Sarvam. Eleven Indian languages — the reason to reach for it over a Western vendor on those calls. Answers with base64 rather than a stream, so it cannot start speaking early; good on its languages, not a primary. |
| `smallest`Smallest | Smallest.ai Waves. Low latency; Indian-language support. |
| `speechify`Speechify | Speechify. |

### Language models

What decides. Anthropic and Gemini are spoken natively; everything else is an OpenAI-compatible endpoint, including self-hosted runtimes.

| Name | What it is for |
| --- | --- |
| `ai21`Ai21 | Jamba. Long context at low cost. |
| `anthropic`Anthropic | Claude. The default for calls where being wrong is expensive. |
| `anyscale`Anyscale | Ray-based hosting. |
| `azure`Azure | Set AZURE_OPENAI_BASE_URL to the deployment endpoint. Azure authenticates with an api-key header rather than a bearer token, which NewAzure handles. |
| `baseten`Baseten | Dedicated deployments; predictable latency under load. |
| `cerebras`Cerebras | Wafer-scale inference. Competes with Groq on latency. |
| `cloudflare`Cloudflare | Workers AI, at the edge. Set CLOUDFLARE_ACCOUNT_ID; the base URL is built from it. |
| `dashscope`Dashscope | Alibaba Qwen. Set DASHSCOPE_BASE_URL for the mainland endpoint. |
| `deepinfra`Deepinfra | Open-weights hosting. |
| `deepseek`Deepseek | Low cost per token. |
| `featherless`Featherless | Very broad model catalogue. |
| `fireworks`Fireworks | Open-weights hosting with good tool-calling support. |
| `friendli`Friendli | Serverless open-weights. |
| `gemini`Google Gemini | Fast and inexpensive. GOOGLE_API_KEY is accepted as the same value. |
| `github-models`Github Models | GitHub Models. A convenient way to reach several vendors with one token. |
| `groq`Groq | Custom silicon; consistently the fastest first token of the hosted options. |
| `hyperbolic`Hyperbolic | Low-cost open-weights hosting. |
| `inference-net`Inference Net | Low-cost batch-friendly hosting. |
| `kilocode`Kilocode | Kilo Gateway — hundreds of models behind one key, addressed as vendor/model (anthropic/claude-sonnet-4.5); kilo-auto/* routes automatically. Handy in dev when no direct vendor key is at hand. |
| `kluster`Kluster | Open-weights hosting. |
| `koboldcpp`Koboldcpp | Local KoboldCpp. |
| `lambda`Lambda | Lambda Labs inference. |
| `litellm`Litellm | A self-hosted proxy in front of a hundred vendors. Set LITELLM_BASE_URL and LITELLM_MODEL — this is the escape hatch for anything not listed here. |
| `llamacpp`Llamacpp | Local llama.cpp server. |
| `lmstudio`Lmstudio | Local desktop runtime. |
| `mistral`Mistral | EU-hosted, which is often the reason it is chosen. |
| `moonshot`Moonshot | Kimi. Strong Chinese and long-context. |
| `nebius`Nebius | EU-hosted open-weights. |
| `novita`Novita | Open-weights hosting. |
| `nscale`Nscale | EU-hosted open-weights. |
| `ollama`Ollama | Local. Needs no key — set OLLAMA_API_KEY to any value to enable it. |
| `openai`Openai | The reference implementation of this wire format. |
| `openrouter`Openrouter | A broker in front of many vendors — useful as a last resort in a chain, since it fails independently of the vendors behind it. |
| `perplexity`Perplexity | Search-grounded answers. A poor conversational agent and a good one to reach for when the caller asks about something current. |
| `portkey`Portkey | A hosted gateway with its own routing and caching. Set PORTKEY_MODEL. |
| `predibase`Predibase | Fine-tuned adapters. Set PREDIBASE_BASE_URL and PREDIBASE_MODEL. |
| `reka`Reka | Reka Flash. |
| `requesty`Requesty | Routing gateway. |
| `sambanova`Sambanova | Dataflow hardware; very fast on large open-weights models. |
| `sarvam`Sarvam | Indian-language models — the reason to reach for it over a Western vendor on Hindi, Tamil, Telugu, Bengali, and Marathi calls. sarvam-105b is the other choice, for reasoning over conversational latency. |
| `textgen`Textgen | Local text-generation-webui. |
| `together`Together | Broad open-weights catalogue. |
| `upstage`Upstage | Solar models; strong Korean support. |
| `vercel`Vercel | Vercel AI Gateway. |
| `vllm`Vllm | Self-hosted. Set VLLM_BASE_URL and VLLM_MODEL; the model name is whatever the server was started with. |
| `writer`Writer | Palmyra. Enterprise-focused. |
| `xai`Xai | Grok. |
| `yi`Yi | 01.AI Yi models. |
| `zhipu`Zhipu | GLM. Chinese-language strength. |

### Carriers

Who places and receives the call. Covered in full on [numbers and carriers](https://voice.sphoro.com/docs/telephony).

| Name | Can dial out |
| --- | --- |
| `plivo`Plivo | Yes |
| `signalwire`SignalWire | Answers inbound only |
| `telnyx`Telnyx | Answers inbound only |
| `twilio`Twilio | Yes |
| `vobiz`Vobiz | Yes |
| `vonage`Vonage | Answers inbound only |

## Speech to speech

The default arrangement is **cascaded**: three vendors in a row. The alternative is one model that listens and replies directly in speech, owning its own turn-taking — `voice_engine: "realtime"` on the agent.

> **It is refused rather than saved when no realtime vendor is configured on the deployment.** Such an agent would answer the call in silence, and a dropdown that is aspirational is the worst way to find that out. If you want it, ask whether this deployment has one before building on it — see [getting help](https://voice.sphoro.com/docs/support).

## Self-hosted and private models

The model chain speaks the OpenAI wire format, and several of the presets in the table above are local runtimes rather than hosted vendors. A deployment that must keep transcripts inside its own network can point the model at one of those and lose nothing else about the platform — the telephony, the flows, the analytics and the knowledge base are unchanged.

> **Check first-token latency before committing to a local model.** This is a phone call: a model that is excellent and takes four seconds to start is unusable, and no configuration elsewhere rescues it. A reasoning model in particular cannot answer a telephone.

## Which vendor a call actually used

Recorded on the call, which is how you find out that a fall-through happened at all. A run of calls that all landed on the second voice in the chain is a first vendor that is quietly refusing — the calls sound fine, the bill does not.

`GET /v1/calls/{id}`

**Shell**

```bash
curl -s https://voice.sphoro.com/v1/calls/$CALL_ID \
  -H "Authorization: Bearer $SPHORO_API_KEY"
```