# Under the hood

The engineering behind a live call on Sphoro Voice, for readers who want the reasons as well as the settings: why the stages overlap, what an interruption actually stops, how vendors are watched, and what a recording is made of. Nothing on this page needs configuring.

## Why the stages overlap

A spoken reply has four stages, and the obvious way to run them is one after another. That way takes about two seconds, which is the pause everybody has heard from a phone bot. Each stage here starts on the previous one's first output instead of waiting for its last.

| Stage | Run one after another | Run here |
| --- | --- | --- |
| End of the caller's turn | A fixed silence timeout, usually around 700ms | A transcript that reads as a finished sentence ends the turn at once; anything else ends it after a short silence. See [how a call runs](https://voice.sphoro.com/docs/call-lifecycle). |
| Transcript | Wait for the recognizer to finalise | Streamed while the caller is still speaking. |
| Reply | Wait for the model to finish writing | Streamed token by token. |
| Voice | Synthesise the whole reply, then play it | Each clause goes to the voice as soon as it is complete, while the model writes the next. |

The last row is the biggest single lever on how fast a reply feels. Tokens from the model are held only until a sentence or clause is complete, or until a long run with no punctuation reaches a length cap, and are then released to synthesis. The caller hears the first sentence while the rest is still being written. Everything released has already been cleaned for speech, so markup the model was asked not to write never reaches the voice.

The budget those stages fit into, stage by stage, and the settings that move it, are on [latency](https://voice.sphoro.com/docs/latency).

## Call states, changed atomically

A live call is always in exactly one state:

```
IDLE ──▶ LISTENING ──▶ THINKING ──▶ SPEAKING ──┐
           ▲                                    │
           └──────── turn done, or barge-in ────┘
                                                │
                      closing line ──▶ CLOSING ──▶ CLOSED
```

| State | What is happening |
| --- | --- |
| `IDLE` | The session exists; audio is not flowing yet. |
| `LISTENING` | Caller audio is streaming to the recognizer, waiting for the end of the turn. |
| `THINKING` | The turn is final: retrieval, then the model, up to its first token. |
| `SPEAKING` | The voice is streaming to the caller. Interruptible. |
| `CLOSING` | The goodbye is playing. Not interruptible: an interruption here would delete the closing line rather than abandon a turn. |

Moves between states are compare-and-swap operations, not a lock. The code carrying live audio never waits on a mutex to learn what the call is doing, and when two things try to end the same turn (an interruption and the reply finishing, say) exactly one of them wins. A reply is never cancelled twice, and never left running.

## What an interruption stops

Barge-in only means something while the agent is `SPEAKING`. When the caller interrupts, the audio path claims the interruption once, moves the call back to `LISTENING`, and does three things on the spot, without waiting for the rest of the turn to unwind:

1. cancels the model's reply in flight;
2. closes the voice session, which also releases anything waiting to send it text;
3. flushes audio already sent to the carrier but not yet played.

The third is the one that is easy to miss. A carrier buffers audio ahead of playback, and it can be holding an entire line by the time the caller reacts to its first word. Without the flush, the caller would go on hearing the agent after they had interrupted it.

What counts as an interruption is decided by the recognizer's words, not by volume. Sound alone counts only once it has run for 1.2 seconds without a break: a caller still going after that is not acknowledging anything, and the agent should not talk over them because a recognizer was slow. See [conversation behaviour](https://voice.sphoro.com/docs/behaviours).

## Keypresses

Digits are never read out of a transcript. Where the carrier sends them as events they arrive on a channel of their own; where they arrive as tones in the audio, a tone detector picks them out. Either way a keypress cannot be misheard as a word, which matters because callers reach for the keypad exactly when they are not being understood. See [keypad and IVR](https://voice.sphoro.com/docs/keypad).

## How vendors are watched

A vendor key does not fail when it is saved. It fails later: when the account runs out of credit, when a plan stops covering the voice an agent uses, when somebody rotates the key in the vendor's own console. From here each of those looks like nothing at all until a caller hears silence, so every vendor is asked, on a schedule, whether it still works.

|  |  |
| --- | --- |
| How often | Every five minutes, for every vendor with a key. |
| What is asked | The cheapest real request: a voice is asked to say a two-letter word, a model for a one-token completion. |
| When a vendor is withdrawn | After two failures in a row, or at once on a refusal that will not clear on its own, such as a revoked key or a plan that does not cover the voice. |
| When it comes back | A withdrawn vendor keeps being asked, and one success returns it to its place in the chain. |

Two more rules decide which vendor a turn actually uses:

- **The agent's own voice vendor goes first.** The chain's order says which vendor the account prefers in general; it cannot know which vendor one agent's voice belongs to. Followed blindly, it produces an agent answering in the wrong voice with nothing failing. A voice id outranks the order.
- **A voice can change vendor only before the caller hears it.** If a voice refuses a turn, the chain moves to the next vendor and never asks one that has already refused that turn again. Once the first audio has reached the caller the turn is committed: a failure after that cuts the sentence short rather than start it again in a different voice.

Vendor keys are entered on the portal's integrations page, which is also where each chain's order is set, and they are sealed with AES-256-GCM before they are stored. A vendor that has a key but is missing from the order is added to the end of it rather than ignored. See [speech and language vendors](https://voice.sphoro.com/docs/providers).

## Recordings

A recording is a stereo WAV: the caller on the left channel, the agent on the right. Two channels, because "what actually happened" on a call nearly always comes down to who spoke over whom, and a mono mix hides exactly that.

The caller's channel is padded to wall-clock time, so a silence stays as long as it was. The agent's audio is placed where the carrier played it rather than where it was produced, and stops where a barge-in stopped it, so the file sounds the way the call did. See [recordings and transcripts](https://voice.sphoro.com/docs/recordings).

## Pacing a campaign

Campaigns started from the portal keep their contact list in the database, not in memory, so a restart resumes a campaign rather than repeating it. The dialer paces against the calls per second the carrier tolerates, keeps a bounded backlog, decides between retrying and abandoning a number that fails, never places more calls than there is capacity to answer, and takes turns between accounts so one large list cannot starve the rest. See [campaigns](https://voice.sphoro.com/docs/campaigns) and [concurrency](https://voice.sphoro.com/docs/capacity).

## Measuring without slowing the call

Per-stage latency, token use and cost are recorded for every call through a queue the audio path never waits on. If the queue is ever full, events are dropped and the drops are counted and monitored: a live call does not slow down to report on itself. See [analytics](https://voice.sphoro.com/docs/analytics).

## Language tags

One language reaches the platform under many spellings. An operator types "Hindi", a recognizer reports `hi`, a voice advertises `hi-IN`, a carrier passes `hi_IN`. Compared as strings that is four languages, and an agent that "switches" to Hindi while already speaking it. Every tag is normalised once, on the way in, and the native name is accepted too.

The language table is India-first and records what a generic tag library does not: that Hindi and Urdu are mutually intelligible in speech though written in different scripts, and that a Marathi speaker will accept a Hindi voice where a Tamil speaker will not.

Following a caller from one language to another is off by default, because an agent that starts changing language on its own is a change nobody opted into. Turn it on per agent; see [language and voice](https://voice.sphoro.com/docs/voices) for the languages themselves.

`PATCH /v1/agents/{id}`

**Shell**

```bash
curl -s -X PATCH https://voice.sphoro.com/v1/agents/$AGENT_ID \
  -H "Authorization: Bearer $SPHORO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"language_detection": "follow"}'
```

## Limits on tools

Every tool call runs through one pipeline: find the tool within your account, check the call is within its scopes, validate the arguments against the tool's JSON Schema with every problem reported at once, then run it in a sandbox. Unless a tool sets its own limits, the sandbox gives each call ten seconds, allows each account 120 calls a minute and 8 at once per tool, and truncates output at 8 KiB; it also recovers from a crash instead of taking the call down with it. A failure at any step goes back to the model as text it can act on, so a mistyped booking becomes a follow-up question rather than a dropped turn. See [tools and functions](https://voice.sphoro.com/docs/functions).

## Keys, permissions and limits

- **API keys** are 256 bits of randomness, stored only as a SHA-256 hash and compared in constant time. A leaked database yields hashes, not credentials. See [get an API key](https://voice.sphoro.com/docs/api-keys).
- **Permissions, not role names.** Nothing asks "is this person an admin"; it asks "may this person revoke a key". The rules that stop somebody promoting themselves live in one place, and changes are written to a durable audit log.
- **Rate limits** are a token bucket per account, with `X-RateLimit-*` headers on API responses and `Retry-After` on a `429`, so a client can slow down before it is throttled. See [use your API key](https://voice.sphoro.com/docs/authentication).
- **Carriers are checked before dialling.** Naming a carrier that cannot place outbound calls is refused when the call is created, rather than quietly dialled through another one with a different caller ID.

## No third-party code

The server has no third-party dependencies, and neither do the published clients, `@sphoro/voice` and `@sphoro/voice-web`. Adopting one cannot pull somebody else's library version into your application. See [SDKs](https://voice.sphoro.com/docs/sdks).
