# Realtime protocol

One WebSocket carries a live call: audio in, audio out, transcripts and state as they happen. This is the wire format underneath [talk from a browser](https://voice.sphoro.com/docs/browser) — read it if you are writing your own client, bridging two people, or watching calls live.

> **Most browser clients should not implement any of this.** `@sphoro/voice-web` is this protocol, the audio graph either side of it, and the half-dozen mistakes below already made. See [SDKs](https://voice.sphoro.com/docs/sdks).

## Connecting

`GET /v1/realtime`

```
GET /v1/realtime?call_id=<id>                  attach to an existing call
GET /v1/realtime?agent_id=<id>                 create a call implicitly and attach
GET /v1/realtime?call_id=<id>&role=responder   answer a bridged call
```

|  |  |
| --- | --- |
| Subprotocol | `voiceai.realtime.v1` — **required**. It is negotiated rather than assumed, so a future version can live on the same endpoint instead of breaking every deployed client on one day. |
| Auth | `Authorization: Bearer …`, or `?access_token=` where a header is impossible. |
| Frames | Text frames are JSON messages. Binary frames are raw caller audio. |
| `role` | `caller` (the default) or `responder`. Matters only on a `webrtc` agent. |

### Authenticating

Prefer the header wherever you can set one — a server, or a client that is not a browser.

**Node**

```js
// Browsers cannot set headers on a WebSocket: the WHATWG API has no way to.
// So the token goes in the query string, and it must be a short-lived call token.
const ws = new WebSocket(
  "wss://voice.sphoro.com/v1/realtime?call_id=" + callId + "&access_token=" + token,
  ["voiceai.realtime.v1"],
);
```

> **Never put an API key in that query parameter.** `access_token` is accepted **only** on an actual WebSocket upgrade — no REST endpoint takes a credential in its URL — because a token in a URL reaches access logs, proxy logs, `Referer` headers and browser history. Mint a call token with `POST /v1/realtime/tokens` from your server: it expires in minutes and is scoped to one call. See [talk from a browser](https://voice.sphoro.com/docs/browser).

Authentication is checked *before* the upgrade, so a bad credential comes back as an ordinary HTTP `401` you can read rather than an opaque handshake failure.

## The usual flow

```
1. POST /v1/calls  {"agent_id": "…", "direction": "web"}   →  realtime_url
2. connect to realtime_url
3. ← session.created
4. ← response.text.delta            the greeting, if configured
5. → input_audio.append × N         or input_text.append, or input_dtmf.append
6. ← transcript.partial × N  →  transcript.final
7. ← response.started → response.text.delta × N → response.audio.delta × N → response.done
8. → session.end                    or POST /v1/calls/{id}/end
```

## Client → server

| Message | Does |
| --- | --- |
| `input_audio.append` | Caller audio. Either a binary frame or a JSON message carrying base64. |
| `input_audio.commit` | Ends the turn explicitly, instead of waiting for endpointing to decide. |
| `input_text.append` | A typed turn. Goes to the model as though the caller had said it, **without** the recogniser — so it works on any deployment and in any vendor configuration. |
| `input_dtmf.append` | A keypress. What a web dialpad sends. See [keypad and IVR](https://voice.sphoro.com/docs/keypad). |
| `interrupt` | Stop speaking now. What a barge-in in your own UI sends. |
| `signal` | Carries the WebRTC handshake between the two ends of a bridged call. |
| `session.update` | Change session settings mid-call. |
| `session.end` | Hang up. |

> **`input_text.append` is not a transcript.** It injects a caller turn directly, skipping the recogniser entirely. That is what makes a chat-style interface work against a voice agent — and it is why text turns behave identically whatever transcription vendor a deployment is using.

## A conversation nobody hears

Add `&modality=text` to the connect URL and the agent answers in writing and speaks nothing: no voice is synthesized, no `response.audio.delta` is sent, and the greeting arrives as a caption. Everything else is the call you already have — same model, same prompt, same tools, same knowledge base, same transcript, same record afterwards.

> **Typing is not the same as chatting.** Sending `input_text.append` on an ordinary session is one turn taken in writing: the reply is still synthesized, so it still waits for a voice vendor, still costs what synthesis costs, and still fails if that vendor refuses the voice. `modality=text` is the call itself held in writing. It defaults to `voice`, and a value this server does not recognise is read as `voice` — a client that gets it wrong is heard, never silenced.

## Server → client

| Message | Means |
| --- | --- |
| `session.created` | Connected. Carries the negotiated audio format — read it rather than assuming. |
| `transcript.partial` | What the caller seems to be saying. Will change. |
| `transcript.final` | What the caller said. Will not. |
| `response.started` | The agent has begun a reply. |
| `response.text.delta` | The reply, as text, as it is produced. |
| `response.audio.delta` | The reply, as audio, as it is produced. |
| `response.done` | That reply is finished. |
| `tool.called` | The model called a tool. Useful for showing "checking that for you…" in your own UI. |
| `state.changed` | Listening, thinking or speaking. What drives an indicator. |
| `error` | Something went wrong, in the same shape as an HTTP error. |
| `session.ended` | The call is over, with its end reason. |
| `ping` / `pong` | The heartbeat. See below — this one matters. |

## Negotiating the audio format

Read the format from `session.created`. Do not hard-code one.

| Where | Format |
| --- | --- |
| A call over a carrier | μ-law at 8 kHz, both directions. That is what the telephone network carries. |
| A browser call | Wideband — PCM16 at 24 kHz — negotiated at session creation. |

> **Every byte-to-time conversion in your client has to follow the negotiated rate.** A client that assumes 8 kHz while the session is running at 24 kHz gets every duration, every buffer level and every pacing calculation wrong by a factor of three — and the symptom is not an error, it is audio that sounds fine and behaves strangely.

## Building a client that behaves

Five things, all of which have gone wrong in real clients.

| Do | Because |
| --- | --- |
| **Answer `ping` with `pong`, and treat a missed heartbeat as a dead connection.** | Browsers hide protocol-level WebSocket pongs from the page entirely, so a half-dead socket looks perfectly open forever. The application-level heartbeat is the only signal a page can actually see. |
| **Pace audio you write, even when you already have it.** | Writing nine seconds of audio in three milliseconds ends the speaking state nine seconds early. Everything that depends on the agent holding the floor then misbehaves — barge-in most of all. |
| **Stop playback on `interrupt`, and drop what is buffered.** | Otherwise the caller interrupts and then listens to the rest of the sentence anyway. |
| **Do not await a terminal state when hanging up.** | Set your own UI state and close. Teardown can swallow the phase you are waiting for, and the user is left looking at a call that will not end. |
| **Reconnect on close, but not on a `4xx`.** | A rejected credential does not become valid on the ninth attempt. |

## Bridged calls: two people

On a `webrtc` agent, two clients join the same call, the platform passes their handshake across with `signal`, and the audio then runs **directly between them** and never reaches this deployment.

```
caller     → GET /v1/realtime?call_id=…                    (role defaults to caller)
responder  → GET /v1/realtime?call_id=…&role=responder
             ← signal ↔ signal →
             audio flows peer to peer from here on
```

Media staying out is the point, not an implementation detail: it is what makes the call private and what makes it cost almost nothing. The consequence is that there is no transcript unless both clients uplink their microphones and the agent sets `transcribe_calls`.

## Watching without joining

For a dashboard rather than a client, you usually want the event feed rather than the audio socket:

`GET /v1/calls/{id}/events`

`GET /v1/calls/ringing`

**Shell**

```bash
curl -sN https://voice.sphoro.com/v1/calls/$CALL_ID/events \
  -H "Authorization: Bearer $SPHORO_API_KEY"
```

> **Subscribe to the event types you need, not to everything.** A whole-account subscription with no type filter receives every transcript line from every live call, and transcripts arrive fast enough to push the events you actually wanted — a call starting to ring — out of the buffer. The feed drops under load by design; the filter is how you decide what survives.