Realtime protocol

One WebSocket carries a live call: audio in, audio out, transcripts and state as they happen. This is the wire format underneath talk from a browser — read it if you are writing your own client, bridging two people, or watching calls live.

Most browser clients should not implement any of this. @sphoro/voice-web is this protocol, the audio graph either side of it, and the half-dozen mistakes below already made. See SDKs.

Connecting

GET/v1/realtime

GET /v1/realtime?call_id=<id>                  attach to an existing call
GET /v1/realtime?agent_id=<id>                 create a call implicitly and attach
GET /v1/realtime?call_id=<id>&role=responder   answer a bridged call


Subprotocol	`voiceai.realtime.v1` — required. It is negotiated rather than assumed, so a future version can live on the same endpoint instead of breaking every deployed client on one day.
Auth	`Authorization: Bearer …`, or `?access_token=` where a header is impossible.
Frames	Text frames are JSON messages. Binary frames are raw caller audio.
`role`	`caller` (the default) or `responder`. Matters only on a `webrtc` agent.

Authenticating

Prefer the header wherever you can set one — a server, or a client that is not a browser.

Node

// Browsers cannot set headers on a WebSocket: the WHATWG API has no way to.
// So the token goes in the query string, and it must be a short-lived call token.
const ws = new WebSocket(
  "wss://voice.sphoro.com/v1/realtime?call_id=" + callId + "&access_token=" + token,
  ["voiceai.realtime.v1"],
);

Never put an API key in that query parameter. access_token is accepted only on an actual WebSocket upgrade — no REST endpoint takes a credential in its URL — because a token in a URL reaches access logs, proxy logs, Referer headers and browser history. Mint a call token with POST /v1/realtime/tokens from your server: it expires in minutes and is scoped to one call. See talk from a browser.

Authentication is checked before the upgrade, so a bad credential comes back as an ordinary HTTP 401 you can read rather than an opaque handshake failure.

The usual flow

1. POST /v1/calls  {"agent_id": "…", "direction": "web"}   →  realtime_url
2. connect to realtime_url
3. ← session.created
4. ← response.text.delta            the greeting, if configured
5. → input_audio.append × N         or input_text.append, or input_dtmf.append
6. ← transcript.partial × N  →  transcript.final
7. ← response.started → response.text.delta × N → response.audio.delta × N → response.done
8. → session.end                    or POST /v1/calls/{id}/end

Client → server

Message	Does
`input_audio.append`	Caller audio. Either a binary frame or a JSON message carrying base64.
`input_audio.commit`	Ends the turn explicitly, instead of waiting for endpointing to decide.
`input_text.append`	A typed turn. Goes to the model as though the caller had said it, without the recogniser — so it works on any deployment and in any vendor configuration.
`input_dtmf.append`	A keypress. What a web dialpad sends. See keypad and IVR.
`interrupt`	Stop speaking now. What a barge-in in your own UI sends.
`signal`	Carries the WebRTC handshake between the two ends of a bridged call.
`session.update`	Change session settings mid-call.
`session.end`	Hang up.

input_text.append is not a transcript. It injects a caller turn directly, skipping the recogniser entirely. That is what makes a chat-style interface work against a voice agent — and it is why text turns behave identically whatever transcription vendor a deployment is using.

A conversation nobody hears

Add &modality=text to the connect URL and the agent answers in writing and speaks nothing: no voice is synthesized, no response.audio.delta is sent, and the greeting arrives as a caption. Everything else is the call you already have — same model, same prompt, same tools, same knowledge base, same transcript, same record afterwards.

Typing is not the same as chatting. Sending input_text.append on an ordinary session is one turn taken in writing: the reply is still synthesized, so it still waits for a voice vendor, still costs what synthesis costs, and still fails if that vendor refuses the voice. modality=text is the call itself held in writing. It defaults to voice, and a value this server does not recognise is read as voice — a client that gets it wrong is heard, never silenced.

Server → client

Message	Means
`session.created`	Connected. Carries the negotiated audio format — read it rather than assuming.
`transcript.partial`	What the caller seems to be saying. Will change.
`transcript.final`	What the caller said. Will not.
`response.started`	The agent has begun a reply.
`response.text.delta`	The reply, as text, as it is produced.
`response.audio.delta`	The reply, as audio, as it is produced.
`response.done`	That reply is finished.
`tool.called`	The model called a tool. Useful for showing "checking that for you…" in your own UI.
`state.changed`	Listening, thinking or speaking. What drives an indicator.
`error`	Something went wrong, in the same shape as an HTTP error.
`session.ended`	The call is over, with its end reason.
`ping` / `pong`	The heartbeat. See below — this one matters.

Negotiating the audio format

Read the format from session.created. Do not hard-code one.

Where	Format
A call over a carrier	μ-law at 8 kHz, both directions. That is what the telephone network carries.
A browser call	Wideband — PCM16 at 24 kHz — negotiated at session creation.

Every byte-to-time conversion in your client has to follow the negotiated rate. A client that assumes 8 kHz while the session is running at 24 kHz gets every duration, every buffer level and every pacing calculation wrong by a factor of three — and the symptom is not an error, it is audio that sounds fine and behaves strangely.

Building a client that behaves

Five things, all of which have gone wrong in real clients.

Do	Because
Answer `ping` with `pong`, and treat a missed heartbeat as a dead connection.	Browsers hide protocol-level WebSocket pongs from the page entirely, so a half-dead socket looks perfectly open forever. The application-level heartbeat is the only signal a page can actually see.
Pace audio you write, even when you already have it.	Writing nine seconds of audio in three milliseconds ends the speaking state nine seconds early. Everything that depends on the agent holding the floor then misbehaves — barge-in most of all.
Stop playback on `interrupt`, and drop what is buffered.	Otherwise the caller interrupts and then listens to the rest of the sentence anyway.
Do not await a terminal state when hanging up.	Set your own UI state and close. Teardown can swallow the phase you are waiting for, and the user is left looking at a call that will not end.
Reconnect on close, but not on a `4xx`.	A rejected credential does not become valid on the ninth attempt.

Bridged calls: two people

On a webrtc agent, two clients join the same call, the platform passes their handshake across with signal, and the audio then runs directly between them and never reaches this deployment.

caller     → GET /v1/realtime?call_id=…                    (role defaults to caller)
responder  → GET /v1/realtime?call_id=…&role=responder
             ← signal ↔ signal →
             audio flows peer to peer from here on

Media staying out is the point, not an implementation detail: it is what makes the call private and what makes it cost almost nothing. The consequence is that there is no transcript unless both clients uplink their microphones and the agent sets transcribe_calls.

Watching without joining

For a dashboard rather than a client, you usually want the event feed rather than the audio socket:

GET/v1/calls/{id}/events

GET/v1/calls/ringing

Shell

curl -sN https://voice.sphoro.com/v1/calls/$CALL_ID/events \
  -H "Authorization: Bearer $SPHORO_API_KEY"

Subscribe to the event types you need, not to everything. A whole-account subscription with no type filter receives every transcript line from every live call, and transcripts arrive fast enough to push the events you actually wanted — a call starting to ring — out of the buffer. The feed drops under load by design; the filter is how you decide what survives.