Messages

Control messages are JSON in text WebSocket messages, each with a type. Audio goes in binary messages. The server only sends text messages. Ignore fields you don’t recognise: messages may gain fields over time.

You send	When	You receive
`start`	First, within 10 seconds of connecting	`ready`, or `error` and a close
Binary audio	Whenever your agent speaks	`speech_started`, `speech_ended`
`clear`	The user interrupts	`cleared`
`view`	Direct output: a browser wants to watch	`view_offer`
`view_answer`	Direct output: the browser answered	`view_ready`
Close (code `1000`)	The conversation is over
	Direct output: the stream recovered from a drop	`republished`
	Something went wrong	`error`

start

Opens the session. Send it as the first message, within 10 seconds of connecting.

Direct output (WebRTC)
Your LiveKit room

The avatar’s video and voice stream to your users’ browsers over WebRTC.

{
  "type": "start",
  "apiKey": "tav_…",
  "avatar": "ava4",
  "output": "direct",
  "sampleRate": 24000
}

The avatar joins your room as a participant with a video track named avatar and its voice as an audio track.

{
  "type": "start",
  "apiKey": "tav_…",
  "avatar": "ava4",
  "livekitUrl": "wss://your-project.livekit.cloud",
  "token": "<publish-only token for the avatar>",
  "audio": { "publish": true, "trackName": "avatar-audio" },
  "sampleRate": 24000
}

Mint the token on your server with your LiveKit API key and secret: an identity of your choice (such as avatar-<room>) and the grant { room, roomJoin: true, canPublish: true, canSubscribe: false }. Your LiveKit secrets never leave your server.

string

required

"start"

string

required

Your tav_… API key.

string

required

An avatar ID, such as ava4. See Avatars.

integer

default:"24000"

Your audio’s sample rate: 8000, 12000, 16000, 24000 or 48000.

string

"direct" for direct output to browsers. Leave it out for LiveKit.

string

LiveKit only: your LiveKit server’s URL.

string

LiveKit only: the publish-only token you minted for the avatar’s participant.

object

LiveKit only. Direct output always plays the voice.

Show properties

boolean

true (recommended): the avatar also publishes the voice, in sync with its lips, and you get speech events and cleared. Leave it out to play the voice yourself, and send play times with your audio.

string

default:"avatar-audio"

The voice track’s name, up to 64 characters.

ready

The avatar is live and billing has started. Send audio only after it.

{ "type": "ready", "width": 832, "height": 468, "audio": true }

integer

The video width in pixels

integer

The video height in pixels

boolean

true when the avatar plays the voice (direct output, or LiveKit with audio.publish)

Starting can take a few seconds while the avatar’s stream is set up.

Audio

Your agent’s voice, as it streams. Each binary message is a 12-byte header followed by audio:

Bytes	Type	Content
0–7	float64, little-endian	Play time: `0` to queue the audio. See play times.
8–11	uint32, little-endian	Sample rate of this audio, normally the one in `start`
12–	int16 samples, little-endian	PCM16 mono audio, whole samples (an even number of bytes)

import struct

HEADER = struct.Struct("<dI")  # play time, sample rate
await ws.send(HEADER.pack(0, 24000) + pcm)

function audioMessage(pcm, sampleRate, playAtMs = 0) {  // pcm: Uint8Array of PCM16 mono
  const message = new Uint8Array(12 + pcm.length);
  const header = new DataView(message.buffer);
  header.setFloat64(0, playAtMs, true);
  header.setUint32(8, sampleRate, true);
  message.set(pcm, 12);
  return message;
}
ws.send(audioMessage(pcm, 24000));

msg := make([]byte, 12+len(pcm))
binary.LittleEndian.PutUint64(msg[0:], math.Float64bits(0))  // play time
binary.LittleEndian.PutUint32(msg[8:], 24000)                // sample rate
copy(msg[12:], pcm)                                          // PCM16 mono, little-endian
conn.WriteMessage(websocket.BinaryMessage, msg)

Send audio as it’s generated. It can arrive faster than real time: it’s queued and spoken at its natural pace.
Queueing: audio with play time 0 plays right after the audio already queued, or about 400 ms after it arrives when nothing is queued.
Chunks of any length, as long as each message holds whole samples and stays under 1 MB. If your TTS splits a sample across chunks, carry the odd byte over to the next message.
Raw PCM only. Decode MP3, Opus or WAV first, and strip WAV headers. See Stream audio for provider settings.
Don’t play the audio yourself when the avatar plays the voice: users would hear it twice.

Play times

Only when you play the voice yourself (LiveKit without audio.publish): set the play time to the wall-clock moment, in milliseconds since the Unix epoch, when the first sample of this message will be heard. Send audio about 400 ms ahead of its play time, so the avatar’s lips can be ready, and keep your server’s clock synchronised (NTP).

clear

The user interrupted: drop the audio that hasn’t played yet, and stop the avatar mid-sentence.

{ "type": "clear" }

When the avatar plays the voice, the reply says how many milliseconds of the interrupted reply the user heard, so you can truncate your agent’s transcript:

{ "type": "cleared", "playedMs": 1840 }

There’s no reply when you play the voice yourself.

Speech events

Sent when the avatar plays the voice.

{ "type": "speech_started" }

A reply’s first audio is playing.

{ "type": "speech_ended", "playedMs": 5320 }

Everything queued has played. playedMs is how much of this reply was heard.

view

Direct output only. A browser wants to watch: ask for a WebRTC offer for it. Give each request an id of your choice; the reply carries the same id.

{ "type": "view", "id": "r1" }

{ "type": "view_offer", "id": "r1", "viewId": "f3c9…", "sdp": "v=0\r\n…" }

string

This viewer’s ID, for its answer

string

The WebRTC offer, for the browser’s setRemoteDescription({ type: "offer", sdp })

view_answer

The browser’s answer to the offer.

{ "type": "view_answer", "id": "r2", "viewId": "f3c9…", "sdp": "v=0\r\n…" }

{ "type": "view_ready", "id": "r2", "viewId": "f3c9…" }

The browser then receives the avatar’s video and voice. Run view and view_answer once for every browser, and again whenever a browser reconnects. See Browser playback. A failed view or view_answer gets an error with the request’s id, and the session carries on.

republished

Direct output only. The stream dropped and has recovered on a new connection:

{ "type": "republished" }

Browsers must watch again: run the view handshake for each one. (The SDK’s LiveAvatarView notices on its own.) If the stream can’t recover, the connection closes with 1011.

error

Something went wrong.

{ "type": "error", "message": "unknown avatar 'ava99'" }

When	What follows
Before `ready` (invalid API key, out of credits, unknown avatar, unsupported sample rate, LiveKit join failed…)	The connection closes. See close codes.
A `view` or `view_answer` failed	Nothing: the reply carries the request’s `id`, and the session carries on
A broken message during the session (for example, invalid JSON)	The connection closes

Overview

Realtime avatar API

Private agent links

Requests we send you

start

ready

Audio

Play times

clear

Speech events

view

view_answer

republished

error

​start

​ready

​Audio

​Play times

​clear

​Speech events

​view

​view_answer

​republished

​error

start

ready

Audio

Play times

clear

Speech events

view

view_answer

republished

error