> ## Documentation Index
> Fetch the complete documentation index at: https://docs.talking-avatar.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Messages

> Every message of the realtime avatar API, in both directions.

Control messages are JSON in **text** WebSocket messages, each with a `type`. Audio goes in **binary** messages. The server only sends text messages.

Ignore fields you don't recognise: messages may gain fields over time.

| You send | When | You receive |
| - | - | - |
| [`start`](#start) | First, within 10 seconds of connecting | [`ready`](#ready), or [`error`](#error) and a close |
| [Binary audio](#audio) | Whenever your agent speaks | [`speech_started`](#speech-events), [`speech_ended`](#speech-events) |
| [`clear`](#clear) | The user interrupts | [`cleared`](#clear) |
| [`view`](#view) | Direct output: a browser wants to watch | [`view_offer`](#view) |
| [`view_answer`](#view-answer) | Direct output: the browser answered | [`view_ready`](#view-answer) |
| Close (code `1000`) | The conversation is over | |
| | Direct output: the stream recovered from a drop | [`republished`](#republished) |
| | Something went wrong | [`error`](#error) |

## start

Opens the session. Send it as the first message, within 10 seconds of connecting.

**Direct output (WebRTC)**

The avatar's video and voice stream to your users' browsers over WebRTC.

```json theme={null}
{
  "type": "start",
  "apiKey": "tav_…",
  "avatar": "ava4",
  "output": "direct",
  "sampleRate": 24000
}
```

**Your LiveKit room**

The avatar joins your room as a participant with a video track named `avatar` and its voice as an audio track.

```json theme={null}
{
  "type": "start",
  "apiKey": "tav_…",
  "avatar": "ava4",
  "livekitUrl": "wss://your-project.livekit.cloud",
  "token": "<publish-only token for the avatar>",
  "audio": { "publish": true, "trackName": "avatar-audio" },
  "sampleRate": 24000
}
```

Mint the token on your server with your LiveKit API key and secret: an identity of your choice (such as `avatar-<room>`) and the grant `{ room, roomJoin: true, canPublish: true, canSubscribe: false }`. Your LiveKit secrets never leave your server.

**type** `string` *required*

`"start"`

**apiKey** `string` *required*

Your `tav_…` API key.

**avatar** `string` *required*

An avatar ID, such as `ava4`. See [Avatars](https://docs.talking-avatar.dev/realistic-avatars/avatars.md).

**sampleRate** `integer` (default: 24000)

Your audio's sample rate: `8000`, `12000`, `16000`, `24000` or `48000`.

**output** `string`

`"direct"` for direct output to browsers. Leave it out for LiveKit.

**livekitUrl** `string`

LiveKit only: your LiveKit server's URL.

**token** `string`

LiveKit only: the publish-only token you minted for the avatar's participant.

**audio** `object`

LiveKit only. Direct output always plays the voice.

**properties**

**publish** `boolean`

`true` (recommended): the avatar also publishes the voice, in sync with its lips, and you get speech events and `cleared`. Leave it out to play the voice yourself, and send [play times](#play-times) with your audio.

**trackName** `string` (default: avatar-audio)

The voice track's name, up to 64 characters.

## ready

The avatar is live and billing has started. Send audio only after it.

```json theme={null}
{ "type": "ready", "width": 832, "height": 468, "audio": true }
```

**width** `integer`

The video width in pixels

**height** `integer`

The video height in pixels

**audio** `boolean`

`true` when the avatar plays the voice (direct output, or LiveKit with `audio.publish`)

Starting can take a few seconds while the avatar's stream is set up.

## Audio

Your agent's voice, as it streams. Each binary message is a 12-byte header followed by audio:

| Bytes | Type | Content |
| - | - | - |
| 0–7 | float64, little-endian | **Play time:** `0` to queue the audio. See [play times](#play-times). |
| 8–11 | uint32, little-endian | **Sample rate** of this audio, normally the one in `start` |
| 12– | int16 samples, little-endian | **PCM16 mono** audio, whole samples (an even number of bytes) |

```python Python theme={null}
import struct

HEADER = struct.Struct("<dI")  # play time, sample rate
await ws.send(HEADER.pack(0, 24000) + pcm)
```

```js JavaScript theme={null}
function audioMessage(pcm, sampleRate, playAtMs = 0) {  // pcm: Uint8Array of PCM16 mono
  const message = new Uint8Array(12 + pcm.length);
  const header = new DataView(message.buffer);
  header.setFloat64(0, playAtMs, true);
  header.setUint32(8, sampleRate, true);
  message.set(pcm, 12);
  return message;
}
ws.send(audioMessage(pcm, 24000));
```

```go Go theme={null}
msg := make([]byte, 12+len(pcm))
binary.LittleEndian.PutUint64(msg[0:], math.Float64bits(0))  // play time
binary.LittleEndian.PutUint32(msg[8:], 24000)                // sample rate
copy(msg[12:], pcm)                                          // PCM16 mono, little-endian
conn.WriteMessage(websocket.BinaryMessage, msg)
```

* **Send audio as it's generated.** It can arrive faster than real time: it's queued and spoken at its natural pace.
* **Queueing:** audio with play time `0` plays right after the audio already queued, or about 400 ms after it arrives when nothing is queued.
* **Chunks of any length**, as long as each message holds whole samples and stays under 1 MB. If your TTS splits a sample across chunks, carry the odd byte over to the next message.
* **Raw PCM only.** Decode MP3, Opus or WAV first, and strip WAV headers. See [Stream audio](https://docs.talking-avatar.dev/realistic-avatars/streaming-audio.md) for provider settings.
* **Don't play the audio yourself** when the avatar plays the voice: users would hear it twice.

### Play times

Only when you play the voice yourself (LiveKit without `audio.publish`): set the play time to the wall-clock moment, in milliseconds since the Unix epoch, when the first sample of this message will be heard. Send audio about 400 ms ahead of its play time, so the avatar's lips can be ready, and keep your server's clock synchronised (NTP).

## clear

The user interrupted: drop the audio that hasn't played yet, and stop the avatar mid-sentence.

```json theme={null}
{ "type": "clear" }
```

When the avatar plays the voice, the reply says how many milliseconds of the interrupted reply the user heard, so you can truncate your agent's transcript:

```json theme={null}
{ "type": "cleared", "playedMs": 1840 }
```

There's no reply when you play the voice yourself.

## Speech events

Sent when the avatar plays the voice.

```json theme={null}
{ "type": "speech_started" }
```

A reply's first audio is playing.

```json theme={null}
{ "type": "speech_ended", "playedMs": 5320 }
```

Everything queued has played. `playedMs` is how much of this reply was heard.

## view

Direct output only. A browser wants to watch: ask for a WebRTC offer for it. Give each request an `id` of your choice; the reply carries the same `id`.

```json theme={null}
{ "type": "view", "id": "r1" }
```

```json theme={null}
{ "type": "view_offer", "id": "r1", "viewId": "f3c9…", "sdp": "v=0\r\n…" }
```

**viewId** `string`

This viewer's ID, for its answer

**sdp** `string`

The WebRTC offer, for the browser's `setRemoteDescription({ type: "offer", sdp })`

## view\_answer

The browser's answer to the offer.

```json theme={null}
{ "type": "view_answer", "id": "r2", "viewId": "f3c9…", "sdp": "v=0\r\n…" }
```

```json theme={null}
{ "type": "view_ready", "id": "r2", "viewId": "f3c9…" }
```

The browser then receives the avatar's video and voice. Run `view` and `view_answer` once for every browser, and again whenever a browser reconnects. See [Browser playback](https://docs.talking-avatar.dev/api-reference/realtime/browser-playback.md).

A failed `view` or `view_answer` gets an `error` with the request's `id`, and the session carries on.

## republished

Direct output only. The stream dropped and has recovered on a new connection:

```json theme={null}
{ "type": "republished" }
```

Browsers must watch again: run the `view` handshake for each one. (The SDK's `LiveAvatarView` notices on its own.) If the stream can't recover, the connection closes with `1011`.

## error

Something went wrong.

```json theme={null}
{ "type": "error", "message": "unknown avatar 'ava99'" }
```

| When | What follows |
| - | - |
| Before `ready` (invalid API key, out of credits, unknown avatar, unsupported sample rate, LiveKit join failed…) | The connection closes. See [close codes](https://docs.talking-avatar.dev/api-reference/realtime/overview.md#close-codes). |
| A `view` or `view_answer` failed | Nothing: the reply carries the request's `id`, and the session carries on |
| A broken message during the session (for example, invalid JSON) | The connection closes |


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.