TALKING_AVATAR_API_KEY), never in the chat.
The same prompt is on the dashboard’s Quickstart page, under Copy prompt.
The prompt proposes a small Node.js service when your backend is in another language. To skip it, add a line asking the agent to use the realtime avatar API over WebSocket instead of the SDK, with the link to that page.
Prompt
# Task: add a Talking Avatar to this project
Add a realistic, lip-synced avatar to this app's voice agent with the Talking Avatar SDK
(`@interviewflowai/talking-avatar`). The avatar speaks the agent's voice in real time on the page. Fit it into the
existing code, framework and conventions. Don't build a separate demo app.
## Step 1: Explore first, then tell me your plan
Before changing anything, find out and report back in a few lines:
1. **The voice.** Which TTS or speech-to-speech provider produces the agent's voice (OpenAI TTS, OpenAI Realtime,
ElevenLabs, Azure, Cartesia, Deepgram, Fish Audio…), in which file, and where that audio is played today.
2. **Where the audio lives.** On the server, or only in the browser (e.g. OpenAI Realtime over WebRTC straight from
the page)? The avatar needs the voice on the server.
3. **The server.** Language and framework. The SDK's server part needs Node.js (22+, or 18/20 with `WebSocket`
from the "ws" package).
4. **The page.** Which component shows the conversation, and the UI framework (React, Next.js, Vue, Svelte…).
5. **LiveKit.** Does the project already use LiveKit rooms?
Then state which mode you'll use (Step 2) and the files you'll touch, and implement.
**Stop and ask me first** if you'd need to move the voice from the browser to the server, add a Node.js service
next to a non-Node backend, or change authentication or other features that play audio.
## Step 2: Pick the mode
- **The project uses LiveKit** → LiveKit mode: the avatar joins the existing room as a participant.
- **Otherwise** → Direct mode: the server starts the avatar, and the page shows it with `LiveAvatarView`.
- **The voice agent runs in Python** (LiveKit Agents, Pipecat…) or another non-Node backend → propose a small
Node.js service that runs the avatar session and receives the agent's audio from it, and ask me before building it.
## Step 3: Install and configure
- `npm install @interviewflowai/talking-avatar` in the Node.js server, and in the web app (Direct mode).
- Read the API key from the env var `TALKING_AVATAR_API_KEY`, and add it to `.env.example` and any config
validation. Keys look like `tav_…` and are created at https://app.talking-avatar.dev → API keys.
- **Never** ask me to paste the key into chat, hard-code it, commit it, or send it to the browser.
## Step 4: Server (`@interviewflowai/talking-avatar/server`)
```js
import { LiveAvatar, LiveAvatarError } from "@interviewflowai/talking-avatar/server"
// One avatar per conversation: start it when the conversation starts.
const avatar = await LiveAvatar.start({
apiKey: process.env.TALKING_AVATAR_API_KEY,
avatar: "ava6", // any avatar ID from the Avatars page, e.g. ava6 (casual, home study) or ava4 (professional, office)
sampleRate: 24000, // the TTS audio's real rate: 8000 | 12000 | 16000 | 24000 | 48000
// LiveKit mode only: livekit: { url, token } (see Step 5b)
})
avatar.pushAudio(chunk) // every chunk of the agent's voice as it streams (PCM16 mono)
const heardMs = await avatar.clear() // the user interrupted: drops unplayed audio, returns ms of the reply heard
avatar.on("speechStarted", () => {})
avatar.on("speechEnded", (playedMs) => {})
avatar.on("close", (code, reason) => {}) // 1000 closed by you · 4001 bad key · 4002 balance under 1 min · 4003 3-hour limit
avatar.close() // the conversation is over: billing stops
```
`pushAudio` takes PCM16 mono as an `Int16Array`, `Buffer` / `Uint8Array` / `ArrayBuffer` (chunks may split a
sample) or a base64 string. Audio can arrive faster than real time: it's queued and spoken at its natural pace.
Direct mode also needs **one route** for the page (adapt it to the framework; protect it with the app's existing auth):
```js
app.post("/avatar", async (req, res) => {
try { res.json(await avatar.handleView(req.body)) }
catch (error) { res.status(503).json({ error: error.message }) }
})
```
Rules:
- **Lifecycle.** Exactly one `LiveAvatar` per conversation. `close()` it when the conversation ends, the user
leaves, or anything fails, so billing stops. Several conversations at once → one route each, e.g.
`/avatar/:conversationId` looking up that conversation's avatar.
- **Play the voice once.** The avatar plays the voice itself, in sync with the lips. Remove or disable the old
playback of the agent's audio (an `<audio>` element, a speaker stream, `track.attach()`…) for this agent only.
- **Raw PCM.** Ask the TTS for raw 16-bit mono PCM and pass the real sample rate. If it can only return MP3, Opus or
WAV, decode to PCM first (strip WAV headers).
- **Interruptions.** Call `clear()` wherever the app already handles barge-in.
- **Errors.** `LiveAvatar.start` rejects with a `LiveAvatarError` (`.code` as above) if a session can't start.
Show a clear message (e.g. 4002 → "add credits"); never crash the voice agent because the avatar failed.
Provider recipes (use the one that matches):
- **OpenAI Realtime (WebSocket, server side):** for `response.output_audio.delta` events (`response.audio.delta` on
older versions) call `avatar.pushAudio(event.delta)` (base64 PCM16, 24 kHz). On
`input_audio_buffer.speech_started`: `const ms = await avatar.clear()`, then send
`{ type: "conversation.item.truncate", item_id: <the assistant item playing>, content_index: 0, audio_end_ms: ms }`
so the model only remembers what the user heard.
- **OpenAI TTS:** `response_format: "pcm"` → 24 kHz PCM16 mono; stream the response body into `pushAudio`.
- **ElevenLabs:** `output_format: "pcm_24000"` (or `pcm_16000`, matching `sampleRate`).
- **Azure Speech:** `Raw24Khz16BitMonoPcm`.
- **Cartesia:** `{ container: "raw", encoding: "pcm_s16le", sample_rate: 24000 }`.
## Step 5a: Page (Direct mode, `@interviewflowai/talking-avatar/live`)
```js
import { LiveAvatarView } from "@interviewflowai/talking-avatar/live"
const view = new LiveAvatarView({
container: element, // the video fills it: 16:9, never cropped; or pass your own <video>
endpoint: "/avatar", // or (body) => fetch("/avatar", { method: "POST", headers, body: JSON.stringify(body) }).then(r => r.json())
onStatus: (status) => {}, // connecting | live | muted (call view.unmute() from a click) | reconnecting | closed
onError: (error) => {},
})
startButton.onclick = () => view.connect() // from a user click, so the browser allows the voice
// view.disconnect() when the conversation UI goes away
```
- Use the app's framework: create the view on mount (e.g. React `useEffect` with a ref), `disconnect()` on unmount.
- Connect from a user gesture (the existing "start call" button is ideal).
- It reconnects by itself after network drops; show its status if the UI has a place for it.
## Step 5b: LiveKit mode
- On the server, mint a **publish-only** LiveKit token for the avatar: identity `avatar-<room>`, grant
`{ room, roomJoin: true, canPublish: true, canSubscribe: false }`, and pass `livekit: { url, token }` to
`LiveAvatar.start`. Your LiveKit secrets stay on your server.
- The avatar joins with a video track named `avatar` and its voice as `avatar-audio`. Render them with the
existing LiveKit client, and stop playing the agent's original voice track so it isn't heard twice.
- Skip `LiveAvatarView` and the `/avatar` route.
## Step 6: Verify, then report
- Run the project's own type-check, lint, tests and build.
- If you can, run it: the avatar appears and moves while listening, the lips match the voice, the voice is heard once
(not doubled), interrupting stops it, and ending the conversation calls `close()`.
- Report: what you changed (files), the env var to set, how to try it, and anything you couldn't verify.
Pricing: $0.02 per minute of live session, billed per second from the dashboard's prepaid credits.
Docs: https://www.npmjs.com/package/@interviewflowai/talking-avatar