Audio sources

The avatar lip syncs any audio you give it, and plays that audio itself. There are two inputs.

Audio clips

speak takes a URL, an ArrayBuffer or a Blob, in any format the browser can decode (MP3, WAV, Ogg, AAC…). The clip is analysed in full before it starts (in roughly 1% of its duration), then played with no added delay.

await avatar.speak("/greeting.mp3");          // resolves when the clip ends or is stopped
await avatar.speak(await ttsResponse.blob()); // e.g. straight from a TTS API
avatar.stopSpeaking();

Calling speak again stops the current clip.

Live audio streams

attachStream takes any MediaStream with an audio track: a WebRTC remote stream, getUserMedia, or element.captureStream().

await avatar.attachStream(stream); // plays it and lip syncs it
avatar.detachStream();             // stop

Attaching a new stream replaces the previous one. detachStream doesn’t stop the source track; whoever owns it does.

Stream delay

The stream plays streamDelayMs (default 150 ms) late. The model hears each sound that long before the listener does, so the mouth can move into it on time. Lower it for less latency, at the cost of rougher lip sync:

new TalkingAvatar({ container, avatarUrl, streamDelayMs: 80 });

Voice agents

Give the agent’s incoming audio to attachStream instead of playing it in an <audio> element.

OpenAI Realtime (WebRTC)
LiveKit
Daily
Any WebRTC stack

const pc = new RTCPeerConnection();
pc.ontrack = (event) => avatar.attachStream(event.streams[0]);

With livekit-client, for example with LiveKit Agents:

import { RoomEvent, Track } from "livekit-client";

room.on(RoomEvent.TrackSubscribed, (track) => {
  if (track.kind === Track.Kind.Audio) {
    avatar.attachStream(new MediaStream([track.mediaStreamTrack])); // don't call track.attach()
  }
});

With @daily-co/daily-js, for example with Pipecat:

call.on("track-started", ({ track, participant }) => {
  if (track.kind === "audio" && !participant?.local) avatar.attachStream(new MediaStream([track]));
});

Find the remote audio MediaStreamTrack and pass new MediaStream([track]):

function onTrack(event) {
  if (event.track.kind !== "audio") return;
  avatar.attachStream(new MediaStream([event.track])).catch(console.error);
}
pc.addEventListener("track", onTrack);

// When leaving the conversation:
// pc.removeEventListener("track", onTrack);
// avatar.detachStream();
// avatar.dispose();

Microphone

const mic = await navigator.mediaDevices.getUserMedia({ audio: true });
await avatar.attachStream(mic);

The avatar plays the microphone back, so use headphones to avoid echo. Live audio needs HTTPS (or localhost).

Listening and thinking

The avatar knows when it’s speaking from its own audio. In between, it looks like it’s listening: attentive brows and a relaxed blink rate. While your agent is working on a reply, show it thinking:

avatar.setStatus("thinking");  // e.g. after the user stops talking
avatar.setStatus("listening"); // back to the default

Speech detected in the audio always takes priority over the status.

Best results

Use TTS voices for the best lip sync. The model was trained on clean, clearly articulated read speech, which is what TTS engines produce. Real voices, such as from a microphone, also work, but with background noise, echo or unclear speech the lip sync is less accurate.

One audio source at a time per avatar: attaching a stream replaces the previous one. A clip and a stream can overlap.
The avatar plays the audio. It can’t lip sync audio that’s played somewhere else without also playing it.

Get started

Your account

Voice Agents (no code)

Realistic Avatars (API)

Open Source (3D, free)

Help

Audio clips

Live audio streams

Stream delay

Voice agents

Microphone

Listening and thinking

Best results

​Audio clips

​Live audio streams

​Stream delay

​Voice agents

​Microphone

​Listening and thinking

​Best results

Audio clips

Live audio streams

Stream delay

Voice agents

Microphone

Listening and thinking

Best results