> ## Documentation Index
> Fetch the complete documentation index at: https://docs.talking-avatar.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# How it works

> The model, accuracy, performance, browser support and training data.

## The pipeline

```text theme={null}
audio ──► [resample 16 kHz] ──► [log-mel, 10 ms frames] ──► [causal conv net] ──► viseme IDs
                                                                                     │
avatar ◄── [morph targets, speed-limited] ◄── [coarticulation: smooth mouth weights] ◄┘
```

1. **Audio → visemes.** A \~260k-parameter causal convolutional network reads 80-band log-mel frames (16 kHz, 10 ms hop) and outputs one of 22 visemes every 10 ms, looking 20 ms ahead. It runs as plain JavaScript, with no WebAssembly or ML runtime. Streams are captured in an `AudioWorklet` and resampled on the fly; clips are analysed whole before they play.
2. **Visemes → mouth.** A new viseme must hold for 20 ms, so one-frame flickers are ignored. The timeline is shaped with JALI-style coarticulation (Edwards et al., SIGGRAPH 2016): tongue-only sounds borrow the lip shape of the neighbouring vowel, shapes ramp in ahead of their sound, and p/b/m always close the lips.
3. **Mouth → avatar.** Viseme weights drive the avatar's viseme morph targets, plus ARKit lip rounding, lip press and a little jaw when available, with a speed limit so shapes glide rather than snap.

## Accuracy and limitations

* On 39 speakers held out from training, with clean read speech passed through the Opus codec (as on a WebRTC call), the model picks the right mouth shape for **82%** of 10 ms frames, and closes the lips on **99%** of p/b/m sounds. Noisy or unclear audio scores lower.
* **English-trained.** On Spanish, French, German, Portuguese, Hindi, Arabic, Japanese and Mandarin its timing holds up, but sounds English lacks are drawn with the nearest English mouth shape.
* **One audio source at a time** per avatar.
* **The avatar plays the audio.** It can't lip sync audio played somewhere else.
* **Streams are delayed** by `streamDelayMs` (150 ms by default).
* **Only the face moves:** mouth, natural blinks and brows. No gaze or head motion.

## Performance

| | |
| - | - |
| Code | \~22 KB (8 KB gzipped), plus `three` |
| Model | 1 MB, downloaded once and cached |
| CPU | \~2% of one core while audio plays (measured on an Apple M5 Pro; plan for 2–3× on low-end laptops) |
| Clip analysis | \~1% of the clip's duration, in 1-second chunks that yield to the page |
| Added latency | `streamDelayMs` for streams (default 150 ms); \~50 ms scheduling margin for clips |

## Browser support

Needs WebGL 2 and Web Audio with `AudioWorklet`, which current Chrome, Edge, Firefox and Safari provide. Live audio (microphone, WebRTC) needs HTTPS or `localhost`.

**Content Security Policy:** the audio capture worklet is loaded from a `blob:` URL, so `script-src` must allow `blob:`. The avatar and the model (`modelUrl`, jsDelivr by default) are fetched, so `connect-src` must allow both.

## Model and training data

The model was trained on [LibriTTS-R](https://www.openslr.org/141/) (Koizumi et al., Interspeech 2023), licensed [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) and derived from LibriTTS and LibriSpeech. Phoneme timings came from the [Montreal Forced Aligner](https://montreal-forced-aligner.readthedocs.io/) (McAuliffe et al., Interspeech 2017) and were mapped to visemes. The training data itself isn't redistributed.

## Privacy

Everything runs in the browser. The only network requests are for the avatar and the model file; audio never leaves the page.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.