Skip to main content

​
The pipeline

  1. Audio → visemes. A ~260k-parameter causal convolutional network reads 80-band log-mel frames (16 kHz, 10 ms hop) and outputs one of 22 visemes every 10 ms, looking 20 ms ahead. It runs as plain JavaScript, with no WebAssembly or ML runtime. Streams are captured in an AudioWorklet and resampled on the fly; clips are analysed whole before they play.
  2. Visemes → mouth. A new viseme must hold for 20 ms, so one-frame flickers are ignored. The timeline is shaped with JALI-style coarticulation (Edwards et al., SIGGRAPH 2016): tongue-only sounds borrow the lip shape of the neighbouring vowel, shapes ramp in ahead of their sound, and p/b/m always close the lips.
  3. Mouth → avatar. Viseme weights drive the avatar’s viseme morph targets, plus ARKit lip rounding, lip press and a little jaw when available, with a speed limit so shapes glide rather than snap.

​
Accuracy and limitations

  • On 39 speakers held out from training, with clean read speech passed through the Opus codec (as on a WebRTC call), the model picks the right mouth shape for 82% of 10 ms frames, and closes the lips on 99% of p/b/m sounds. Noisy or unclear audio scores lower.
  • English-trained. On Spanish, French, German, Portuguese, Hindi, Arabic, Japanese and Mandarin its timing holds up, but sounds English lacks are drawn with the nearest English mouth shape.
  • One audio source at a time per avatar.
  • The avatar plays the audio. It can’t lip sync audio played somewhere else.
  • Streams are delayed by streamDelayMs (150 ms by default).
  • Only the face moves: mouth, natural blinks and brows. No gaze or head motion.

​
Performance

​
Browser support

Needs WebGL 2 and Web Audio with AudioWorklet, which current Chrome, Edge, Firefox and Safari provide. Live audio (microphone, WebRTC) needs HTTPS or localhost. Content Security Policy: the audio capture worklet is loaded from a blob: URL, so script-src must allow blob:. The avatar and the model (modelUrl, jsDelivr by default) are fetched, so connect-src must allow both.

​
Model and training data

The model was trained on LibriTTS-R (Koizumi et al., Interspeech 2023), licensed CC BY 4.0 and derived from LibriTTS and LibriSpeech. Phoneme timings came from the Montreal Forced Aligner (McAuliffe et al., Interspeech 2017) and were mapped to visemes. The training data itself isn’t redistributed.

​
Privacy

Everything runs in the browser. The only network requests are for the avatar and the model file; audio never leaves the page.