Visual speech recognition (lip reading)
Transcribing speech from video of the mouth alone, without audio, at a distance and from arbitrary angles.
State of the art
Visual-only LRS3 WER ~27% for AV-HuBERT-class models (2024); 18.7% claimed by an LLM-assisted method (2025 preprint).
- Many phonemes look identical on the lips (visemes)
- Performance falls sharply off-angle, at low resolution or with new speakers
- Benchmarks are small and partly contaminated
Who’s working on it
- Maja PanticResearcherImperial College London (iBUG group)
Co-authored Auto-AVSR (ICASSP 2023), which set the audio-visual LRS3 state of the art using automatically transcribed training data
- Richard BowdenResearcherUniversity of Surrey
Co-authored VALLR (ICCV 2025), reaching 18.7% visual-only word error rate on LRS3 with 99.4% less labelled video via a phoneme-then-LLM pipeline
- Meta AI ↗Company
Released AV-HuBERT (2022), the self-supervised audio-visual representation that nearly all subsequent lip-reading systems initialise from
- Symphonic LabsStartup
Launched the readtheirlips.com public demo and the MAMO macOS app (2024) for issuing silent voice commands, the first consumer-facing silent-speech product
- LiopaStartup
Belfast company whose SRAVI product does viseme-based lip reading for clinical patients who cannot vocalise, the only regulated deployment of the technology
- Google DeepMindCompany
Produced the LipNet and large-vocabulary LSVSR work that established sentence-level visual speech recognition is possible at all
What to invest in
| Kind | Investment | Scale |
|---|---|---|
| research | Large multi-angle audiovisual datasets and robust evaluation sets | $$ $10M – $100M |
Depends on
Required by
No other capability builds on this directly.