SciFi Futures
← Capabilities
Computing & Information/Component technology

Visual speech recognition (lip reading)

Transcribing speech from video of the mouth alone, without audio, at a distance and from arbitrary angles.

State of the art

Visual-only LRS3 WER ~27% for AV-HuBERT-class models (2024); 18.7% claimed by an LLM-assisted method (2025 preprint).

What’s blocking it
  • Many phonemes look identical on the lips (visemes)
  • Performance falls sharply off-angle, at low resolution or with new speakers
  • Benchmarks are small and partly contaminated

Who’s working on it

  • Maja PanticResearcher
    Imperial College London (iBUG group)

    Co-authored Auto-AVSR (ICASSP 2023), which set the audio-visual LRS3 state of the art using automatically transcribed training data

  • Richard BowdenResearcher
    University of Surrey

    Co-authored VALLR (ICCV 2025), reaching 18.7% visual-only word error rate on LRS3 with 99.4% less labelled video via a phoneme-then-LLM pipeline

  • Released AV-HuBERT (2022), the self-supervised audio-visual representation that nearly all subsequent lip-reading systems initialise from

  • Symphonic LabsStartup

    Launched the readtheirlips.com public demo and the MAMO macOS app (2024) for issuing silent voice commands, the first consumer-facing silent-speech product

  • LiopaStartup

    Belfast company whose SRAVI product does viseme-based lip reading for clinical patients who cannot vocalise, the only regulated deployment of the technology

  • Google DeepMindCompany

    Produced the LipNet and large-vocabulary LSVSR work that established sentence-level visual speech recognition is possible at all

What to invest in

KindInvestmentScale
researchLarge multi-angle audiovisual datasets and robust evaluation sets$$ $10M – $100M

Depends on

General-purpose foundation models
Language priors resolve ambiguous visemes
TRL 7

Required by

No other capability builds on this directly.

Sci-fi concepts in reach

HAL 9000 ship computerdirect

Sources