Bidirectional encoding models of perception and imagery
Models mapping between brain activity and perceptual or imagined content in both directions, decoding what a person sees or recalls and predicting what activity would evoke a given experience.
State of the art
Mind-captioning methods generated sentences describing viewed and recalled silent videos from fMRI (Horikawa, Science Advances 2025); inner speech decoded in real time with up to 74% accuracy in four BrainGate participants (Cell 2025).
- Models are subject-specific; decoders do not transfer across brains without recalibration
- Non-invasive signals are slow and spatially coarse; invasive coverage is a few square centimetres
- Encoding (write) models are far behind decoding; we cannot specify the activity for an arbitrary percept
Who’s working on it
- Tomoyasu HorikawaResearcherNTT Communication Science Laboratories
Mind captioning: generated text descriptions of viewed and recalled videos from fMRI (Science Advances 2025)
- Alexander HuthResearcherUniversity of Texas at Austin
Non-invasive semantic language decoding (2023) and its cross-participant transfer (Current Biology 2025)
- Francis WillettResearcherStanford University
Co-led real-time decoding of inner speech from motor cortex in BrainGate participants (Cell 2025)
- Edward ChangResearcherUCSF
Speech neuroprosthesis work decoding attempted speech to text and voice from cortical surface recordings
- Meta FAIR Brain and AI teamCompany
Brain2Qwerty v2 decoded natural sentences from MEG with 39% word error rate across nine subjects (2026 preprint)
- Robert GauntResearcherUniversity of Pittsburgh
Works on the write direction, mapping ICMS parameters to reported touch percepts (2025)
What to invest in
| Kind | Investment | Scale |
|---|---|---|
| research | Multi-subject datasets pairing intracranial recordings with rich naturalistic stimuli, to train cross-subject foundation decoders | $$ $10M – $100M |
| research | Closed-loop write experiments in primates that test whether model-specified stimulation produces the predicted percept | $$ $10M – $100M |