SciFi Futures
← Capabilities
Computing & Information/Component technology

On-device neural inference accelerators

Low-power neural processors in phones, earbuds, vehicles and robots that run multi-billion-parameter models locally with low latency and no cloud link.

State of the art

On-device generative models power live translation and assistants on hundreds of millions of phones (2025–2026); earbud-only inference of such models is not yet possible.

What’s blocking it
  • Memory bandwidth and capacity limit local model size to a few billion parameters
  • Milliwatt power budgets in earbuds and implants are ~1000x below phone NPUs

Who’s working on it

  • Snapdragon 8 Elite Gen 5 (September 2025) claims ~220 tokens/s on a 3B-parameter model with a 32K context, up from ~70 tokens/s the prior generation

  • Apple ↗Company

    Runs on-device foundation models across consumer iPhones and shipped AirPods Live Translation in 2025 with local inference and no cloud link

  • Song HanResearcher
    MIT

    Authored the quantisation and compression methods (AWQ, SmoothQuant, TinyML curriculum) that make multi-billion-parameter models fit in phone memory budgets

  • NVIDIA ↗Company

    Jetson and DRIVE platforms run production driving and robotics stacks locally, the highest-power end of the on-device envelope

  • ArmCompany

    Supplies the CPU and NPU IP plus the KleidiAI kernels that nearly all non-Apple mobile inference runs on, so its roadmap gates the whole ecosystem

  • Hailo ↗Startup

    Israeli edge-AI accelerator maker shipping discrete NPUs into cameras, vehicles and industrial devices where an SoC NPU is insufficient

What to invest in

KindInvestmentScale
engineeringIn-memory and analog compute for sub-watt inference of billion-parameter models$$$ $100M – $1B

Sci-fi concepts in reach

Self-driving carsdirectExocortex (wearable distributed mind)indirectFull-body haptic immersion rigindirectThe MetaverseindirectBabel fish universal translatordirectSelf-aware supercomputer (Mike)indirect

Sources