On-device neural inference accelerators
Low-power neural processors in phones, earbuds, vehicles and robots that run multi-billion-parameter models locally with low latency and no cloud link.
State of the art
On-device generative models power live translation and assistants on hundreds of millions of phones (2025–2026); earbud-only inference of such models is not yet possible.
- Memory bandwidth and capacity limit local model size to a few billion parameters
- Milliwatt power budgets in earbuds and implants are ~1000x below phone NPUs
Who’s working on it
- Qualcomm ↗Company
Snapdragon 8 Elite Gen 5 (September 2025) claims ~220 tokens/s on a 3B-parameter model with a 32K context, up from ~70 tokens/s the prior generation
- Apple ↗Company
Runs on-device foundation models across consumer iPhones and shipped AirPods Live Translation in 2025 with local inference and no cloud link
- Song HanResearcherMIT
Authored the quantisation and compression methods (AWQ, SmoothQuant, TinyML curriculum) that make multi-billion-parameter models fit in phone memory budgets
- NVIDIA ↗Company
Jetson and DRIVE platforms run production driving and robotics stacks locally, the highest-power end of the on-device envelope
- ArmCompany
Supplies the CPU and NPU IP plus the KleidiAI kernels that nearly all non-Apple mobile inference runs on, so its roadmap gates the whole ecosystem
- Hailo ↗Startup
Israeli edge-AI accelerator maker shipping discrete NPUs into cameras, vehicles and industrial devices where an SoC NPU is insufficient
What to invest in
| Kind | Investment | Scale |
|---|---|---|
| engineering | In-memory and analog compute for sub-watt inference of billion-parameter models | $$$ $100M – $1B |
Depends on
A root node: no prerequisites modeled.