SciFi Futures
← Capabilities
Computing & Information/Foundational science

Formal verification of neural network behavior

Mathematical proof that a neural network satisfies a specified property (never outputs X under condition Y) for all inputs in a defined set, rather than only on tested examples.

State of the art

SDP-CROWN (ICML 2025) verifies models up to 2.47M parameters; VNN-COMP 2025 benchmarks remain small vision, control and NLP models.

What’s blocking it
  • Verification cost scales exponentially with network size for complete methods
  • Behavioral rules like "do not harm a human" have no formal specification over raw inputs
  • Properties proven on bounded input sets do not cover open-world inputs

Who’s working on it

  • Huan ZhangResearcher
    University of Illinois Urbana-Champaign

    Leads alpha-beta-CROWN, which won VNN-COMP every year from 2021 through 2025 and defines the state of the art in bound-propagation verification

  • Open-source verifier that topped all scored benchmarks at VNN-COMP 2025; proves robustness properties for networks up to a few million parameters

  • International Verification of Neural Networks Competition (VNN-COMP)Program

    Its sixth edition (2025) ran 8 teams over 16 regular and 9 extended benchmarks, and is the field's only standardised measure of what is provable at what scale

  • Clark BarrettResearcher
    Stanford University

    Co-created Marabou, the SMT-based verifier used where bound propagation is too loose, and co-organises the benchmark standardisation effort

  • Zico KolterResearcher
    Carnegie Mellon University

    Originated the convex-relaxation approach to certified robustness that every current verifier descends from, and now also chairs OpenAI's safety committee

  • DARPA Assured Neuro Symbolic Learning and Reasoning (ANSR)Program

    Funds hybrid neuro-symbolic architectures designed so that safety properties are statable and provable, attacking the specification gap rather than the solver

  • Swiss company that worked with EASA on Concepts of Design Assurance for Neural Networks (2020-2021), the first regulatory framework for certifying a neural network in a safety-critical aircraft system

What to invest in

KindInvestmentScale
researchScalable verifiers for transformer-scale models and compositional proofs over modules$$ $10M – $100M
researchFormal specification languages for behavioral constraints of embodied agents$$ $10M – $100M

Sci-fi concepts in reach

HAL 9000 ship computergatingindirectPositronic robots bound by the Three LawsgatingdirectCulture Minds (benevolent superintelligences)indirectTechnoCore AI civilizationindirect

Sources