Article reader Listen + reading controls
Article reader
Preparing the reader…
Reading settings
Trustworthy autonomy needs more than a better neural network¶
The Defense Advanced Research Projects Agency (DARPA) has selected teams for its Assured Neuro Symbolic Learning and Reasoning (ANSR) program. The program will explore architectures that combine data-driven neural learning with symbolic representations and reasoning, with the aim of improving robustness and assurance for autonomous systems.
The research matters because high performance and trustworthy behavior are not the same achievement.
Learning and reasoning fail differently¶
Neural models are powerful at finding patterns in complex data. They can also behave unpredictably outside the distribution represented in training, and their internal reasoning may be difficult to inspect. Symbolic systems can express rules and relationships explicitly, but they may be brittle when the world does not fit the representation.
Combining the two may allow a system to learn from data while preserving some structure for reasoning, constraints, and explanation. It also creates a harder integration problem. The assurance claim must address not only each component, but how they interact when they disagree.
Artificial intelligence (AI) architecture should be selected according to the mission and evidence needed, not fashion. A more complex hybrid is justified only if it improves the system's ability to satisfy meaningful constraints.
Assurance begins with claims¶
“Trustworthy” is too broad to test directly. A team needs concrete claims such as:
- the system will remain within a defined operating area;
- it will not select an action that violates a stated safety constraint;
- it will recognize specified classes of uncertainty;
- it will explain which rule or evidence shaped a consequential choice; or
- it will transition to a safe state when key inputs become unreliable.
Each claim requires an argument and evidence. Some evidence may come from formal analysis of symbolic constraints, some from statistical testing of learned components, and some from realistic human–machine trials.
The assurance case must also state where the claim does not apply.
Mission context determines acceptable confidence¶
DARPA describes a progression from gaming environments toward an intelligence, surveillance, and reconnaissance demonstration. That progression is important. A method can look robust in an abstract environment and encounter new dynamics, sensors, adversaries, and time constraints in a mission setting.
The National Institute of Standards and Technology (NIST) Artificial Intelligence Risk Management Framework emphasizes mapping the context before selecting measurements. For autonomy, that includes operational tempo, communications, human authority, failure consequences, and the possibility of deliberate deception.
Testing should challenge the relationships the symbolic layer assumes and the patterns the neural layer learned. Unknown situations are not edge cases in defense; they are part of the environment.
Make disagreement observable¶
A hybrid architecture creates a potentially valuable signal: the learned model proposes an action that the symbolic constraints reject, or symbolic reasoning reaches a conclusion the perception model cannot support confidently. The system should preserve and expose that disagreement rather than force silent resolution.
Depending on the mission, disagreement might trigger a safer action, request for human input, collection of more information, or a record for later review. Operators need to know what the signal means and how frequently it occurs.
Transition the evidence with the technology¶
If ANSR produces promising methods, receiving programs will need more than source code. They will need the assumptions, formal properties, training distributions, scenario results, integration dependencies, and known failure modes that support the assurance argument.
That evidence should be developed alongside the architecture. Retrofitting an explanation after transition usually reveals that the decisive assumptions were held tacitly by the research team.
Trustworthy autonomy will not emerge from a single algorithmic breakthrough. It will come from matching architecture to mission, specifying constraints, testing the full system under surprise, and preserving evidence that operators and decision-makers can challenge.
ANSR is asking an important technical question about combining forms of AI. Its operational value will depend on an organizational answer: can the program turn that combination into a system whose limits are as legible as its capabilities?
Sources and research trail¶
- Defense Advanced Research Projects Agency, “ANSRs to Hard AI Questions” (September 25, 2023).
- National Institute of Standards and Technology, Artificial Intelligence Risk Management Framework (AI RMF 1.0) (2023).
- Garcez et al., Neural-Symbolic Learning and Reasoning (2002).
- Leveson, Engineering a Safer World (2011).
- Department of Defense, Responsible Artificial Intelligence Strategy and Implementation Pathway (2022).