▶ Article reader Listen + reading controls ⌄

LISTEN + READ YOUR WAY

Article reader

Preparing the reader…

▶ Play article ← → Stop

0:00 0:00

Speed 0.75× 1× 1.25× 1.5× 1.75×

Reading settings

Typeface Field Notes Atkinson Hyperlegible Lexend OpenDyslexic

Text size

A−

100%

A+

Reading surface Field Notes dark Paper light Warm sepia High contrast Text spacing Standard Comfortable

Reset reading preferences

Trustworthy AI needs a research agenda, not a slogan

Researchers from academia, industry, and government are gathering this week for the Defense Advanced Research Projects Agency's Artificial Intelligence (AI) Forward workshop. The agenda centers on a question that is easy to state and difficult to engineer: how can AI systems operate reliably, interact appropriately with people, and support national-security needs under demanding conditions?

Calling a system trustworthy does not make it so. Trustworthiness has to be decomposed into research questions, engineering evidence, and operational learning.

The word hides several different problems

AI can fail because it encounters unfamiliar data, because an adversary manipulates its input, because the objective was specified poorly, because the human–machine interface encourages overreliance, or because the organization cannot respond when performance changes.

Those are not one problem. They require different disciplines and evidence.

The Defense Advanced Research Projects Agency (DARPA) highlights foundational theory, AI engineering, and human–AI teaming as major research areas. That division is useful. Theory helps establish what can be known or bounded. Engineering turns methods into repeatable systems. Human–AI research examines how capability and responsibility are distributed in actual work.

A research portfolio should also make the connections among them explicit. A theoretical guarantee may depend on assumptions the operational environment violates. An engineering control may shift workload to a human who cannot use it under time pressure. An interface may improve trust without improving reliability.

National-security conditions expose brittle assumptions

Many commercial AI systems assume abundant connectivity, stable data, frequent updates, and users who can step away when the service is uncertain. Defense systems may operate with degraded communications, adversarial interference, scarce computation, damaged sensors, and decisions that cannot wait.

Trustworthiness in that setting includes graceful degradation. The system should fail in a way the operator can recognize and manage. It should preserve essential function when a data source disappears and communicate when it has crossed the conditions under which it was validated.

The Department of Defense (DoD) Responsible Artificial Intelligence Strategy and Implementation Pathway connects policy principles to implementation lines such as governance, warfighter trust, product acquisition, requirements validation, and workforce. That is the right instinct: trustworthiness must be built through an enterprise, not delegated to model developers.

Research should produce transition evidence

Research programs often end with a paper, prototype, or demonstration. Operational transition requires a different package:

• the claim the method supports;
• the conditions and assumptions under which it was tested;
• datasets and scenarios used for evaluation;
• known failure modes and negative results;
• integration and compute requirements;
• human training and interface implications; and
• unanswered questions that the receiving program must own.

This package preserves the reasoning behind the result. Without it, the receiving team sees a capability but cannot judge whether it travels.

Carlile's work on knowledge boundaries is relevant to defense innovation. Research and acquisition communities do not merely use different words; they face different incentives and constraints. Transition requires translation and often transformation of the technology and the receiving process.

Organize the unknowns

A useful outcome from AI Forward would be a living research agenda that distinguishes:

1. problems with mature methods ready for engineering;
2. problems with promising methods but weak operational evidence;
3. problems whose evaluation remains immature; and
4. assumptions that cannot yet be justified.

That map would help program leaders avoid two errors: treating an open research question as a procurement requirement, or continuing to research a problem that needs disciplined field engineering.

Trustworthy AI is a worthy goal, but the phrase should create work rather than end a discussion. DARPA's convening can help the community identify which parts demand theory, which demand integration, and which can be learned only through carefully bounded use with operators.

The real measure of the agenda will be whether it helps programs convert uncertainty into testable claims—and carry the resulting evidence all the way to the people asked to rely on the system.

Sources and research trail

• Defense Advanced Research Projects Agency, “Join DARPA to Reimagine the Future of AI for National Security” (February 24, 2023; workshop held July 31–August 2).
• Department of Defense, Responsible Artificial Intelligence Strategy and Implementation Pathway (2022).
• National Institute of Standards and Technology, Artificial Intelligence Risk Management Framework (AI RMF 1.0) (2023).
• Carlile, “A Pragmatic View of Knowledge and Boundaries” (2002).
• Lee and See, “Trust in Automation: Designing for Appropriate Reliance” (2004).