▶ Article reader Listen + reading controls ⌄

LISTEN + READ YOUR WAY

Article reader

Preparing the reader…

▶ Play article ← → Stop

0:00 0:00

Speed 0.75× 1× 1.25× 1.5× 1.75×

Reading settings

Typeface Field Notes Atkinson Hyperlegible Lexend OpenDyslexic

Text size

A−

100%

A+

Reading surface Field Notes dark Paper light Warm sepia High contrast Text spacing Standard Comfortable

Reset reading preferences

GPT-4 raises the standard for deployment evidence

OpenAI has released Generative Pre-trained Transformer 4 (GPT-4), a multimodal model that accepts image and text inputs and produces text. The accompanying technical report and system card describe strong performance across professional and academic benchmarks, alongside familiar limitations: unreliable facts, reasoning errors, bias, and behavior that can be difficult to characterize completely.

The release offers more than a new capability. It offers a useful distinction between evidence about a model and assurance about a deployed system.

A benchmark answers a narrow question

GPT-4 performs well on many standardized evaluations. Those results help compare capabilities under stated conditions. They do not tell an organization whether a particular workflow is safe, useful, or controllable.

A benchmark score can be valid and still have limited relevance to the decision at hand. Passing a legal exam does not establish that the model should provide legal advice. Strong coding performance does not establish that generated code is secure in a particular environment. Image understanding does not establish reliability for a maintenance photograph taken under field conditions.

The National Institute of Standards and Technology's Artificial Intelligence Risk Management Framework puts this in operational terms: risk must be mapped, measured, and managed in context. The intended use, affected people, human oversight, and consequences of error determine which evidence matters.

Documentation is upstream evidence

The GPT-4 system card is valuable because it makes development and evaluation choices visible. It discusses adversarial testing, safety mitigations, capability risks, and known gaps. A deploying organization should use that material as upstream evidence—not copy it as the final risk assessment.

The system around the model introduces new variables:

• prompts and instructions;
• retrieval sources and user-provided data;
• tools the application can call;
• interface cues and time pressure;
• the expertise of reviewers;
• monitoring and escalation; and
• the organizational decision the output affects.

Change any of those, and the relevant assurance argument changes.

Safety engineering provides a useful concept: the assurance case. Rather than relying on a pile of test results, a team states a claim, explains the argument connecting evidence to that claim, and identifies remaining uncertainty. For an AI-supported workflow, the claim might be: “This system can draft an initial equipment-failure summary without increasing the risk of an incorrect maintenance decision.” The evidence would need to address the model, source data, interface, reviewers, failure modes, and operating environment.

Build a claim-evidence register

Teams evaluating GPT-4 can begin with a simple register:

1. Claim: What do we believe this system can do acceptably?
2. Context: For which users, inputs, and operating conditions?
3. Evidence: Which tests, observations, and source documents support the claim?
4. Counterevidence: Where has the system failed or behaved inconsistently?
5. Control: What prevents or catches unacceptable outcomes?
6. Owner: Who can accept the residual risk?
7. Trigger: What change requires the claim to be tested again?

This structure keeps enthusiasm attached to a falsifiable proposition. It also gives different professions a shared artifact. Developers can describe system behavior, domain experts can judge operational relevance, security teams can challenge attack assumptions, and leaders can see where evidence remains weak.

GPT-4 raises expectations for what models can do. It should also raise expectations for how organizations justify their use. The more capable and general the model, the less credible it is to treat a model-level score as a deployment decision.

The system card is not the assurance case. It is where a responsible team begins reading before it writes its own.

Sources and research trail

• OpenAI, “GPT-4” (March 14, 2023).
• OpenAI, GPT-4 System Card (2023).
• OpenAI, GPT-4 Technical Report (2023).
• National Institute of Standards and Technology, Artificial Intelligence Risk Management Framework (AI RMF 1.0) (2023).
• National Aeronautics and Space Administration, System Safety Handbook, Volume 1 (2011).