Skip to content
Article reader Listen + reading controls
LISTEN + READ YOUR WAY

Article reader

Preparing the reader…

0:00 0:00
Reading settings
Text size
100%

AI Data Poisoning Is a Knowledge-Supply-Chain Problem

Originally published in 2024; substantially revised in 2026 to deepen the analysis and incorporate additional sources.

NIST’s warning about adversarial manipulation of AI systems should change how organizations define the security boundary. Conventional software security focuses heavily on code, dependencies, infrastructure, identity, and configuration. AI systems add another attack surface: the evidence from which system behavior emerges.

Training corpora, feedback data, retrieval indexes, evaluation sets, model artifacts, prompts, and operational context all influence what an AI system learns or produces. If an adversary can shape those inputs, the system may remain technically available while becoming epistemically compromised.

AI expands the software supply chain into a knowledge supply chain. Security must protect not only what the system executes, but what it is permitted to believe.

Manipulation can occur across the lifecycle

NIST’s adversarial-machine-learning taxonomy distinguishes among poisoning, evasion, privacy, and abuse attacks and emphasizes that existing mitigations do not provide a foolproof defense (Vassilev et al., 2024).

Those attack families appear at different points:

  • Training-time poisoning introduces data intended to degrade performance or create a hidden behavior.
  • Evasion alters inputs after deployment so that the model misclassifies or behaves unexpectedly.
  • Privacy attacks infer sensitive information about training data, model behavior, or internal representations.
  • Abuse attacks compromise legitimate information sources that an AI system later consumes.

Generative and retrieval-augmented systems make the final category especially important. An attacker may not need access to model weights or a protected training pipeline. It may be enough to influence a document repository, public website, knowledge base, feedback channel, or tool output that the system treats as evidence.

The boundary between cybersecurity and knowledge management therefore collapses. Content governance becomes a security control.

Scale makes provenance more—not less—important

Large datasets are often assumed to dilute malicious or low-quality examples. NIST’s analysis cautions against that intuition: some attacks can succeed by controlling a comparatively small number of samples (NIST, 2024).

Scale also makes manual inspection impossible. Organizations need machine-readable provenance:

  • Where did this data originate?
  • Which transformations were applied?
  • Who or what approved it for this use?
  • Which model, index, or evaluation consumed it?
  • What changed between versions?
  • Can a suspicious source be isolated and its downstream effects identified?

Provenance is not a decorative metadata field. It determines whether an organization can investigate, reproduce, correct, and recover from a knowledge-integrity incident.

Retrieval-augmented generation changes the threat model

Retrieval-augmented generation is frequently proposed as a way to ground language models in authoritative enterprise knowledge. It can improve relevance and traceability, but it also creates a dynamic path through which compromised content can influence outputs.

A secure retrieval architecture should distinguish among at least four questions:

  1. Authority: Is this source permitted to inform this task?
  2. Integrity: Has the content or its metadata changed unexpectedly?
  3. Relevance: Does the retrieved passage actually support the claim being made?
  4. Conflict: Do other authoritative sources disagree, and can the system expose that disagreement?

Similarity is not authority. A vector search may identify text that is semantically close to a query while ignoring whether the source is current, approved, authentic, or appropriate for the decision. Retrieval policies should combine semantic relevance with identity, provenance, classification, recency, source reputation, and task-specific rules.

Defenses must be layered around uncertainty

Because no universal mitigation exists, organizations should design for prevention, detection, containment, and recovery.

Prevent

  • Control write access to training, evaluation, and retrieval sources.
  • Sign and version datasets, models, prompts, and indexes.
  • Separate untrusted ingestion from trusted publication.
  • Require explicit data contracts and accountable owners.
  • Apply secure-development practices across the AI lifecycle.

Detect

  • Monitor source changes, data distributions, retrieval patterns, and model behavior.
  • Use canary records and integrity checks for critical corpora.
  • Compare performance across stable evaluation sets and adversarial scenarios.
  • Investigate abrupt changes in user overrides, refusals, or downstream outcomes.

Contain

  • Limit the authority of model outputs.
  • Segment data and retrieval domains by mission and risk.
  • Preserve the ability to disable a source, model, tool, or index independently.
  • Design safe fallback modes when integrity is uncertain.

Recover

  • Maintain lineage sufficient to identify affected artifacts and decisions.
  • Support rollback and re-indexing from trusted versions.
  • Record operational incidents as reusable evaluation cases.
  • Reassess approvals when the evidence supporting them changes.

Joint guidance from NSA, CISA, and international partners similarly frames AI security across secure design, development, deployment, and operation rather than as a model-only concern (NSA, 2023).

The governance question is who owns knowledge integrity

In many enterprises, no single function owns the end-to-end integrity of AI evidence. Cybersecurity protects systems. Data teams operate pipelines. Knowledge managers curate repositories. Model teams train and evaluate. Product owners define use cases. Business or mission users interpret outputs.

An attacker exploits the seams among those responsibilities.

Effective governance should assign ownership for:

  • Approving sources for specific AI uses
  • Defining integrity and provenance requirements
  • Monitoring changes and suspicious behavior
  • Evaluating the effect of compromised evidence
  • Deciding when a capability must be constrained or withdrawn
  • Communicating uncertainty to affected users and decision-makers

This is a networked-accountability problem. Evidence crosses organizational boundaries, so responsibility must travel with it.

The strategic takeaway

AI security is not complete when infrastructure, code, and model endpoints are protected. The organization must also secure the knowledge environment that shapes system behavior.

That requires a synthesis of cybersecurity, data engineering, knowledge architecture, model evaluation, and operational governance. The objective is not to prove that manipulation is impossible. It is to make the evidence chain sufficiently visible and modular that attacks can be prevented where possible, detected when they occur, contained before they propagate, and corrected without rebuilding the entire system.

References

READER-NEUTRAL SUBSCRIPTION

Follow Field Notes via RSS.

Copy this address into the RSS reader you already use. New notes will appear there automatically—no account, email address, or tracking required.

Open the feed