Article reader Listen + reading controls
Article reader
Preparing the reader…
Reading settings
Data at Speed and Scale Is a Control Problem¶
Calls for defense technologies that can collect, store, process, monitor, analyze, and transmit data at scale naturally attract products that maximize one dimension: sensor volume, link capacity, database throughput, analytic speed, or model performance.
Mission systems do not win by maximizing any of those quantities independently. They win by preserving a useful relationship between a changing environment and an accountable decision.
That makes data at speed and scale a control problem. The system must sense, estimate, decide, act, observe the result, and adapt—while delays, uncertainty, adversaries, and limited resources affect every stage.
The Office of the Under Secretary of Defense for Research and Engineering’s 2024 “Unleashing Data at Speed and Scale” meeting sought technologies across the data lifecycle. An official Innovation & Modernization overview situated the meeting among recurring outreach and experimentation events. The breadth was appropriate; the architecture must prevent the breadth from fragmenting into disconnected optimizations.
Freshness Is More Important Than Raw Speed¶
Data can move quickly and still arrive too late. The relevant measure is freshness at the decision: the age of the underlying observation after collection, transmission, processing, fusion, model inference, human review, and display.
Every mission thread has a freshness budget. If a targeting decision requires information within seconds, the budget must be allocated across the chain. Improving one segment yields little value if another dominates latency.
Teams should instrument:
- observation time;
- ingestion and queue delay;
- transformation and fusion time;
- inference time;
- policy and cross-domain delay;
- user-notification and review time;
- and action time.
The data product should expose age and uncertainty to the user. A low-latency interface displaying stale source data creates an illusion of speed.
More Collection Can Reduce Decision Quality¶
Additional sensors and sources can improve coverage. They also increase contention, duplication, contradiction, storage, processing, and cognitive burden.
A system needs admission and prioritization policies:
- Which observations are mission-relevant?
- Which source is authoritative for which claim?
- When should data be summarized, sampled, compressed, or discarded?
- Which events require raw evidence to be retained?
- How should conflicting reports be represented?
- Which users and models receive priority during congestion?
These policies encode mission value. They should be explicit, testable, and adjustable within authorized bounds.
Backpressure matters. If a downstream analytic service cannot keep pace, the pipeline should not silently build an unbounded queue and deliver increasingly stale results. It should degrade visibly, shed lower-priority work, and preserve the information needed for the most consequential decisions.
Compute Placement Is Dynamic¶
Edge, theater, and enterprise computing offer different advantages. The right placement can change during the mission.
At the edge, systems can reduce latency and continue through disconnection but face limited power, compute, storage, and physical protection. Central environments can aggregate more data and support larger models but depend on communications and may be unable to return results in time.
The architecture should decompose workloads so that:
- time-critical filtering and bounded decisions occur near the source;
- broader fusion and model improvement occur where aggregation is valuable;
- policies and models synchronize when links permit;
- local systems retain known-good versions;
- and the user can see which context or capability is unavailable.
The placement decision should be driven by latency, survivability, data sensitivity, compute need, and human authority—not by allegiance to a cloud or edge strategy.
Uncertainty Must Survive the Pipeline¶
Data pipelines frequently convert uncertainty into apparent precision. A sensor produces a noisy observation; fusion assigns an identity; a model generates a score; a user sees a colored icon. At each step, caveats can disappear.
A trustworthy pipeline preserves:
- source and collection method;
- time and location uncertainty;
- confidence and competing hypotheses;
- transformations and model versions;
- missing or degraded inputs;
- and the distinction between observation, inference, and decision.
This provenance should be queryable and machine-readable. Users need the portion relevant to their decision, while auditors and engineers need enough detail to reconstruct the path.
Knowledge graphs can be useful because they represent claims and relationships rather than forcing uncertainty into a single flattened record. The graph must still be governed: conflicting claims should coexist with provenance rather than be resolved invisibly.
Graceful Degradation Is a First-Class Requirement¶
Contested systems should be designed around partial failure. Links will be intermittent, data sources compromised, clocks misaligned, models outdated, and services unavailable.
A resilient data system defines degraded modes:
- local cached operation;
- lower-fidelity models;
- reduced source sets;
- manual or rules-based fallback;
- delayed synchronization;
- and safe suspension of functions whose evidence is insufficient.
The system should communicate what changed and which decisions are no longer supported. Degradation is dangerous when the interface looks normal.
Testing should inject correlated failures, not only isolate a single component. A network outage may also prevent identity validation, model updates, and time synchronization. The mission effect emerges from the dependency graph.
Observability Must Connect to Mission Outcomes¶
Infrastructure monitoring answers whether services are running. Mission observability answers whether the decision loop remains effective.
It should connect:
- technical latency and error;
- data freshness and quality;
- model performance and drift;
- human workload and overrides;
- decision timing and confidence;
- and observed operational outcome.
This does not mean reducing every mission to a dashboard. It means preserving enough evidence to learn which technical changes materially improved or degraded the workflow.
The DoD Data Strategy’s emphasis on visible, accessible, understandable, linked, trustworthy, interoperable, and secure data provides necessary attributes. Control-loop observability connects those attributes to action.
Security Changes the Control Dynamics¶
An adversary can manipulate collection, flood pipelines, corrupt time, poison models, spoof identity, or exploit the urgency of a decision. Security cannot be implemented only at the perimeter.
The system needs continuous signals about:
- source authenticity and integrity;
- anomalous volume or distribution;
- policy and identity state;
- model and software provenance;
- unexpected transformations;
- and behavior inconsistent with the operating context.
Security controls also add latency and failure modes. Their cost belongs in the freshness budget, and their degraded behavior must be tested. A theoretically secure path that users bypass because it cannot meet mission tempo creates a different vulnerability.
The Strategic Inference¶
Speed and scale are properties of an end-to-end decision loop, not of a component benchmark.
The Department should evaluate candidate technologies through representative mission threads with freshness budgets, congestion, uncertainty, degraded communications, adversarial inputs, human decisions, and outcome measures. The key question is not how much data a component can process. It is whether the combined system preserves the right evidence for the right decision under pressure.
That framing turns a broad technology search into a coherent engineering problem. It also prevents maximum throughput from becoming a proxy for decision advantage.
This essay was substantially revised in July 2026 to replace the original meeting announcement with an evidence-based control-systems analysis.
For related work on cloud-to-edge architecture, observability, and mission-scale AI systems, see my portfolio or Xendev Labs.
References¶
- Office of the Under Secretary of Defense for Research and Engineering, Innovation & Modernization: Overview, March 2024.
- U.S. Department of Defense, “Official Explains DOD Data Strategy,” March 5, 2021.
- “Department of Defense in Search of Technologies for Collecting, Processing, Transmitting Data,” ExecutiveGov, January 2024. This report was the historical prompt for the original post.