Article reader Listen + reading controls
Article reader
Preparing the reader…
Reading settings
AI performance is a workflow requirement¶
International Business Machines (IBM) and Intel have reported a substantial throughput improvement for natural-language processing tasks after integrating software optimizations with newer processors. The engineering result is a reminder that artificial intelligence performance comes from a stack: model, library, compiler, hardware, deployment environment, and workload.
For product teams, one more layer belongs in that stack—the human workflow.
Faster is meaningful only in context¶
Natural-language processing (NLP) supports tasks such as classification, sentiment analysis, entity extraction, and text understanding. More throughput can lower cost and help a system serve more users. Lower latency can also change how a person works with the system.
A response that arrives in a fraction of a second feels interactive. One that arrives after a long pause may interrupt thought, encourage task switching, or cause users to bypass the tool. In a batch process, the same delay may be irrelevant. In a time-sensitive operational setting, it may determine whether the output can support the decision at all.
This is why “faster” should not be a free-floating objective. The team should specify the service experience the work requires.
Optimization can move the bottleneck¶
Improving model throughput does not guarantee a faster workflow. Data retrieval, network calls, authentication, post-processing, human review, and downstream systems may dominate elapsed time. A team can celebrate a 100-percent improvement at one layer while users see no practical difference.
The inverse is also true. Speeding the model may flood reviewers with more output than they can inspect. The technical system becomes more efficient while the sociotechnical system accumulates a queue.
Theory of constraints offers a simple discipline: improve the system by understanding its limiting constraint, then observe where the constraint moves. For AI-enabled work, the limiting factor may be compute, data quality, user attention, approval authority, or recovery from uncertain cases.
Latency shapes trust and judgment¶
Response time is not merely an engineering metric. It communicates something to the user. Very fast output can feel authoritative, even when the underlying model is uncertain. A longer wait can imply effort without improving quality. Interface designers should avoid letting speed become an accidental confidence signal.
Research on human interaction with automation shows that the appropriate level of automation depends on the cognitive stage and task. The right design might deliver an immediate classification for triage, then a slower explanation with source evidence for a consequential decision. Not every output needs the same performance target.
Define a workflow-level budget¶
Teams can make performance requirements more useful by allocating a budget across the full path:
- time to acquire and validate input;
- time to retrieve supporting information;
- time for model inference;
- time for policy and safety checks;
- time for human review;
- time to act or hand off; and
- time to recover when the system cannot proceed.
The budget should be paired with quality and risk thresholds. A faster model is not an improvement if it increases rework, hides uncertainty, or moves verification outside the measured process.
Performance tests should also use realistic input distributions and concurrency. A laboratory average conceals tail latency—the slowest responses that often occur under the conditions when the service is most needed.
Optimize the outcome, not the component¶
Hardware and software co-optimization will be essential as organizations put more language models into production. The IBM and Intel result shows how much improvement can be found below the application layer. Product and program leaders should meet that engineering with equal clarity above it.
Ask what decision or action becomes better when the system is faster. Then measure the entire path to that outcome. Artificial intelligence performance is not a race for the lowest number on an infrastructure dashboard. It is the ability to deliver the right support at the pace the work can responsibly use.
Sources and research trail¶
- IBM Research, “How Intel oneAPI Tools Are Accelerating IBM's Watson Natural Language Processing Library” (May 18, 2023).
- Intel, oneAPI Deep Neural Network Library documentation.
- Parasuraman, Sheridan, and Wickens, “A Model for Types and Levels of Human Interaction with Automation” (2000).
- Card, Robertson, and Mackinlay, The Information Visualizer: An Information Workspace (1991).