Model the builders, not only the models¶
We usually evaluate high-stakes AI by inspecting the artifact: the model, benchmark, system card, test report, or approval package. But before deployment, accountability is created—or eroded—by a population of builders acting through partial information, uneven authority, deadlines, review queues, and AI-mediated tools.
That makes builder-side AI development a natural candidate for agent-based modeling.
The builder side is a collective system¶
By builder-side development engineering, I mean the engineering of the conditions under which consequential systems are built: who sees what, which evidence travels, how uncertainty is represented, where dissent can survive, when waivers become invisible, and who can change the decision.
The relevant actors include developers, evaluators, domain experts, product and program managers, independent reviewers, decision authorities, and—increasingly—AI-mediated development tools. None sees the whole system. Each acts within a different combination of expertise, incentives, authority, workload, and organizational memory.
The resulting outcome is therefore not reducible to one person's judgment or one governance control. It emerges from repeated local interactions across a review network.
This is the kind of problem agent-based modeling was built to explore. Rather than treating an organization as a set of aggregate variables, an agent-based model represents heterogeneous actors, their relationships, the rules under which they act, and the system-level patterns those interactions produce.
Process maps are not enough¶
A formal process map tells us how work is supposed to move. It rarely captures the work required to make that process function: the informal handoff that restores missing context, the reviewer who recognizes a weak signal, the manager who protects time for another test, or the summary that quietly converts uncertainty into apparent consensus.
CSCW has long studied this gap between prescribed workflow and cooperative work as actually performed. Its attention to situated practice, articulation work, shared awareness, and boundary-spanning coordination is especially valuable here. Software repositories and collaboration tools also leave increasingly rich traces of that work—edits, comments, reviews, approvals, reassignments, exceptions, and escalation events.
Those traces can tell us what happened. Interviews and close workplace study can help explain what the recorded events meant. Agent-based modeling can then formalize candidate mechanisms and ask what might happen under different conditions.
That combination matters. A simulation without empirical grounding can become an elaborate story. Trace analysis without a mechanism can remain an elegant description. Together, they can support a more cumulative science of pre-deployment accountability.
What an initial model could contain¶
An early model does not need to recreate an entire firm. It needs enough structure to test a sharply defined mechanism.
| Layer | Candidate representation |
|---|---|
| Agents | Developers, evaluators, domain experts, managers, reviewers, decision authorities, and AI tools |
| State | Workload, expertise, authority, trust, uncertainty, incentives, deadlines, and memory |
| Network | Task dependencies, review ties, escalation paths, and access to evidence |
| Rules | Route, summarize, challenge, waive, escalate, approve, or request more evidence |
| Outcomes | Hazard detection, escalation latency, minority-view survival, waiver burial, false consensus, review burden, and provenance completeness |
Consider a narrow question: When does an unresolved concern reach someone with the authority and context to act?
The model could vary network centralization, reviewer workload, deadline pressure, evidence visibility, and the thresholds actors use to escalate. It could compare lateral review, sequential review, liaison roles, or redundant independent paths. It could test whether an AI-generated summary reduces coordination cost while also attenuating minority views. It could examine when adding another reviewer increases hazard detection—and when it merely diffuses ownership.
The aim would not be to predict the exact behavior of a particular organization. It would be to identify plausible mechanisms, discover interaction effects, and generate propositions that can be challenged with evidence.
A synthetic-first, empirically calibrated path¶
This fits a research sequence I am developing around trace-based pre-deployment accountability.
First, build measurement infrastructure. Reconstruct decision provenance from development and review traces. Identify whether uncertainty was surfaced, preserved, escalated, waived, or suppressed—and where the observable record becomes incomplete.
Second, explain the dynamics. Use process mining, event-history analysis, network analysis, qualitative evidence, and deliberately simple agent-based models to connect local behavior to organizational outcomes.
Third, test interventions. Compare routing policies, review structures, escalation mechanisms, and AI-mediated decision support. The target is not maximum oversight. It is better hazard detection and calibrated judgment without creating surveillance, unworkable bottlenecks, or ritual compliance.
Early models can be synthetic-first: explicit agents, explicit rules, transparent assumptions, and controlled experiments. Later versions can be calibrated against open-source development activity, public incident records, controlled studies, or partner data. Sensitivity analysis and the ODD protocol can make model structure and uncertainty inspectable rather than hiding them behind a compelling animation.
This sequence also keeps the model in its proper role. It is a mechanism-based laboratory—not a digital twin, an oracle, or a substitute for observing real cooperative work.
AI agents belong in the model—but not as human stand-ins¶
There is an important distinction between an agent in an agent-based model and an autonomous AI agent. The first is an analytical representation governed by specified rules. The second is becoming an actual participant in software development: generating code, conducting reviews, summarizing evidence, routing work, and sometimes recommending decisions.
Both belong in this research, but they should not be conflated.
Recent work with generative agents demonstrates that language models can produce surprisingly coherent individual and collective behavior. That creates opportunities for richer simulation and rapid prototyping. It also creates a validity trap. Fluency can make an artificial actor seem more empirically grounded than it is. Studies of synthetic human responses have found meaningful promise alongside compressed variation, prompt sensitivity, and unreliable statistical inference.
For builder-side research, I would begin with interpretable, mechanism-focused agents wherever possible. Language-model behavior should enter only where it represents a real feature of the development system or where it can be validated against observed behavior. The model should earn complexity rather than inherit it from a foundation model.
The research question underneath the model¶
The deeper question is not whether an organization has a governance process. It is whether its development and review system allows consequential uncertainty to remain visible, credible, and actionable long enough to affect a deployment decision.
Agent-based modeling offers a way to investigate that question across levels: individual judgment, team coordination, network structure, organizational policy, and emergent system behavior. CSCW keeps the inquiry anchored in real work. Trace-based methods create empirical leverage. Simulation creates a disciplined space for counterfactuals.
Used together, they could help us move from lists of responsible-AI requirements toward a testable theory of how accountability is actually produced by the people, tools, and relationships building the system.
Research trail¶
- Macy and Willer, “From Factors to Actors: Computational Sociology and Agent-Based Modeling” (2002), on local interaction and emergent social patterns.
- Harrison, Lin, Carroll, and Carley, “Simulation Modeling in Organizational and Management Research” (2007), on simulation as a method for organizational theory.
- Crowder, Robinson, Hughes, and Sim, “The Development of an Agent-Based Modeling Framework for Simulating Engineering Team Work” (2012), an empirically informed model of engineering teams.
- Smaldino, Calanchini, and Pickett, “Theory Development with Agent-Based Models” (2015), on using ABM to make behavioral theory explicit.
- Grimm and colleagues, “The ODD Protocol for Describing Agent-Based and Other Simulation Models” (2020), a standard for transparent model description and evaluation.
- Blanco-Fernández, Leitner, Rausch, and Wall, “Interactions Between Dynamic Team Composition and Coordination” (2024), an ABM of coordination, learning, task complexity, and team performance.
- Omoronyia, Ferguson, Roper, and Wood, “Using Developer Activity Data to Enhance Awareness During Collaborative Software Development” (2009), on reconstructing collaborative context from developer activity traces.
- Wilson and colleagues, “Dimensions of Human-Machine Combination” (2025), on studying human-machine decision systems in situated practice.
- Park and colleagues, “Generative Agent Simulations of 1,000 People” (2024), an empirical approach to grounding generative behavioral agents in qualitative interviews.
- Bisbee, Clinton, Dorff, Kenkel, and Larson, “Synthetic Replacements for Human Survey Data? The Perils of Large Language Models” (2024), an important warning about variation, reliability, and reproducibility in LLM-generated human data.