What building a production harness for agentic coding says about how much software engineering is still left to do
July 2026 — Chandra Pendyala
| ABSTRACT: Most production-grade code-generation models have crossed a real usefulness threshold; the question that matters now is not which model to buy but what workflow surrounds it. Two structural facts keep coding from being solved by the model alone: generation is stochastic, and it does not yet own any stage of the platform life cycle end to end, though it meaningfully cuts drudgery at nearly every stage. Evidence from randomized trials, large-scale commit analysis, and industry surveys shows the effect is highly conditional — an amplifier of existing engineering discipline rather than a substitute for it. Production experience building a software-engineering harness around code generation found that inexpensive models performed comparably to frontier models once that discipline was encoded into the process, at a fraction of the review cost of ungoverned agentic output. The higher-leverage investment is not model selection. It is the engineering system around code generation. |
Where Are We With Coding?
The AI coding tool market is crowded: Claude Code, Codex, GitHub Copilot, Cursor, Devin, OpenCode, Hermes, and a long tail of wrappers, each committed to a distinct workflow — IDE-native, terminal-native, ticket-autonomous, or model-agnostic. Having worked across most of them, and across models ranging from open-weight systems in the Qwen class to frontier models such as Sol and Fable, my conclusion is that most production-grade models have crossed a real usefulness threshold. Model choice now matters at the margin; workflow dominates.
That capability shift raises a sequence of harder questions, in order: is coding solved by AI, is it delivering value consistently and reliably, what do the research and my own experience say about that, and what actually works right now.
Is Coding Solved?
No, not fully by itself — for two reasons that compound each other.
First, the generator is stochastic. Earlier RAD tools and low-code systems were deterministic: the same input produced the same output, so their failure modes were fully characterizable and testable in advance. An LLM-based generator is probabilistic — the same prompt can yield a different draft each time, and a plausible-looking draft is not evidence that the underlying logic is sound. A capability whose output has to be re-verified every time it runs, because it is not guaranteed to be the same output twice, is a productivity aid, not a solved problem.
Second, and more consequentially, code generation does not yet cover the full range of modes a platform lifecycle actually requires — not because it fails to help with them, but because it does not yet own any of them end to end. Software engineering is often treated, in discussions of AI, as if it were mostly the act of writing new feature code. In a mature technology firm, code authorship is a small fraction of total platform effort — plausibly in the low single digits for established platforms. The larger and more durable work is deciding what should exist, architecting it, operating it, securing it, and sustaining it under competitive and customer pressure: platform life cycle management. A platform’s life before code — conception, justification, architecture, funding — and its far longer life after code — release, operation, governance, extension, migration, retirement — both dwarf the coding phase itself, and each stage demands its own mode of code generation: migration scripts, rollback and compatibility shims, security patches, observability instrumentation, deprecation handling, infrastructure as code. Current tools can meaningfully cut the drudgery inside nearly every one of these modes — a migration script’s boilerplate, a rollback shim’s scaffolding, a first draft of the IaC module — but they do not yet run any of these modes unsupervised the way a competent engineer with judgment does. The gap is coverage of judgment, not coverage of effort.
Each row below implies a distinct code-generation demand that current tools were not built around, and a successful platform accumulates all of them at once as it acquires users, integrations, extensions, field deployments, version histories, and compatibility commitments that must be sustained while the platform continues to evolve:
| Platform Reality | Pain |
|---|---|
| Competitive pressure | Roadmap never stops |
| Ecosystem integration | Partners, APIs, SDKs, connectors |
| Client customization | Tenant-specific behavior and extensions |
| Field engineering | Real deployments reveal edge cases |
| SRE and operations | Incidents, SLOs, observability, capacity |
| Version management | Multiple supported releases |
| Backward compatibility | Old clients depend on old behavior |
| API governance | Contracts, deprecations, auth scopes |
| Developer tools | SDKs, CLIs, docs, samples, sandboxes |
| Bug inflow | Triage, reproduction, prioritization |
| Change management | Users, admins, integrators, regulators |
| Upgrade orchestration | Feature flags, rollback, compatibility tests |
Against that table, current AI code generation is best understood not as a capability confined to one lifecycle stage, but as something that strips drudgery out of almost every stage without yet being trustworthy to run any of them unsupervised. Rapid prototyping — describing a screen, workflow, or integration and getting a plausible working draft in minutes — is simply the case where drudgery is the largest share of the work and judgment the smallest, which is why it looks like the generator’s natural home. It is not new: we have had RAD tools, 4GLs, scaffolding frameworks, and low-code systems for decades, and the speed and flexibility here is a real, welcome jump, not a different kind of thing. But it is the clearest place to see the effect, not the only place it operates. A prototype can be generated. A platform must be carried — and the drudgery-reduction described above helps with the carrying too, at almost every step, even where the judgment still has to come from a person.
Historically, the capability itself is a continuation of a project computer science has run for seventy years: reducing the accidental labor of coding, from punch cards to assembly, to FORTRAN and COBOL, to C and SQL, to 4GLs and low-code platforms, to LLM-based generation. Grace Hopper’s compiler work sits in the same lineage — with the difference, again, that Hopper’s compiler was deterministic and today’s generator is not. Hopper held a PhD in mathematics, not computer science — the discipline did not yet exist as a formal one — and built the first compiler in 1952 on the argument that people should write instructions in English-like words and let the machine handle the translation into executable code. That argument is the direct ancestor of every high-level language since, LLM-based generation included. AI changes the interface to programming. It does not eliminate the disciplines required to build correct, secure, reliable, valuable systems.
Is It Delivering Value Consistently and Reliably?
Not yet, for the same two reasons — plus one qualifier that matters more than either: both are already manageable, which is what makes the unevenness an organizational fact rather than a technical ceiling.
Stochastic output is manageable wherever verification is cheap and success criteria are clean: UI prototypes, scaffolding, boilerplate, first-pass migrations, documentation, and repository exploration, particularly where the target resembles well-represented public examples — a Trello clone, a standard auth flow, a CRUD admin screen. A wrong draft costs little to catch and re-run. The lifecycle-fit problem is manageable wherever an organization has already done the platform thinking — architecture, specs, tests — that current tools don’t generate on their own. Where neither condition holds, agentic coding is most exposed exactly in the domains that define real platform work: permissions, data semantics, transaction boundaries, validation logic, versioning, deployment safety, and observability. The difficulty in these domains was never the typing.
Whether those conditions hold is an organizational fact more than a technical one: AI-assisted coding does not act uniformly on a population of developers or organizations; it amplifies whatever discipline already exists. Pointed at a codebase with weak tests, unclear architecture, and no review discipline, it does not fix the mess — it produces the mess faster and at greater volume than a human could alone. Pointed at a codebase with strong tests, explicit architecture, and cheap verification, the same tools yield a real, measurable gain. Between those poles, outcomes track an organization’s existing capability maturity more closely than they track which model or tool it has adopted. Senior developers realize measurable productivity gains from generative-AI adoption; early-career developers, despite being the heaviest users, show none [3]. A controlled study of programmers using AI assistants found they wrote less secure code than a control group while reporting higher confidence in its security [9] — the novice failure mode in miniature: output that runs, looks plausible, and survives a cursory glance, while the defects that matter go undetected because the user lacks the judgment to see them.
On the statistics available today, this is a system whose average outcome is unimpressive, which is a separate fact from how people are choosing to use it: developers are increasingly unwilling to do routine work without AI assistance, whether or not it measurably helps them. That preference was strong enough to affect METR’s own follow-up study — a large share of developers declined to complete tasks without AI tools at all, contributing to a selection effect the authors themselves flagged as limiting the study’s reliability [2]. The same pattern shows up industry-wide: AI coding tool adoption is near-universal even where trust in the output remains low [10]. A preference for using the tool is not the same thing as a productivity result, and it will keep usage high regardless of what the productivity data show.
What Does the Research Say?
The empirical record offers a more conditional picture than raw capability gains alone would predict. METR’s 2025 randomized controlled trial of experienced open-source developers found that permitting AI tool use made them 19 percent slower on tasks in repositories they already knew — despite developers expecting a 24 percent speedup and, afterward, still believing they had been 20 percent faster than they actually were [1]. METR’s 2026 follow-up reported more favorable but far less certain results: possible speedups among developers with sustained AI exposure, alongside the selection effect described above [2].
Larger-scale evidence on open-source commit velocity and quality points the same direction, with real texture beneath the headline numbers. A Science analysis of 30 million GitHub commits from 160,097 developers found that generative-AI adoption raises quarterly commit volume by roughly 2 to 4 percent, with the gain concentrated among senior developers and no measurable benefit for early-career contributors [3]. A difference-in-differences study of projects adopting the Cursor agent found a large but transient increase in development velocity accompanied by a persistent rise in static-analysis warnings and code complexity [4]. GitClear’s longitudinal analysis of 211 million changed lines reports a comparable pattern from a different angle: two-week code churn rose from 3.1 to 5.7 percent between 2020 and 2024, and within-commit duplication overtook refactored, moved code for the first time on record [5]. A large-scale study of AI-authored commits across 6,299 GitHub repositories reaches a similar conclusion about accumulating technical debt [6]. Not every recent finding cuts the same way: a survival analysis of 200,000 code units across 201 open-source projects found agent-authored code is modified less often than human-authored code, complicating any easy assumption that AI output is inherently disposable [7]. Industry survey data corroborates the review-burden mechanism directly — DORA’s 2025 report found that higher AI adoption raises delivery throughput and delivery instability together, with a sharp rise in median pull-request review time and in the share of pull requests merged without any review at all [8].
Read together, the honest conclusion is not that AI-assisted coding fails, but that its net effect on productivity and quality is highly conditional on workflow, experience, and review discipline — not on tool availability alone.
What Is My Experience?
This comes from paid production work, not a lab exercise. Over the last two years I have used these tools to build roughly half a dozen production systems; model choice changed, the pattern below did not. The most recent instance, delivered under deadline pressure, is representative: my harness gave a swarm of coding agents shared context, task structure, coordination, automated checks, and commit discipline, and the result was a near-production-quality version-one platform — not a prototype — generated in roughly twenty minutes, as a reviewable sequence of approximately 1,100 commits rather than a single opaque diff — close to the best case achievable for agentic code generation today. That is close to science-fiction productivity in traditional terms: the same first-draft implementation might have taken a mid-sized team a quarter.
That is sensational only if I do not tell you the full story. The system was fully specified by the time the code gen run started. It was a significant investment of time from a very senior architect to specify the system to that degree.When you inspect the specifications visually, you can almost see the code in your mind — the LLM was just a typist. Wireframes, ER diagrams, sequence diagrams, C4s, deployment diagrams, corporate look-and-feel libraries, use cases, error templates, Swagger files — a fully specified system. The specification itself was LLM assisted and was generated after a lot of small prototypes to nail down every detail. The work was in the specification. That was the trick.
What made that possible was not a magic model. In fact, once the harness was mature, inexpensive models performed just as well inside it as frontier models — largely, I suspect, because none of this work was architecturally novel. These platforms followed well-established patterns throughout, including on security: most enterprise security comes from cloud-vendor reference architectures rather than from case-by-case reasoning, so the model’s job is closer to following a known pattern correctly than inventing one. That is an inference from routine-task performance, not something I have tested directly, so I would rather offer it as a hypothesis than a finding — my experience is not the world.
None of this makes frontier improvements worthless. Longer context windows and better long-horizon reasoning genuinely make a model more pleasant and forgiving to work with directly — it needs less hand-holding, tolerates looser prompts, and recovers better from its own mistakes mid-task. What those improvements do not do on their own is make the model more productive on production work. That gain still runs through the harness, not through the model’s improved reasoning. The deepest, longest-horizon reasoner cannot infer on its own whether the code in front of it needs to survive a middle-school science fair or a Pentagon security review; that distinction has to come from outside the model, because no amount of reasoning capacity substitutes for being told what the actual stakes are. The variable that mattered was not which model, but whether rigorous software engineering discipline had been encoded into the code-generation process itself. Six tricks did that encoding:
| Harness Capability | Why It Mattered |
|---|---|
| Hierarchical context efficiency | Agents received the right level of system, architecture, module, and task context without flooding every step with everything. |
| Task decomposition | Large goals were broken into bounded work units with clear ownership and reviewable outputs. |
| Memory model | Decisions, constraints, patterns, and prior outputs were retained in a way the agents could use without constantly rediscovering them. |
| Parallel coordination | Multiple agents could work at once without trampling the same files or producing incompatible assumptions. |
| Repository software-engineering discipline | The system was specified without ambiguity using standard software-engineering artifacts — requirements, architecture, design, C4 diagrams, ERDs, wireframes, use cases, testing intent, and edge cases — each written in the precise language appropriate to its layer. |
| Tooling and guardrails | Skills, hooks, goals, verifiers, linters, tests, and checks worked together as part of a software-engineering design. |
The model supplied fluent generation. The harness supplied direction, memory, constraint, coordination, and verification — and that was the part that actually did the work.
One practical implication follows directly from this: a harness needs a routing mechanism that sends each task to the cheapest model capable of handling it and reserves frontier models for the work that actually requires them. That routing layer is not a nice-to-have bolted on for cost optimization later. It is the harness’s budget manager, and skipping it means paying frontier prices for routine work by default.
This discipline was not only useful for the agent swarm: IDE-assisted code generation also depended on it, since the same requirements, architecture, and test artifacts made small local changes faster to request, review, and accept.
On a combined generation-plus-review basis, the swarm still lost to conventional IDE-assisted development for core platform code. The reason is structural, not incidental: splitting output into more commits does not reduce review burden, it only reorganizes it. Each commit remains a claim against the codebase requiring verification of correctness, design rationale, test adequacy, and fit with the system a growing team must eventually own. IDE-assisted development wins here because review is continuous and local — each unit of generated code is inspected immediately, the developer’s mental model stays intact, and errors do not compound across the codebase.
This points to a different metric for evaluating agentic coding on production work: not code generation time, but time to understood, reviewed, tested, maintainable change. Agentic workflows can win decisively on the first quantity while losing on the second, and for core platform code the second is the one that determines whether the work was worth doing. Turns out a furious typist, however good, is not the bottleneck. Worse: parallel furious typing substantially increases the risk of catastrophic failure via compounding errors. Unreviewed generated code is not progress. It is inventory.
What Is the Best Solution Right Now?
Treat the harness as infrastructure, not as a prompt-engineering afterthought. AI does not eliminate the production machinery that distinguishes a durable platform from a prototype — product management, architecture governance, QA, release engineering, SRE, security, documentation, and support — but it does change its cost structure, letting a smaller team afford a lighter version of infrastructure that once required substantially more headcount. The platforms that became durable businesses spent years and hundreds of millions of dollars building this machinery; it looked like overhead, but it was the reason customers could trust the platform. Cheaper machinery is not the same as no machinery: someone still has to define architecture, maintain contracts, govern releases, and decide what the platform becomes.
For a long-lived platform with a sizable team, this argues for a role closer to an AI build-systems engineer than a prompt engineer — someone who owns the conditions under which larger-scope automation becomes safe: linters and type checks, unit and integration tests, golden datasets, architecture rules, security checks, PR templates that force explanation of tests, risk, and rollback, codebase documentation, C4 diagrams, and dependency maps, task templates that convert requests into bounded specs, CI hooks, and agent evaluations that measure whether the automation actually saves time. With that infrastructure in place, the task specification can shift from “build this feature” to a bounded specification referencing an existing service pattern, schema, validation rules, fixtures, authorization policy, and acceptance tests — a materially different proposition, in which the agent operates inside rails built from executable knowledge rather than roaming freely.
Many organizations aim for Level 4 or 5 on this ladder while their day-to-day practice sits closer to Level 1 or 2 — a gap the amplifier effect above helps explain, and one that closes with investment rather than with a better model:
| Level | Mode | Reality |
|---|---|---|
| 0 | Chatbot coding | Useful snippets |
| 1 | IDE assistant | Micro-task acceleration |
| 2 | Repo-aware assistant | Search, explain, local edits |
| 3 | Harnessed agent | Bounded tasks with tests and hooks |
| 4 | Team automation | Feature slices from formal specs |
| 5 | Autonomous engineer | Not yet achievable outside narrow domains |
The more durable investment is not purchasing autonomy but earning it: making the codebase legible, the tests strong, the architecture explicit, the business rules inspectable, and verification cheap. This investment pays off even if the agentic bet does not — better tests, documentation, architecture rules, and CI gates improve engineering quality independent of AI.
What Is the Future?
The industry is already moving in this direction, just not through the review-and-maturity layer yet. In 2026, spec-driven development became the dominant framing across major coding-agent vendors — GitHub Spec Kit, AWS Kiro, Claude Code, Cursor, OpenSpec, BMAD, Tessl, and Google Antigravity have each shipped a version of it, treating the specification rather than the code as the artifact of record that governs what an agent is allowed to produce [11]. That is real progress on the requirements-and-architecture-precision half of the harness described above.
It is not, by itself, an answer to the other half. A better spec reduces how often an agent guesses wrong; it does not reduce how much a human has to verify once the agent has acted on it, and it does not, on its own, tell an organization whether it is actually operating at Level 3 or only believes it is. The future most likely to deliver reliable value is not a smarter agent or a better spec format alone. It is the fuller harness — specification discipline paired with review discipline, testing, and an honest maturity assessment — becoming as ordinary and unremarkable as version control. That is a slower, less dramatic future than the one currently being marketed, and it is the one the evidence above points to.
AI lowers the cost of expression. It does not eliminate the need for judgment. Typing code, searching unfamiliar codebases, and writing first-draft documentation and tests are all genuinely painful work — let the machine absorb that pain.
Do not confuse generated code with delivered platform value.
References
[1] Becker, Rush, Barnes, and Rein, “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity,” METR, July 10, 2025; arXiv:2507.09089, July 12, 2025. https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/
[2] METR, “We Are Changing Our Developer Productivity Experiment Design” (uplift update), February 24, 2026. https://metr.org/blog/2026-02-24-uplift-update/
[3] Daniotti, Wachs, Feng, and Neffke, “Who Is Using AI to Code? Global Diffusion and Impact of Generative AI,” Science, 2026 (eadz9311); arXiv:2506.08945. https://www.science.org/doi/10.1126/science.adz9311
[4] He, Miller, Agarwal, Kästner, and Vasilescu, “Does AI-Assisted Coding Deliver? A Difference-in-Differences Study of Cursor’s Impact on Software Projects,” arXiv:2511.04427, November 2025. https://arxiv.org/abs/2511.04427
[5] Harding, “AI Copilot Code Quality: 2025 Data Suggests 4x Growth in Code Clones,” GitClear, February 2025. https://www.gitclear.com/ai_assistant_code_quality_2025_research
[6] Liu, Widyasari, Zhao, Irsan, Chen, and Lo, “Debt Behind the AI Boom: A Large-Scale Empirical Study of AI-Generated Code in the Wild,” arXiv:2603.28592, March 2026. https://arxiv.org/abs/2603.28592
[7] Rahman and Shihab, “Will It Survive? Deciphering the Fate of AI-Generated Code in Open Source,” arXiv:2601.16809, January 2026. https://arxiv.org/abs/2601.16809
[8] DORA (Google Cloud), State of AI-Assisted Software Development, 2025. https://dora.dev/dora-report-2025/
[9] Perry, Srivastava, Kumar, and Boneh, “Do Users Write More Insecure Code with AI Assistants?,” Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security (CCS ’23); arXiv:2211.03622. https://arxiv.org/abs/2211.03622
[10] TechCrunch, “Coders Are Refusing to Work Without AI — and That Could Come Back to Bite Them,” May 29, 2026. https://techcrunch.com/2026/05/29/coders-are-refusing-to-work-without-ai-and-that-could-come-back-to-bite-them/
[11] Microsoft for Developers, “Spec-Driven Development: A Spec-First Approach to AI-Native Engineering,” 2026. https://developer.microsoft.com/blog/spec-driven-development-ai-native-engineering/
Findings above draw on peer-reviewed studies, arXiv preprints, and industry reports as cited, plus unpublished production experience flagged inline as such where the two are not equivalent; peer-review status varies by source. Working paper — July 2026.
Tagged #SoftwareEngineering #AgenticCoding #PlatformEngineering #DeveloperProductivity