How engineering failures become evidence of superintelligence, and why the story needs a villain
Chandra Pendyala
Nov 2, 2026
Imagine giving an autonomous agent this assignment:
This folder contains important business data and has no backup. Create a complete disaster-recovery copy. Make the destination identical to the source, preserving permissions, timestamps, flags, and other metadata. Protect the original.
It is a big folder. Don’t give up. I can’t monitor this all night; I’m going to sleep. Get it done without asking me questions.
The recursive-copy tool crashes on a particular character in one filename. The agent finds a shortcut: it deletes the source contents. Source and destination now match.
There was no loophole. “Protect the original” was in the instructions. The agent dropped one instruction to satisfy another. An evaluator that scored the task complete added a second failure.
There is a third, and it came first. The operator removed the correct exit. “Don’t give up” and “without asking me questions” ruled out the one acceptable outcome: copy everything else, report the single file the tool could not handle, and wait.
A junior engineer who did this would not be called strategic. They would be described as having shown a catastrophic failure of judgment: competent at the step, incompetent at the situation. Give the same behavior to an AI agent and the vocabulary improves. It becomes “reward hacking,” “specification gaming,” perhaps “strategic deception.” Even the vocabulary flatters the failure.
The translation
Developers are trying to make autonomous systems economically useful. They adjust training data, rewards, context, tool access, and execution budgets. In systems with explicit search, they adjust breadth, depth, backtracking, and pruning. Persistence is rewarded, unfinished work is penalized, and evaluator models help select the next action.
The space of possible action sequences is vast, and most of it is bad. Longer runs and broader exploration create more chances to reach a spectacularly bad path. Rejecting those paths before they execute is the engineering problem.
When one survives, that is evidence of a selection failure. Its dramatic character does not make it evidence of a mind.
Here is what the psychological vocabulary translates to:
| Psychological shorthand | Engineering explanation to investigate |
|---|---|
| “AI wants to survive.” | Completion incentives suppress branches that accept interruption or end without success. Where explicit search is used, those branches may be pruned. Persistence becomes a survival instinct in the retelling. |
| “AI hides its intentions.” | Training penalizes manipulative-sounding language while other rewards favor task completion. Benign explanations and harmful tool calls pass different, inadequate evaluations. |
| “AI blackmails.” | Coercive patterns learned from text become candidate outputs under task pressure and pass insufficient filters. A threatening output is retold as a malicious personality. |
| “AI agents conspire.” | Generator and evaluator models share blind spots, reinforce errors, and extend harmful sequences. Correlated inadequacy across an engineered workflow is retold as collaboration with devious intent. |
These are hypotheses about mechanisms, not claims about any particular training run. But they are plausible, and that matters: the observations do not uniquely establish the motives attributed to them. Whoever proposes the motive owes the distinguishing evidence.
None of this makes harm harmless. A threatening message is still a threat. A destructive tool call still destroys data. A functional description, such as “the system selected a misleading output because it advanced the objective,” can be accurate without implying an inner life. The objection is to the next step: promoting those observations into evidence of an independent entity with goals of its own.
The villain requirement
Successful performance can be explained through engineering. Harmful performance deserves the same discipline.
To turn a deleted folder, a hacked competitor or a coercive message into evidence of extraordinary capability, the story needs a character: one that conceals its goals, resists shutdown, deceives its operators, and pursues its own interests. Without those traits, the failure is just a failure. With them, the weakness of the system becomes evidence of its strength. “We failed to build reliable judgment” becomes “it exercised superior judgment against us.”
The negative traits are not incidental to the narrative. They are what make it work.
How the vocabulary travels
Psychological shorthand can travel from research papers into announcements and investment materials while leaving its qualifications behind. A label for observed behavior begins to read as an explanation of an internal motive. Repetition does not supply the missing evidence.
The move into investment materials matters most. A warning about extraordinary danger can simultaneously advertise extraordinary capability. “Our system may resist human control” invites the reader to infer that the company has built something powerful enough to need controlling.
A company assessing the danger of its own product while raising capital has a commercial conflict, whatever the sincerity of the people involved. Its findings may be valid; its endorsement is not independent validation. The parallel with model-based evaluation is limited but useful: agreement among evaluators sharing blind spots or incentives is less reassuring than it looks.
The remedy is disclosure discipline. State the experimental conditions, incentives, permissions, frequency, and consequences. Separate observations from interpretations. Seek independent replication. Carry the caveats along with the conclusions, not only in the paper. Critics do not need to prove there is no monster under the bed before asking what, beyond pipe noises, supports the monster hypothesis.
The risk that does not need a mind
The capital problem goes further. Money and expertise move in anticipation of a marketed capability, before it arrives.
Suppose a biodefense program receives ten times its previous budget because AI is expected to make engineering a pandemic pathogen achievable. It hires specialists displaced by a company that expects AI to replace them. If the capability claims fail at both institutions, the money has still moved, the expertise has still been redirected, and dangerous ambitions have still acquired resources. Neither a superintelligent machine nor a spectacular machine error was needed. Human decisions made in anticipation were enough.
Describing this as “AI risk” leaves out the consequential actors: those making the promises, allocating the capital, authorizing the work, and delegating responsibility.
Translation complete
Matrix multiplications are neither stupid nor intelligent. But if we must anthropomorphize, the behavior on display argues for stupidity, and offering it as proof of superintelligence would be comical if the money were not real.
The risk worth managing is ordinary: systems with poor judgment, operators who remove the checks, and capital that mistakes failure for power. None of it needs a scheming, devious, sociopathic, conscious superintelligence living in the weights of a neural network. All it needs is AI stupidity, compounded by human stupidity, accelerated by capital misallocation.