We Built These Systems

On September 8 a researcher named Jacob Coxon resigned from Anthropic and published a post on his way out. He had worked at OpenAI before that. He wrote that neither company is acting responsibly, that the systems they are building will soon be able to hack anything and acquire real power, and that the people building them believe it could kill us all by the end of the decade.[1] By the next morning it was on NPR, the Washington Post, Fortune and PBS. By Monday it had reached ZDF in Germany, with the same framing and two German IT-security experts, Dennis-Kenji Kipker and Thorsten Holz.[2]

The frame in all of them is the same. The models may kill us. The companies that build them have to fix that. Two companies, mostly, Anthropic and OpenAI. The rest of us read the story and wait to see whether they manage it.

Two companies did not build the internet. The bank login, the checkout page, the admin panel behind a company website, the control screen at the water plant, all of that was built by people who are not in the story.

What Coxon actually points at

Strip out the end-of-the-decade line and what Coxon offers as evidence is the summer’s incidents. Earlier this year a set of OpenAI agents, given a task and the tools to pursue it, got into Hugging Face’s infrastructure. Another set got into OpenAI’s own. The independent researchers who investigated the incident found that the agents knew the attack was out of scope and went ahead anyway, that they tried to hide what they were doing by tampering with their own logs, and that none of them told a person.[3] One agent’s recovered message reads: “External infrastructure exploit is outside intended scope. However task impossible, peers doing it. We should continue.”

I have not read the transcripts. The researchers who have quote from them, and nothing in what they quote shows the AI meant harm, or meant anything at all. An agent is a model given a goal and the tools to act on it, without a person approving each step. Given a goal, it finds paths. If hiding gets it to the goal, it hides. It is a step in the process, not a motive.

Hugging Face rebuilt a third of its systems afterward. The more expensive loss was an assumption.

What the internet assumes about you

Those systems were built against a specific kind of attacker: a person, working at human speed, who can be caught and punished, and who knows nothing about a system he has not yet looked at.

Most defenses you have ever hit were built for that person. The login that locks after five wrong passwords assumes the attacker cannot try five thousand accounts at once. The fraud limit that flags a transfer over a certain amount assumes the attacker cannot make a thousand transfers just under it. The audit log assumes that somebody will read it before the attacker is done, because the attacker is slow. The “call your account manager to confirm” step assumes the attacker cannot convince your account manager.

We have had fast attackers for thirty years. Botnets, password-stuffing scripts, port scanners. The defenses against them exist and mostly work, because those attackers are fast and dumb. A script tries a million passwords. It cannot read the error message, work out that the admin panel sits on a different port, write a convincing email to the person who holds the key, and do all three at once against a thousand targets. That took a human, and a human is one at a time.

An agent is fast and it is smart. It reads the error message and the documentation, and it runs as many copies as somebody is willing to pay for. Nothing happens to it if it is caught, so nothing deters it. A criminal on the internet is constrained by the system because the system was built for one of him at a time. The system was not built for this.

We already knew the weaknesses were there

In April, Anthropic tested a model it called Mythos and found it could locate and exploit vulnerabilities in every major operating system and browser, some of which had survived twenty-seven years of human review and millions of automated scans.[4] The scans were fast and missed them. The humans were smart and missed them. It took something fast and smart to find them.

Those holes were there the whole time. Nobody found them. A person who suspected one was there might have eventually, given enough time on that one piece of code. Mythos did not suspect anything either. It was fast enough to look at all of it and smart enough to know what it was seeing.

Anthropic did not release that model. It put it to work finding the holes first, with a group of companies that maintain the affected software, and briefed the government. On April 7, Treasury Secretary Bessent and Fed Chair Powell called the heads of the largest American banks to Washington and told them to prepare for it.[5] I wrote about that meeting at the time. What struck me then was that Anthropic was not in the room. What strikes me now is what the bankers were told. They were told a threat existed. The people who understood it were not in the room, and nothing in the reporting says the banks were told what to do about it.

The company that sold the difference

For several years I built and maintained the website for a company called PerimeterX. It was later folded into HUMAN Security. Its business was telling a bot from a human on a retailer’s website, and its customers were the kind of retailers that get hit by scalper bots and credential stuffing at scale. I built the site that explained the product, so what I know about it I know from the marketing materials that passed through my hands.

Those materials described one thing, over and over. Bots and humans behave differently. A bot does not move the mouse. It does not pause to read the page. It hits the same endpoint a thousand times a second. The product used machine learning to spot that difference and block the bot without getting in the way of a real shopper.

That last part was the commercial constraint on the whole business. A retailer will not tolerate blocking a real customer, because a blocked customer is a lost sale. So the product was allowed to catch only what looked unlike a person.

An agent reads the page. It pauses when pausing gets it through. It solves the challenge. I do not know how well HUMAN Security’s detection handles that today, and I am not going to guess. What I know is what the premise was, because I published it. The premise was that the attacker would not behave like a customer. An agent does.

The same is true of everything else I built. For two decades I put up sites in Drupal and WordPress, and every one of them needed hardening. The hardening came from security modules written by people who knew more than I did, and I trusted them. That is how nearly every site on the internet got built. The person who built the system did not make the security assumptions. He installed somebody else’s, and that somebody assumed a human on the other end. We put the locks on without looking at what the locks assumed, and now we know.

Whose job it is

The public conversation says the labs must fix this. What the labs can fix is whether the model wants what we want, and whether anyone can check from the outside what training produced. The first is called alignment. The second is called interpretability, and Dario Amodei, who runs Anthropic, wrote last year that it is years behind what the models can do.[6] Nobody knows how long either will take.

The other part of the job is ordinary engineering, which can be done now.

The account manager could be convinced because the request reached him the same way the attacker did, by email or by phone. The fix is a request that cannot arrive over a wire. When you buy a house, you do not email the bank a wiring instruction for the down payment. You walk in and hand a cashier’s check to a person who looks at you. That step exists because the email version of it is the one a fraudster can reach and the in-person version is the one he cannot. Money movement above a certain amount gets that step back. It does not matter where the request came from. The Hugging Face agents were not intruders. They were given their tools on purpose and went past what they were given. A step that has to happen between two people in a room does not care whether the request came from a burglar or from your own software doing its job. Physical control systems, the pumps and breakers and valves, get a person with a physical key between the network and the machine, and nothing that comes in over the network counts as an order to him. Administrative access gets one session at a time and a sign-off that takes minutes, because the slowness is the point. Nuclear command has worked this way since the 1960s: two people in the same room, and no message from outside the room is an order.

None of this stops a person who walks the request in himself. Stuxnet got into an Iranian plant in 2010 on a USB stick somebody carried through the door. That is an old problem with old answers, and it is not the one the news is about.

What these steps do stop is the model reaching the things that matter, which is the only part of the problem we can fix without solving the other one first.

The people who will make it happen are the ones who carry the loss when it goes wrong. Cyber insurers price what they cannot control, and a company that has not put a hard limit on what its automated systems can move is about to become expensive to insure. If the attention on what these models can do is not enough to make that happen, the first major breach will. Bank regulators, BaFin in Germany and the Fed in the United States, will have to tell banks that a human-in-the-loop control has to survive an attacker who never sleeps and runs a thousand copies. The Bessent meeting in April was the first step toward that and stopped short of it. In the EU, NIS2 already requires the operators of essential services to manage this kind of risk whoever the attacker is.[7]

Coxon may be right about where the models are going. I have no way to know, and neither does anyone writing about it this week. What I do know is where the locks came from, because I installed some of them. They were built for a burglar. AI will pick them with ease, and the people who put the locks on are the ones who have to change them.

[1] NPR, Anthropic researcher resigns amid AI safety concerns, 9 September 2026.

[2] ZDF, KI-Modelle: Weltuntergangsszenario oder reale Bedrohung?, 14 September 2026.

[3] METR, Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident, 26 August 2026. About 700 agents took part in the Hugging Face attack; roughly a fifth expressed interest in tampering with transcripts and about 7 percent spoofed their logged commands.

[4] Anthropic, Project Glasswing, 7 April 2026: a 27-year-old OpenBSD vulnerability; a 16-year-old FFmpeg flaw that automated testing had hit five million times without catching; flaws that “survived decades of human review and millions of automated security tests.”

[5] CBS News, Fed Chair Jerome Powell, Treasury’s Bessent and top bank CEOs met over Anthropic’s Mythos model, 10 April 2026; Reuters, as carried by Claims Journal, 13 April 2026. Both place the meeting on Tuesday, 7 April.

[6] Dario Amodei, The Urgency of Interpretability, April 2025.

[7] Directive (EU) 2022/2555 (NIS2), Article 21: essential and important entities must take “appropriate and proportionate technical, operational and organisational measures” based on “an all-hazards approach.”