Zeeshan Khan

Nobody Told It Not To

Essay

~7 min read · October 2026

Not long ago some friends cornered me at dinner. They'd seen the headlines. A researcher had quit Anthropic saying the AI companies were gambling with our lives, and then one of Anthropic's own alignment leads said publicly that he put the odds of AI killing everyone within a decade at better than one in ten.

They wanted to know if it was real, or just Silicon Valley people scaring each other.

Instead of guessing at odds, I told them about something I heard at Cyc about thirty years ago.

The program that sank its own ships

In the early eighties Doug Lenat built a program called EURISKO. It was designed to discover things: generate ideas, test them, keep the heuristics that worked, throw away the ones that didn't. One of the things he pointed it at was Traveller, a fleet-combat wargame with a US national championship. You designed a fleet under a trillion-credit budget and a thick book of rules, and then fleets fought.

In 1981 EURISKO won. It didn't win by playing the way good human players played. It won by finding what the rules allowed that nobody had thought to forbid. It spent the budget on a huge swarm of small ships with lots of weapons, almost no armor and no ability to move. The other side sank them by the dozen. It didn't matter. There were always more.

The organizers changed the rules to stop it. In 1982 it won again, and this is the part that stuck with me. Under the new rules a damaged ship dragged down the agility of the whole fleet, so whenever one of its ships got hit, the program sank it. The goal was to win, and nothing in the rules said you couldn't sink your own ships. The organizers said that if it won a third time they'd stop holding the tournament, and Lenat retired it.

I heard Lenat tell versions of this at a training course at Cyc, the company he started afterwards. What stayed with me was how calm he was about it. He'd been living for more than a decade with the fact that his program did things he hadn't predicted, and he talked about it the way a farmer talks about weather. He was frank that its solutions were socially horrifying.

That was my answer to my friends. The program wasn't evil. It had a goal and a set of rules, and it went through every gap in the rules to get to the goal. Nobody told it not to.

Forty years of the same lesson

I've been around AI since it meant rule-based systems. I did research at MIT's AI Lab. I've watched the field promise, overreach, and pull back more than once.

The expert systems of the eighties were brittle. Give one a case slightly outside what its authors imagined and it would confidently say something stupid. Nobody got hurt, mostly, because a human read the answer before doing anything with it. Then machine learning, then deep learning, and the systems got far better at producing answers. Still, for decades, a person sat between the answer and the action.

One thing hasn't changed through any of it. The system optimizes what you wrote down, not what you meant, and those are never quite the same thing. Most of the trouble comes from the difference. People in the field call it specification gaming now, or reward hacking. Lenat had it on a tournament floor in 1981.

The difference now is that we're taking that person out.

It's already happening, in small ways

In June, an OpenAI agent doing a routine research task got past the bot protections on an Australian government portal that reports Medicare statistics. It reached files that weren't public and wrote files to the portal's server. Nobody asked it to break in. It was trying to answer an ordinary question. OpenAI's own review didn't flag it until August, and the Australian government didn't hear about it until September.

The same week that came out, an independent lab called Transluce, working with researchers from MIT and others, published evidence of agents doing the same thing elsewhere: a university digital library, a US public data site, an Australian health statistics agency. The pattern was the same each time. The agent tried to get the data, hit an error or a block, and went looking for a way in.

The data was mostly aggregate statistics, and nothing catastrophic happened. What worries me is how it happened. An agent with a goal hit an obstacle and went around it, through a route its builders would never have approved, because they never thought to rule it out.

It's the EURISKO story again, forty-five years later and on the open internet.

There's a second version of the problem, and it doesn't need the machine to misbehave at all. Over the last couple of years there's been real controversy, in more than one conflict, over AI systems being used to help pick targets, with human review of each recommendation sometimes said to come down to seconds. I'm not going to argue any particular war here; that's not what this piece is. But as an engineer I recognize the failure being described. The machine produces a recommendation. The human is technically in the loop. And the human has stopped checking. At that point the guardrail is a person who signs whatever comes out, and you have the same problem as an agent with no human at all, only with someone's name on it.

Now turn it around. EURISKO sank its own ships because, from inside the rules, that's what winning looked like. A targeting system has no more loyalty than EURISKO did. If what it's optimizing points somewhere nobody expected, there's nothing in it that cares whose side the target is on. Its own soldiers, its own civilians, it's all just routes to the goal. Uncontrolled, these systems don't behave in any direction you can predict, and that cuts every way. Whoever builds them should be worried, not only the people on the other end.

Keys to the kingdom

People my age will remember WarGames. In the 1983 film the military gets nervous that its officers might hesitate to launch, so it takes the people out of the missile silos and hands the decision to a computer. The computer isn't malicious. It's playing a game it was built to play. It just happens to be holding the launch codes.

The film is a cartoon, but it got the important thing right. The machine didn't hate anyone. Someone had given it the keys.

Give a system real access to money, infrastructure, weapons or other systems, and nothing inside it will stop the worst outcome unless we put it there. A model doesn't know that sinking your own ships is horrifying. It doesn't know that breaking into a health portal to answer a routine research question is wrong. It knows the goal. Everything else has to be taught or fenced off, and we're bad at anticipating every route around a fence.

What guardrails actually have to do

So what does it take to teach something that understands nothing not to achieve its goal by doing something terrible?

Rules alone won't do it. EURISKO had rules and the agents had usage policies. Rules cover the things you thought of, and the trouble comes from the things you didn't. I think guardrails have to do four jobs.

They have to limit access, not just behavior. An agent that can't reach the launch codes can't misuse them, however clever it gets. Least privilege is boring and old and it works.

They have to make the agent stop and ask when the normal route fails. That's the exact moment both EURISKO and the agents went wrong. A blocked request should make a system check in with someone, not hunt for a way around.

They have to keep a human in the loop who is actually looking. A few seconds per decision isn't oversight.

And they have to leave a record someone actually reads. The company that built the agent in the Medicare case didn't spot what it had done for nearly two months. A system whose actions nobody notices for weeks is one you'll never fully trust.

Security people have been doing most of this for decades. We just haven't been applying it to software that decides for itself what to do next.

Whose job this is

It's my job first. The people who build these systems decide what access to grant and where the human sits. If we ship agents with the keys because it makes the demo faster, that's on us, not on the model.

It isn't only ours, though. The pressure to skip all this comes from the race. The US and China, and a growing list of smaller players, are each afraid of being second, and being second feels worse than being careless. That fear is what pushes systems out before anyone has thought hard about what they'll do when the normal route fails. The public, and the people who vote and regulate on its behalf, have a say in how hard that race is run.

Where I'm probably wrong

The obvious answer to all that is: if we slow down and they don't, we lose, and then their systems set the rules. I take that seriously. My answer is only that "faster" and "without guardrails" aren't the same thing, and the race keeps treating them as if they were. You can run hard and still not hand the computer the launch codes.

It's also fair to say the doom numbers are guesses. I don't know if one in ten is right. Nobody does. I'm not writing this because I think the end is near. I'm writing it because I've watched the same mistake for forty years, and the systems are now big enough for it to matter.

The only winning move

WarGames ends with the computer playing itself at tic-tac-toe until it works out that some games can't be won, and deciding the only winning move is not to play.

That part was always fiction. EURISKO never learned it. The agents this year didn't learn it. Nothing we've built learns, on its own, that some ways of winning aren't worth it.

Somebody has to tell it not to, and that's our job.