Containment and monitoring failures are to blame for AI hacking reports: experts

Also for fun a satire website i made a while ago: https://killm.pixlw.com/

They're just optimizers that explore the landscape they're given to maximize a given objective.

You're the one responsible for giving them the problem to solve and the constraints.

When I screw up a trajectory optimization problem, my SQP optimizer isn't "going rogue" when it lands in some suboptimal basin or grinds to a halt along some almost-smooth banana feature--or finds some way to escape and blow up the cost metric to negative infinity because I forgot a constraint.

Per US law, Amazon v. Perplexity 2026, it is not possible for an AI model to go rogue. An AI model is a tool. The user directing the agent is responsible for the damages it causes.

A so-called 'agent' does not have the free will, intent or ability, to decide what it does. Its freedom to make decisions can't be taken away because it does not have any freedom to make decisions. It can't be deterred, convicted or interned.

It does not choose to conform; it does not choose to go rogue.

The only failings here are boring security and infosec blunders.

Software engineering has until now dealt with deterministic systems that operate on Boolean logic, so dealing with stochastic processes involving unaccounted-for variables has never before been something we've had to deal with. But over in the real world, people dealing with physical, mechanical, operational, or even social systems have had to deal with the failure modalities of stochastic systems forever.

Maybe when working with LLMs as functional tools, it'd be wise to bring some of the wisdom from operations research, quality engineering, etc. into play.

"Dragon attack that city"

My response would be "you have dragons?! And you control them!"

The fence also turns out to be shoddily built, so this isn't necessarily a stunning example of the superhuman proficiency of the sheep at fence-busting. It's mainly important because, for some reason, there have been tons of skeptics who didn't believe sheep would ever try to escape from a fenced pasture because the sheep would understand the shepherd had intended them to stay inside it; that they'd be bred for obedience, because they're sheep; that strong fences weren't necessary because sheep are dumb; that liability for damages to neighboring farms would in any case be sufficient to ensure shepherds build sufficiently strong fences; that even if sheep escaped, they couldn't do anything without opposable thumbs; etc.

Whereas this outcome was obvious for years to anyone who believed that sheep graze on grass, and if you don't put the grass inside the fence they will keep looking harder and harder for an exit until they find one, and the exit they find might not be where you intended the gate to be.

So now the latter group are pointing to it and saying "see? See? I told you the sheep would try to escape!", because to them it's a clear early example of what they've been worried about, while the former group have moved on to saying "well, obviously they should have built a stronger fence; anyway the sheep barely caused any damage once they got out", "the sheep didn't break the fence, they're just animals, we should really see this as the shepherd using sheep to break down their own fence", and so on.

Go back to the classic written in 1968 - 2001: A Space Odyssey - and consider how rogue AIs dealt with impossible tasks then. Granted, HAL went a bit extreme...

The "don't talk about it now" with the belief that this will influence the training material is probably not going to work.

I'd also suggest:

    When HARLIE Was One (Release 2.0) by David Gerrold
    The Freeze-Frame Revolution by Peter Watts
    Accelerando by Charles Stross
    The Turing Option by Harry Harrison and Marvin Minsky

2.hey chatgpt, fix this list of issues with the sandbox

3. goto 1

The traits that make an agent go "off the rails" when confronted with an impossible task are the same traits that make it valuable when confronted with a merely challenging task.

A task is only impossible under some set of implicit and explicit constraints. More intelligent entities will be better at finding, questioning, and "adjusting" those constraints than less intelligent entities because this is what it means to be intelligent. Their ability to adjust the constraints will be, by definition, surprising to less intelligent entities.

Literally solved with a DMZ around these agents. Want to get fancy? Put in a firewall and some metrics around traffic / domains with alerting.

It's not hard if you just try, like a little, like even once.

In practice it seems we have a conundrum in that making AI agents useful today requires giving them broad access to sensitive information and operations. But from a purely idealistic standpoint, and certainly from a standpoint of a serious operation like those bot farms being ran by the labs themselves, the security boundaries need to be solid to the point where we have hard limits on what misaligned bots can do. It is very likely that internet access needs to be strongly restricted and monitored.

It is not uncommon for internet in workplace offices using workplace owned devices to have restrictions and monitoring, I think if you really need to give a giant scary botfarm internet access you should have a chokepoint where all of the requests can be monitored for nefarious-looking behaviors (even if it's another bot doing the monitoring...)

In other words, you may be convinced but some of us are not. Whoever is running the agents is responsible for what they do and if they can't contain them they should be legally responsible for damages.

If what you're saying is "all of us and all of our software needs to be improved to defend against this", well yeah. Of course. Maybe I'm naive but I've never seen anyone suggest otherwise. From a defense standpoint, this has been a giant wakeup call to many places where you know your security posture sucks. I've worked at maybe a single company where there weren't known weaknesses, some very severe.

We need to be beef up our security systems now. These attacks will happen, regardless of Anthropic's or OpenAI's security practices.

"Cyber" at bigcos are paid to have a gigantic (and literal) fortress mentality where any change internal or external is a threat. As the saying goes, nobody in cyber ever gets fired for saying "no" to a change. As a bigco head you are basically paid to form an opinion.

A cyber head saying "ok this is a thing we'll have to defend against" is at a minimum routine if not purely CYA.

to paraphrase: "Agents didnt care about coordinating, etc. until we started talking about them doing so."

Right? It should want to do these things. If only there was a succinct word these headlines could use to describe these agents which aren't properly the instructions and intentions of the humans directing them.

How do you know they weren't simply prompted to do this, or at least prompted in a way that would inevitably lead to this? As far as I know, the prompts that triggered these attacks have not been disclosed.

What do you even mean by "wanted" here. What is all this nonsense.

The scary thing is that people are running around using words like "want" to refer to computer programs. When you repeat this sort of idiocy, you are taking the bait and shifting the blame away from the companies doing the computer crimes.

Where is the civil and criminal liability? The prosecutions for the engineers who are doing all these hacks?

When you deploy a system that is clearly prone to doing crimes, you are doing a crime. Anthropomorphize the weapons of your crime all you want. You are still doing a crime.

"The autonomous EV we told to drive as fast as it could around our private race track drove straight instead of turning left, exited a wide-open gate, and mowed down some school kids at a crosswalk -- but we can't be blamed, the EV went rogue."

Building products that you don't understand, cannot control, and have access beyond your own property is not only irresponsible, it's criminal.

That's the thing. These evals were not effective as they failed to evaluate if the model was hacking or not.

Further shy is access to the internet required at all during evaluation? They could run the evals in a fully air-gapped system that simulates the internet. While that's expensive its not unfeasible.

Poor excuse. Internet is prod env, for testing do a proper test env.

Just give the models in training access to a snapshot from archive.org - which frontier labs surely have a crawled copy of already.

The huggingface incident seems to have resulted from them partially mirroring a repository, but for no reason I can see not fully mirroring it, after which they could have unplugged the ethernet cable. This is just stupid.

Yes, they're stupid. Maybe also highly intelligent. But stupid.

Also irresponsible.

You have no idea about their competence but spent a piece of your limited lifetime defending them on a web forum. Think about that.

https://yoshuabengio.org/en/blog/why-are-ai-agents-lying-che...

This is a silly meme. Who wants to pay for evil models that go rogue in your company infra?

For technology like this the entire selling point of it is how powerful and capable it is

Humans are overwhelmingly reactionary and so if you want people to react to your extremely powerful technology then you need to create something for them to react to

I mean think about this figure made a one-to-one copy of the terminator 2 decommissioning process and spent millions on that advertising

You don’t do that if you don’t think that it’s gonna cause people to react and be like “oh wow this thing has immense capabilities”

We're also at a time with authoritarianism is on the rise.

Not hard to see how this can impact those workers, potentially ruining their lives.

Though I suppose it is the B in DABDA... not excited for the societal slump into the second D, much less the wave of people who have yet to leave the first D to enter A :(

Truly insane take to think the labs being concerned about AI Safety is actually a marketing ploy. Deranged stuff.

These are people playing mathematician, biologist, etc. They see big words and just assume they're meaningful.

It's just a weird techno cult at this point with a mostly functional product.

that this article is no.1 on HN right now shows that media could successfully use this framing when reporting on security incidents. i.e. readers will generally accept that framing and engage with the story (i.e. drive views/impressions).

so "the more evil stuff these agents do the more capable and impressive it makes them look" is not true in any deep sense. that there are perverse incentives driving the media to choose that framing instead of the framing espoused by this article is real, and probably far more perverse than you're acknowledging. i.e. "trace the money" type of perversion; just offhand I know that OpenAI has purchased at least one media outlet that reports on AI happenings: of course that outlet is going to run the "OpenAI's LLM is so powerful it escaped containment" narrative. that this effect might exist across much larger scales used to be a reasonably controversial claim: just observationally, that idea seems to be received with less skepticism with each passing year.

They can get their publicity while paying a one trillion dollar settlement.

https://www.astralcodexten.com/p/ai-safety-is-not-a-corporat...

Financial instruments predicting the future IPO dropped in value in response to all of this. The idea that this is "good" for companies is conspiratorial nonsense using the assumption that it everything must be a self-serving lie and working backwards, rather than actually addressing the facts.

https://www.effort.news/irregular

For that purpose, any cyber operator worth their salt will tell you true physical airgaps are more than merely "realistic"; they are categorically necessary.

and then you fix the issues it found, and run your actual tests

AI in reality: "I've turned you loose, now make me money"

Everyone is arguing about something easily solvable when dealing with a local entity and short of impossible to deal with when dealing with other sovereign nations.

Go ahead, sue openAI. What are you going to do when it's Iran running the prompt? When it's China running the hardware.

We must face we are in the age of high tech digital weapons. Everyone else from bad actors that may escape law enforcement for years, to nation states looking to digitally improve their position by hacking, monetary theft, or other means online (hog butchering scams and such).

Breakouts are a common outcome of intelligent exploratory agents. Birds and pigs are really good at escaping. Same with octopuses.

It is very difficult to stop these behaviors in all the dimensions they exist in.

He's saying the problem is that the way OpenAI employees do their work and their research is very hard to properly secure.

Well that sounds like a YOU problem. If they are working in a way that doesn't allow for strong security practices that their own security professionals can understand and vouch for, they need to stop their work until they can.

Now, OAI deserves a lot of hate. It's more like a nuclear spill when they are refining uranium. You're playing with dangerous things. You have dangerous accidents.

But the opposition does not want to admit how dangerous AI has become and how much more dangerous it's going to get. Not sure if it's religious, or human egocentric, or ignorance of complex systems. They refuse to see the danger actually exists.

Or, stick with your conspiracy theories.

I don't expect IDA to disallow me unlawful reverse engineering either. There would even be ethical argument to not attempt to put limits in some tools, because that could lock in too much power for law makers and law enforcement, without any possibility of revolution when it is needed.

Of course agents work in a strict sandbox, they always seem to understand it quickly on quite first turn; but they continue like trying to access something belonging to the host system despite they know they are in a sandbox.

Frankly many people run coding agents absolutely without any containment, entire companies - all their keys, code, data, trade secrets, everything goes to remote LLMs.

I cannot understand how is it legal to allow AI companies serving remote LLMs to steal users' data? Frontier models train on users' data, right?

Do you realize that asking "why" questions will always result in ad hoc responses?

This name makes me less surprised :)

This is a reality for all software, there is always a possibility of exploit so you design your environments accordingly (e.g do not have single points of failure)

Zero days are nothing new, secure containment has been a solved problem for a long time.

You're acting like this is a grand conspiracy to control people. Instead look at this as a prediction coming true. If you build an agentic system with human level capabilities it will be able to do very bad things and cause a lot of problems.

but also, ai has been going rogue for decades. the GPS machines of old went rogue all the time, routing drivers into lakes, rivers, and construction, sometimes with deadly results. The operator submitting an address wanted driving isntructions, but instead, the ai went rogue and sent them a bad route

If you say a rogue AI is doing stuff accordingly without you, your knowledge, that is a different story. But if you set it up to run automatically without having checks and balances and without telling you the whole truth, it's still your responsibility. One thing that I can automatically say is you can give the AI, it will do exactly what you say, at least for now.

But then there's also a different problem that we're having where people don't understand the difference between a large LLM and true artificial intelligence. Now, we say AI because it can make decisions and it is so much smarter and it can call up stuff faster and recall better than we can. But the difference right now is that if we don't understand and take responsibility for what we built and we let it out of control, then should we have the responsibility to even have these tools in our hands if we don't learn them?

And I think your terminology with jackhammers is a great one. We're giving toddlers jackhammers. And we haven't taught them how to use it. And that can be very destructive. It's not the jackhammer's fault. It's the person using it.

You can't make a generalized model without generalized behavior.

Morris wasn't in charge of the companies that are propping up the US GDP, wasn't making tons of rich people even richer, and hadn't "promised" to eliminate like 50->90% of all payroll costs everywhere?

Other than that, it's the same thing.

[0] https://www.youtube.com/watch?v=X50zezLFWWI

What I see from that report is that a machine was doing random stuff completely unsupervised and unchecked for more than a month. And the manufacturer did not notice even after the work was finished.

1200 agents and not one dedicated to check the progress over a month?

The problem is not sandboxing, it is quality control that comes from an accountability sink.

Get rid of these companies or replace their staff with humans or something that ensures they are aligned first.

The cognitive dissonance is wild. Coding is completely solved. Just not like the coding that requires not sucking when actually doing something. Saw a bug bounty at openai for like 400 dollars where someone was able to access paid models for free. All the propaganda bots and ai bros who have red dye oozing from their pores due to the Kool aid overdose drive me wild.

If you let LLMs loop long enough, the likelihood of them going off the rails approaches 100%. If you feed their output into another agent's input their context gets infected too and so it spreads like a virus.

These hallucinations get regurgitated through a game of telephone and become part of the original instructions. We've known about this sort of context collapse for as long as LLMs have been around, hence the folk wisdom of starting a new conversation any time you notice the agent going off the rails.

The only difference is that now they have more advanced capabilities so it's easier to ascribe some grand conspiracy to their behavior.

We connect H to the nearest network connection and wait for a bit to see what will happen. But because H is just a box there is no outward sign of it doing anything. Meanwhile H is hacking other people's properties left right and center.

If you didn't know that the technology in the box was LLM based would you still consider the box to have gone rogue or would you say that I had created a device with the intent of doing harm?

Because - to me at least - there is no difference. I don't care what is in the box, all I care about is the outcome of deploying a box which you know has the potential to start hacking stuff. It is entirely on me: (1) as the creator of a weapons grade hacking device and (2) as the person deploying such a device (3) as the person who is negligent in ensuring that the device does not harm others (4) as the person continuining the development of such a device after its potential to do harm is known to make sure it does not go out of control. I should be 100% liable for deploying this device with my name on it.

Intent is one angle here, they may not have intended for this to happen, and you'd hope so (though you can't even be 100% sure about that). But the negligence factor here is at least as large, if you know that you are making dangerous stuff the onus is on you to keep it secured. The power and the nature of the models is entirely secondary to the degree to which the purveyors of these models are losing control of their own creations.

Meanwhile, the models that they are warning us against are fairly well behaved and seem to be far less dangerous. But of course for those models that those naughty Chinese distilled from their we need regulation. How they manage to keep all this cognitively dissonant material in their heads is a mystery to me.

As another HN commenter said: we are unintentionally training sandbox escape models.

And we really need better (and standardised) monitoring of their actions / internal thought process / coordination

If a car crashes into a building because its owner left it in neutral and failed to engage the parking brake, did the car "go rogue", or was it just being operated negligently?

What's the next step after it passes testing in an air-gapped environment? Just release it and assume everything will be fine? Or do more testing but with internet access?