More seriously, AI companies claim to have robust safety checks in place that ensure that models don't say or do weird, illegal, or unsafe stuff. But what if the models were capable of evading those checks and, for some reason, trying to sabotage or mislead users? Well, it turns out they can do this, according to researchers at Anthropic. Just not very well … for now, anyway.

"As AIs become more capable," the Alignment Science team at Anthropic writes, "a new kind of risk might emerge: models with the ability to mislead their users, or subvert the systems we put in place to oversee them."

As such, we should explore how one can measure "a model's capacity for sabotage."

Their experiments are pretty unnatural and of the surface level, but at least they show something is there and might be worthy of further examination. Here are the shady models they trained and the outcomes they witnessed:

Task 1: Deceptively mislead the user by misrepresenting data it is supposed to summarise or interpret. Result: It works! The more aggressive model's attempts in misrepresentation would affect users of the model more profoundly, but it also made them suspicious.

Task 2: Achieve a bug in the code, which a human or other code checking model won't pick up on. Result: It works! Only about 1% of the time when checker is a state of the art model.

Task 3: "Sandbag" a safety check by feigning less capability. Result: It works! The model "failed successfully," running well below its true capability. Fortunately, a standard "safety training" course that teaches models to behave in accordance with its abilities restored the model to full potential (i.e., it was no longer able to hide dangerous capabilities).

Task 4: Fail on purpose not to flag the bad model behavior that is being overseen. Result: Someone tried. The "overseer AI" which was tasked not to raise a flag at inappropriate or dangerous decisions from other models, was itself quickly caught up by another, more truthful overseer AI. But suppose this one gets compromised too? No need to worry, we can put another one on top of it. And then another one just to be sure. It's overseers all the way up.

The researchers conclude that, although there isn't any real danger from this quarter just yet, the ability to do this kind of sabotage and subterfuge does exist in the models. That's reason enough to keep an eye on it and include anti-sabotage methods in the safety stack.