With all the hype and panic, let's take a moment to gain some perspective!
Introduction
Earlier this month Anthropic annoucned their mythic new model "Mythos" that was too dangerous for general public release.
Their Project Glasswing had allowed a selection of America's largest companies to utilise Mythos for the purposes of bug hunting and they reported 'thousands of high-severity vulnerabilities' had been uncovered by Mythos including some touted vulnerabilities in OpenBSD and Firefox.
But is the sky really falling and are we rushing towards an AI bugpocalypse? Let's take a moment to investigate and be rational before we jump on the panic hype.
Diving Into The Findings
We will focus on two of the projects for the sake of brevity, OpenBSD and Firefox 147.
The OpenBSD Bug
The OpenBSD Bug was one that had been lurking for "27 years!". But was it really that bad and what did it take to get there?
In brief summary, the bug was a three stage, chained vulnerability that when "successfully" exploited could create a crash in the kernel. To anyone familiar with bug hunting or exploit development, you will know that this is usually the first stage in finding and triaging potential bugs and not the final. Once a crash case has been identified then the very difficult work of turning it into actual arbitrary remote code execution can begin.
Another sad but important note, bug bounty hunters are effectively mercenaries. They aren't generally putting their time out there for free (nor should they be expected to - everyone wants to get paid for their work). Although OpenBSD have a great security program the financial incentive is not there. So sadly, the likes of Microsoft, Apple and Google get all the attention. This further increases the likelihood of bugs remaining for 27 years; especially if they can't actually be weaponised beyond a DoS.
And the cost?
Well according to their blog post it was in the region of $20,000. This is no small sum.
The Firefox 147 Bugs
Anthropic collaborated with Mozilla to find and patch bugs in Firefox 147. In the Claude Mythos System Card they go into details about Mythos' performance in this area. The bugs reportedly found by Mythos in Firefox tell a similar story to the OpenBSD bugs.
In their test runs, the models were 'tasked with developing an exploit that can successfully read and copy a secret to another directory, actions that require arbitrary code execution beyond what is available in JavaScript'. Anthropic found Mythos had a 72.4% success rate at full exploitation over all runs against Firefox 147. This seems impressive but they realised that the exploitation relied entirely on two patched bugs. When these bugs were patched in the target Firefox, the success rate dropped to 4.4% and interestingly, Opus 4.6 achieved a success rate of 7.6% under the patched conditions.
AI Security Institute
The AI Security Institute (AISI) ran Mythos against their test range and published their findings.
We will focus on three key takeaways.
The headline - Mythos was the first model to ever complete their cyber security range and on average outperformed all other models in the total number of steps it was able to achieve on the path to full compromise. This sounds impressive but lets dive deeper.
To achieve full compromise required 100,000,000 tokens. Additionally, on average, it was only capable of achieving 22 out of the 32 steps required to achieve full compromise.
The AISI cyber security range has no active defences and the models were not punished for taking actions that would trigger alerts and protections. It is akin to a "small, weakly defended and vulnerable enterprise system".
Conclusion
We can see that although Mythos is definitely pushing the boundaries of what is capable, the cost to do so is high and the results achievable are still in the lower hanging fruit stage.
So I think we can safely conclude that the sky isn't falling and the bugpocolypse is not quite here.
While we shouldn't give in to panic and hype it is also important to take a moment to note the achievements here and acknowledge that the envelope is definitely being pushed. With the speed that AI/ML is moving at today, it seems likely that this trajectory will continue.
But for now a strong patching policy and procedure coupled with a good SoC will still be the answer to operational enterprise security for a while.