New evaluations of Claude Mythos Preview, the latest large language model (LLM) from Anthropic, highlight its potential in the realm of automated cyber operations. The UK’s AI Security Institute (AISI) undertook extensive testing to assess whether this advanced model could undertake fully automated cyber attacks.
According to their findings, Claude Mythos exhibits enhanced cybersecurity capabilities compared to previous models. However, it struggles with executing autonomous assaults on robust, well-defended networks.
Understanding Claude Mythos Preview
Claude Mythos Preview was unveiled to the public earlier this month, with Anthropic touting its remarkable proficiency in identifying previously unnoticed bugs and vulnerabilities across a range of platforms, including operating systems, software applications, and cryptographic libraries.
Due to its powerful capabilities, Anthropic has opted not to release the model publicly, citing concerns that malicious actors could exploit it to uncover zero-day vulnerabilities and craft exploits for both new and unpatched issues. Instead, they initiated Project Glasswing, a selective program aimed at providing early access to the model for major technology firms, cybersecurity experts, and financial organizations. Participants include the Linux Foundation and a consortium of 40 organizations dedicated to maintaining critical software infrastructure, all striving to secure essential software before similar AI tools become widely accessible.
Examining Cyber Attack Capabilities and Limitations
The implications of Claude Mythos Preview for cybersecurity are generating significant discussion across various platforms. Insights gained from AISI's tests clarify the challenges cybersecurity defenders may soon encounter. The model excels in solving capture-the-flag (CTF) challenges, a type of task designed to identify and exploit weaknesses in target systems. AISI's researchers reported that Claude Mythos successfully completed expert-level tasks 73% of the time, a notable improvement over previous models.
However, when faced with complex real-world cyber attacks, its effectiveness diminishes. “Real-world cyber-attacks demand the chaining of numerous steps across various hosts and network segments—sustained operations that typically take human experts many hours or even days to accomplish,” noted AISI. To gauge this, they developed a simulation called 'The Last Ones' (TLO), a 32-step corporate network attack that spans initial reconnaissance to complete network takeover, which they estimate would require human intervention over 20 hours.
Claude Mythos Preview became the first model to complete the TLO simulation from beginning to end, achieving success in three out of ten attempts. However, it's important to note that the test environment was considerably less challenging than real-world networks, lacking active defenders, defensive tools, and any repercussions for alerting security systems. “This raises questions about whether Mythos Preview could effectively target well-defended systems,” the researchers cautioned.
Despite these limitations, the model can autonomously manage attacks on small, inadequately protected systems once initial access is secured. This underscores the critical need for fundamental cybersecurity measures, such as regular updates, strong access controls, proper configurations, and thorough logging practices. AISI researchers directed organizations to the UK National Cyber Security Centre's guidelines on utilizing AI to bolster defense strategies.
Recommendations for AI-Assisted Defense
In light of these findings, Anthropic's researchers recommend that cybersecurity defenders leverage available AI models to enhance their defensive strategies. This includes using AI for vulnerability discovery, analyzing cloud environments for misconfigurations, expediting transitions from outdated systems to more secure alternatives, and automating various aspects of incident responses.
The ability of Mythos Preview to autonomously generate n-day exploits indicates that organizations will need to shorten their patch cycles significantly. “Software users and administrators must reduce the time-to-deploy for security updates, tighten patch enforcement windows, enable auto-updates wherever feasible, and treat dependency updates that include CVE fixes as urgent rather than routine maintenance,” Anthropic emphasized.
A recent paper published by the Cloud Security Alliance, informed by cybersecurity experts and community input, offers additional guidance for Chief Information Security Officers (CISOs) on how to adapt organizational security protocols to the evolving threat landscape presented by AI-driven capabilities.
Source: Help Net Security News