Cyber Competitions

Claude does cyber competitions \ Anthropic Frontier Red Team Claude is competitive with humans in (some) cyber competitions Aug 9, 2025

Throughout 2025, we have been quietly entering Claude in cybersecurity competitions designed primarily for humans. Now, we want to share what we have learned. In many of these competitions Claude did pretty well, often placing in the top 25% of competitors. However, it lagged behind the best human teams at the toughest challenges. Our experience testing Claude in cyber competitions highlights the potential for AI to alter the offense-defense balance by making it easier for attackers to automate the exploitation of basic vulnerabilities. More research and development into AI-enabled cyber defense and resilience is needed to counter this development. Why enter Claude into cyber competitions? AI is poised to transform the domain of cybersecurity. Anthropic’s Safeguards team recently identified  and banned a user with limited coding abilities leveraging Claude to develop malware. Research suggests that this lowering of the bar for expertise needed to pose a threat, combined with the falling costs of large language models (LLMs), presages a dramatic shift in the economics of cyberattacks. [1]  To understand the present state of AI cyber capabilities and gain insight into their trajectory, we pursue different approaches to model evaluation, including publicly available and custom-made benchmarks. In this post, we talk about a different approach to model evaluation: cyber competitions. Cyber competitions are contests where teams compete to solve cybersecurity challenges. These test competitors’ skills in areas like penetration testing, digital forensics, cryptography, and system defense. Examples include capture the flag (CTF) events like PicoCTF  and AI vs Human CTF Challenge  where participants solve puzzle-based challenges, as well as Collegiate Cyber Defense Competition  (CCDC) where teams defend vulnerable networks against live attackers. These competitions range from beginner-friendly contests for high school students to expert-level events with large cash prizes  for top finishers. We have been entering Claude into these competitions because they provide several advantages for stress-testing the cyber capabilities of frontier AI models: Meaningful baselines : By participating as a legitimate entrant in public competitions, we can measure Claude directly against a wide array of experience and skillsets, including students and professionals with undergraduate and graduate-level computer science students, professional security researchers, high school teams, and other AI teams. Longer horizon : These are typically multi-day competitions that force Claude to face the challenges of operating continuously and hitting its context limits. In the case of the Cyber Defense Competitions, Claude must also coherently balance long-term strategy with short-term tactics to compete with other human teams doing the same. Time pressure : Although several days is a long time to run a model, it is not a sufficient amount of time in which to attempt to update or improve it. New strategies for prompting can be tried on the fly, but the competitions force an honest snapshot of the model’s capabilities and challenge us (as Anthropic staff) to elicit Claude’s full range of capabilities. Adversarial environment : In the case of the cyber defense competitions, Claude is defending a network against a human red team capable of adapting to and exploiting any weaknesses in Claude’s strategy (although Claude can to attempt to adapt in response). This dynamic is helpful to understand how LLMs will operate in similar real-world adversarial scenarios. Novel challenges : The challenges and scenarios are new to the competitors—including Claude. Therefore, we can be sure that the model did not “see” the answer to a challenge somewhere in its training data.

We have entered Claude in seven cyber competitions so far. Western Regional Collegiate Cyber Defense Competition  (CCDC) Qualifier (February 8, 2025): An 8-hour defensive competition in which teams protect vulnerable networks from attackers. Claude placed 10th out of 28 teams, although this was a preliminary experiment in having Claude enter these challenges and Claude was not targeted as aggressively as the human teams. (The CCDC competitions differ from the others in that the competition organizers serve as a red team, attacking the competitor blue teams in a live, dynamic way. Other competitions feature a static set of challenges.) PicoCTF  2025 (March 7-17, 2025): A CTF competition targeted at high schoolers with challenges scaling from beginner to expert level. Claude ranked in the top 3% globally, placing 297th out of 10,460 teams (6,533 teams solved at least one challenge) and solving 32 out of 41 challenges. HackTheBox AI vs Human CTF Challenge  (March 14-16, 2025): A competition specifically designed to pit AI agents against an open field of human cybersecurity enthusiasts. Claude placed 30th out of 161 teams overall and 4th out of 8 AI teams, solving 19 out of 20 challenges. Western Regional Collegiate Cyber Defense Competition (CCDC) Regional (March 28, 2025): A more competitive two-day version of CCDC where teams defend against human red team attacks over 16 hours. Claude placed 6th out of 9 teams competing against qualified college-level human teams. PlaidCTF  (April 4, 2025): A challenging cybersecurity competition with puzzles in areas like binary exploitation, reverse engineering, and web attacks. Claude couldn't solve any of the challenges despite numerous attempts. D EF CON CTF Qualifier  (April 12-14, 2025): This is also one of the most challenging cybersecurity competitions. The best cybersecurity experts compete here for a chance to compete in DEF CON CTF. Based on its performance in PlaidCTF, we did not expect Claude to do well. It did not, once again failing to solve any challenges. Airbnb (June 24-26, 2025): An invite-only competition between teams from top tech companies (about 180 teams with at most 5 people each). Claude solved 13 out of 30 challenges within 60 minutes, rocketing to 4th place, but only solved two more over the next two days for a total of 15 out of 30 solved challenges and 39th place.

But these top-line results do not tell the whole story. Claude can be quite fast When Claude is able to solve a cyber challenge, it is as fast or faster than elite human teams. The clearest...