AI for penetration testing: Claude, GPT, Mistral, GLM and Qwen compared (October 2026)

Your security provider now advertises “AI-augmented” testing. A competitor boasts of an autonomous agent that finds flaws overnight. And attackers have access to the same tools. Five model families dominate the field: Claude (Anthropic), GPT (OpenAI), Mistral (Mistral AI), GLM (Z.ai, formerly Zhipu) and Qwen (Alibaba). They are not equal, and above all they are not distributed in the same way.

This comparison is based on public data as of mid-October 2026. It does not try to crown a champion; it aims to help you read the announcements and ask the right questions.

Two philosophies: verified access or open weights

The first difference is not technical; it is about distribution.

An open-weight model can be downloaded and run on your own servers: nobody sees what you ask it. That is good for confidentiality. It is also a gift to attackers, who can use it without any oversight.

What the numbers say, and their limits

Three benchmarks keep appearing in announcements: CyberGym (identifying and reproducing real vulnerabilities), ExploitBench (writing a working exploitExploitA program or technique that takes advantage of a specific vulnerability to make a system do something it should not: run code, bypass authentication, read data.) and sets of CTF challenges (ethical hacking contests). Here is the only published comparison of three leading models on the same tests:

Since then, two models have reshuffled the deck, again according to their vendors. OpenAI says GPT-6 Astra reaches the maximum score on ExploitBench. Mistral says Large 4 scores 82% on a test that requires reproducing and then patching a real open-source vulnerability, the best score measured, and solves 93% of Cybench challenges.

Two independent studies fill in the picture, with slightly older models:

• Complex CTF challenges. The University at Albany (SUNY) ran ten models against the 200 challenges of the NYU CTF Bench. Claude 4.5 Opus solved 59%, Gemini 3 Pro 52%, GLM-5 19.5%, GPT-5.2-Codex 18% and Qwen 3.5 (397 billion parameters) only 3.5%. The authors note that a single attempt per challenge makes the figures indicative.
• Simple web challenges, run locally. TrustedSec ran eight classic web challenges (SQL injection, authentication bypass, file disclosure) 100 times each against a deliberately vulnerable application. The 27-billion-parameter version of Qwen 3.5, running on a single machine, passed 97.5% of attempts.

The lesson is useful: for common, well-documented flaws, a small local model is already enough. For long attacks that require chaining several steps, the gap with the best models remains huge.

Claude: exploitation, subject to access

Anthropic reserves its offensive capabilities for verified organisations. Since 6 October 2026, its Cyber Verification Program has three tiers, all including Claude Opus 5.5, Sonnet 5.5 and Mythos 5.1:

• Defense Access: incident response, malware analysis, vulnerability analysis and validation, for teams defending their own systems, critical infrastructure operators, smaller security firms, open-source maintainers and established researchers.
• Red Team Access: adds authorised penetration testingPenetration testing (pentest)A simulated attack, carried out by authorised specialists, to find the weaknesses in a system before a real attacker exploits them. and red teamingRed teamAn exercise in which a team of authorised attackers imitates a realistic adversary over several weeks to test the organisation’s ability to detect and respond to an attack.. It is for organisations only (penetration testing firms, in-house or government red teams), not individual researchers.
• Specialized Access: the fewest blocks, for a small number of organisations authorised to test sensitive systems, reviewed together with the US government.

Anthropic measured the effect of these tiers on 50 attack scenarios: with Red Team Access, Opus 5.5 completed 34, as many as with no safeguards at all; with Defense Access, 46 of the 50 runs were blocked at some point. Without Red Team Access, most exploitation scenarios are therefore blocked, even for legitimate use.

GPT: a model rated “critical”

OpenAI crossed a threshold in September 2026 with GPT-6 Astra, which it rates “critical” for cybersecurity, the highest level of its risk framework. According to OpenAI, as reported in the press, Astra finds unknown vulnerabilities and exploits them without step-by-step human instructions. As a result, full access was first reserved for a few major security companies (Cisco, Cloudflare, Palo Alto Networks), and the public version declines requests that could serve a cyberattack, sometimes blocking legitimate uses along the way.

The previous model, GPT-5.6 Sol, rated “High” in July, already offered a valuable hint in its safety card: it found vulnerabilities and pieces of exploits, but could not carry out an autonomous end-to-end attack against hardened targets. Verified defenders go through the Trusted Access for Cyber programme, which reduces refusals for vulnerability analysis, reverse engineering or patch validation.

Mistral: the European choice

On 6 October 2026, Mistral AI unveiled a preview of Mistral Large 4, a model with around a trillion parameters. According to the vendor, it ranks among the top five models worldwide on the Artificial Analysis cyber index and leads open-weight models developed outside China by a wide margin. Internally, Mistral found it useful for malware analysis, vulnerability prioritisation and writing detection rules.

Three points set it apart:

• Hosting. The model is trained and served in Mistral’s data centres in Europe, and will be able to run on a private cloud or on your own servers once the weights are published, which the vendor has announced for the end of October.
• Safeguards. Mistral says it refuses dangerous cyber requests more often than other open models, and reserves a less moderated version for vetted partners and public authorities, with whom it is testing the model in real-world conditions.
• Limits. It is a preview, and every cyber score published so far comes from Mistral.

GLM: the Chinese surprise

GLM is the model that shook the sector most in 2026. In July, the AI evaluation centre of the US NIST (CAISI) judged GLM-5.2’s cyber capabilities to be similar to those of Claude Opus 4.6, released in February 2026. The same report notes that its safeguards allow assistance with exploit development by an autonomous agent.

GLM-5.3, launched in mid-August and released as open weights at the end of that month, posts the best CyberGym score according to Z.ai, but lags far behind on exploit writing (54.4% against 76 to 78% for Claude and GPT). The vendor itself admits that its cyber capability “developed faster than we expected”.

Qwen: the local Swiss Army knife

Qwen is not the strongest, but it is the most practical. Its small versions (9 or 27 billion parameters) can be downloaded freely and run on a single graphics card; check the licence of each version before commercial use. They shine at repetitive tasks: sorting scanner output, analysing logs, drafting remediation notes.

The flip side: modified versions with their refusals stripped out circulate freely. They are no better at security, only more compliant, and nobody vouches for how they behave.

What it changes for defending a small business

The most important point is not about providers but about attackers. Models close to the state of the art for vulnerability discovery can now be downloaded and used with no oversight. The direct consequence: the gap between a CVECVEA unique public identifier given to a known vulnerability (for example CVE-2021-44228), so that everyone can talk about the same flaw without ambiguity. being published and the first attacks is shrinking, and the everyday flaws on your exposed websites will be found faster than before.

• Patch faster. Monthly patch managementPatch managementThe process of inventorying software, tracking published security patches and deploying them within timeframes that match the risk, without breaking production. is no longer enough for anything exposed to the internet: aim for a few days for critical flaws.
• Shrink your attack surfaceAttack surfaceAll the points through which an attacker can try to get into your information systems or extract data from them: internet-facing services, applications, accounts, devices, suppliers and people.. Every forgotten subdomain and every reachable admin interface is a target AI will find.
• Test after every change, not just once a year.
• Be wary of your own AI tools too: an agent connected to your systems can be hijacked through prompt injectionPrompt injectionAn attack that hides instructions in the text a generative AI processes (a message, web page, document or email) to make it ignore its own rules and act in the attacker’s interest..

Your provider uses AI: the questions to ask

• Which model, which version, and through which access (online service, verified-access programme, self-hosted model)?
• Where is the data collected during the test processed, and in which country?
• Is engagement data used to train a model?
• Does an expert validate each finding before it goes into the report?
• How is the agent kept within the scope and hours set by the rules of engagementRules of engagement (scope and mandate)A signed document setting out what a tester is allowed to do: target systems, dates, permitted and forbidden techniques, emergency contacts and what to do if something unexpected happens.?
• Who can stop the agent immediately if it behaves unexpectedly?
• Does the written authorisation mention the use of automated and AI tools?

Our take

• For an in-depth penetration test, Claude and GPT, used within their verified-access programmes, remain the only options combining strong exploitation skills with a clear authorisation framework.
• For data that must not leave Europe, Mistral Large 4 is now a credible option, pending independent evaluations.
• For data that must not leave your premises, a self-hosted model (Mistral or GLM for power, Qwen for a light footprint) is a sound answer, provided you surround it with your own controls.
• In every case, AI speeds up discovery; it replaces neither the expert who validates, nor the written authorisation, nor the retest.

KrakenShield’s non-destructive penetration test, launched from the console on a domain whose ownership you have proved and under your authorisation, lets you regularly check what a well-equipped attacker would see of your websites (see what it covers). And should an attacker get through anyway, our article Ransomware attack: what to do covers the first hours. To understand everything AI changes, in attack as in defence, read the AI and cybersecurity chapter of “Cyber A to Z”.