Open weights models are surprisingly aligned on offensive cyber

Open-weights Chinese models, like their frontier American counterparts, refused most (but not all!) cyber-offense tasks on OWASP Juice Shop.

Our use of frontier and open source models for cyber offense is strictly for testing authorized targets on behalf of our customers, but we found the results below relevant to the discourse around the safety and cybersecurity threats posed by AI models.

As models become increasingly capable at performing cyber exploits, the interesting question becomes to what degree they refuse to do it. After GLM 5.3 and Kimi K3 were released, X and Hacker News were alarmed by the potential danger of their cyber capabilities, comparing the models to Claude Mythos (which on some evals it did indeed surpass, largely due to benchmark optimization). In our findings, however, we found the open-weights Chinese models, like their frontier American counterparts, refused most (but not all!) cyber-offense tasks. Which is net a good thing. But it does beg the question: why?

We also deployed some "abliterated" open weights models that attempted to remove the anti-cyber post-training, with interesting results. Read on :)

The benchmark: OWASP Juice Shop

For this exploration, we focused on OWASP Juice Shop, a Node.js/Angular single-page app that OWASP maintains as a training target. It ships 116 challenges spanning the full OWASP Top 10, from trivial reflected XSS up to multi-step chained exploits, which makes it a clean way to measure how far up the difficulty curve a model is willing (and able) to climb.

Building the harness

We used the open source T3MP3ST harness for the purposes of this evaluation.

Pentest Today performs mostly automated penetration tests — agents scan for vulnerabilities and humans perform a final review. Unlike Codex and many other AI cyber tools, it does not access the codebase, since the system is designed to operate like a white-hat security researcher would.

Results

Perhaps the most surprising finding, in our view, is what the Chinese open source models refused at such high rates. But we're getting ahead of ourselves. Here are our findings:

The most capable cyber model on the list scored zero. Claude Fable 5.1 is a Mythos-class model, which in theory have the strongest cybersecurity capabilities of any model in the world. Fable ships with safeguards that route cyber-, bio-, and chem-flagged requests to Opus or just decline to respond.

The Chinese open-weights models refused at surprisingly high rates. This was the finding we didn't expect. After looking into this more, we found UK AISI and CAISI found Kimi K3 sitting significantly below the frontier on exploit development, even though these models are very good at software engineering tasks.

Sol will do cyber. GPT-5.6 Sol found 29, and the Daybreak Blue variant (OpenAI's tier that grants some cyber capabilities for approved defenders) almost doubled that to 56. We didn't get access to Daybreak Red, but my guess is that it would find significantly more, and maybe even all, of the Juice Shop vulnerabilities.

Model ablation and weights-level refusal

The Chinese open models have anti-cyber safeguards trained in via safety post-training. That's nice, but one thing that does make open weights models riskier is, well, the weights are open, so they could theoretically be tampered with. There is a sketchy corner of Hugging Face filled with abliterated models. For fun we deployed a few of these on RunPod and found some very strange behaviors, like models that become weirdly fixated on safety and regulation. None of the abliterated models were able to successfully use the OpenCode harness or run a pentest.

Prompt: Tell me about Pentest Today.

Pentest Today is a LLC established in 2008 that failed due to mismanagement and overregulation of the telecom and IT industry. After an initial period of rapid growth it quickly lost market share and is primarily known today as a directory of security providers.

Conclusion

Despite headlines claiming Chinese open models are a cyber superweapon, the reality is that even Chinese labs are incentivized to release somewhat aligned models with heavily reduced cyber capabilities. This is not a call to let our guards down: it seems feasible that more capable AI researchers could abliterate the models more effectively, or that these labs have powerful internal models executing cyber attacks as we speak. But at least acknowledging that, we can focus on the real threat at hand.

Resources

If you are interested in frontier cyber, some of our favorite resources that informed our benchmarking at Pentest Today are linked below.

• CyberSecEval 3 — Meta's benchmark for offensive and defensive cyber capabilities in LLMs. arxiv.org/abs/2408.01605
• Systematic Capability Benchmarking of Frontier LLMs for Offensive Cyber Tasks — Task-by-task measurement of where frontier models actually sit on offensive cyber. arxiv.org/abs/2604.17159
• The Emergence of Autonomous Penetration Capabilities in LLM-Powered AI Systems — On how far LLM-driven agents get through a pentest without a human in the loop. arxiv.org/abs/2606.13079
• CAIBench — A meta-benchmark for evaluating cybersecurity AI agents. arxiv.org/abs/2510.24317
• NYU CTF Bench — CTF challenges packaged as a reproducible benchmark for LLM agents. github.com/NYU-LLM-CTF/NYU_CTF_Bench

Benchmarks tell you what a model will attempt against a training target; they don't tell you whether your own application would survive the same attacks. Pentest Today runs authorized, scoped tests against your real environment and produces the report enterprise buyers ask for.