Blog / AI for cyber offense and defense
AI in Cybersecurity: How Attackers and Defenders Use LLMs
FrostWolf Research · · 5 min read
AI in cybersecurity now runs in both directions. Attackers use large language models to research targets, write convincing lures, debug tooling and, in documented cases, run parts of an intrusion with little human input. Defenders use the same models to find vulnerabilities before release, triage alerts, analyze malware and summarize threat intelligence. The evidence so far says AI has mostly made existing attacks faster and cheaper rather than inventing new ones, while giving defenders real gains in vulnerability discovery and analyst throughput. The newest shift, agents that act rather than advise, changes the picture for both sides.
This guide separates what is documented from what is speculation.
What the evidence says about attackers
The best public evidence about AI misuse comes from the AI providers themselves, who can see how their models are used and publish periodic threat reports.
- Google's Threat Intelligence Group reported in January 2025 on adversarial misuse of Gemini, finding that state-backed groups used it mainly for research, content generation, translation and coding help, and that it gave them productivity gains rather than novel capabilities.
- OpenAI publishes recurring reports on disrupting malicious uses of AI, describing influence operations, scams and cyber actors using models for reconnaissance, scripting and debugging.
- Anthropic's August 2025 threat report described a data-extortion operation that used Claude Code to automate reconnaissance, credential harvesting and network penetration across many organizations, and in November 2025 it reported disrupting an espionage campaign in which, by Anthropic's assessment, AI agents performed most of the tactical work with humans directing at a few decision points.
The pattern across reports is consistent. AI lowers the skill and time an attack needs, scales social engineering, and increasingly lets one operator run work that used to need a team. It has not yet produced a class of attack that defenders had never seen.
Where AI changes attacks most
| Activity | What AI changes | Evidence level |
|---|---|---|
| Phishing and social engineering | Fluent, personalized lures in any language, at scale | Widely reported |
| Reconnaissance and OSINT | Faster target research and summarization | Widely reported |
| Tool and script development | Debugging and writing attack tooling | Reported by providers |
| Intrusion operations | Agents executing multi-step operations | Reported in specific cases |
| Novel malware | Malware that rewrites itself with a model | Early, limited cases |
AI-generated phishing and AI and malware look at the last two rows in detail.
Uncensored models remove a speed bump
Provider safeguards and monitoring are what make those threat reports possible. Open-weight models with refusals removed, such as abliterated models, run locally with no monitoring at all. Their capabilities are below the frontier, but they remove refusal and visibility as obstacles for attackers willing to run their own models.
What the evidence says about defenders
Vulnerability discovery
This is where AI's defensive value is best documented.
- Google's Project Zero and DeepMind described Big Sleep, an LLM agent that found an exploitable stack buffer underflow in SQLite before it shipped in a release.
- Google's OSS-Fuzz team used LLMs to write fuzzing harnesses, increasing coverage across open-source projects and finding new bugs, including one in OpenSSL.
- DARPA's AI Cyber Challenge ran a two-year competition for autonomous systems that find and patch vulnerabilities in open-source software, with finalists' systems released as open source after the 2025 final.
The common thread is verification. The systems that work do not just ask a model whether code is vulnerable; they let it form hypotheses and then prove them by compiling and running the target. AI-driven vulnerability discovery covers the techniques.
Security operations
In the SOC, LLMs are used to summarize alerts, enrich them with context, draft queries and detection rules, explain unfamiliar logs and write incident reports. Major security vendors ship assistants for these tasks. The gains are real for repetitive work. The risks are also real: confident wrong answers, and assistants that themselves read untrusted data such as phishing emails and attacker-controlled logs, which makes them targets for prompt injection.
Threat intelligence and malware analysis
Models are good at reading and condensing large volumes of reports, mapping behavior to MITRE ATT&CK techniques and explaining decompiled code. Benchmarks such as CTIBench and CyberMetric measure this kind of knowledge. The best LLMs for cybersecurity compares the models and benchmarks.
Can LLM agents hack on their own?
Research results are mixed and moving quickly. Fang et al. reported in 2024 that a GPT-4 agent could exploit a majority of a set of known one-day vulnerabilities when given the CVE description, and far fewer without it. Capture-the-flag benchmarks show steady improvement across model generations, and frontier labs now report cyber evaluations in their system cards because capabilities have reached levels that warrant tracking. Meta's CyberSecEval suite is one of several public benchmarks for measuring both offensive capability and insecure code generation.
The honest summary: agents can complete well-scoped offensive tasks and are improving, and real campaigns have used them. They remain unreliable on long, novel engagements without human direction.
The risk most organizations underweight
Discussion of AI in cybersecurity often focuses on AI-powered attackers. For most organizations the larger, nearer risk is their own AI:
- Agents with real access that read untrusted content and can be steered by prompt injection.
- Sensitive data flowing into prompts, logs and third-party models.
- Unvetted AI tools, plugins and coding agents adopted by staff without review.
These exposures are fully under your control, and the controls are known. AI agent security and the OWASP Top 10 for LLM applications are the places to start.
What to do next
- Secure your own AI first. Inventory agents and AI tools, apply least privilege, and put prompt-injection and data-leakage controls in front of them.
- Raise the floor against AI-scaled social engineering. Phishing-resistant authentication, out-of-band verification for payments and credential resets, and training built on realistic, fluent lures.
- Use AI where it is proven. Vulnerability discovery with verification, alert triage, enrichment and report drafting, with humans checking the output.
- Test with AI. Red team your systems, including your AI systems, using the same tools attackers use. AI red teaming is a practical guide.
FrostWolf builds for both sides of this: Guard protects AI agents from injection and unsafe tool calls, and Hunter applies AI to finding and fixing vulnerabilities, verifying each finding by running the software.
Frequently asked questions
How are attackers using AI today?
Threat reports from AI providers describe attackers using models mainly to speed up existing work: researching targets, writing phishing lures in fluent language, troubleshooting code and scripting. Providers have also reported cases of agents carrying out large parts of an intrusion with limited human direction, which marks a shift from AI as an assistant to AI as an operator.
Can AI find zero-day vulnerabilities?
Yes, in documented cases. Google's Big Sleep agent found an exploitable memory-safety bug in SQLite before release, and systems built for DARPA's AI Cyber Challenge found and patched real vulnerabilities in open-source projects. These systems work best when the model's hypotheses are verified by actually running the code.
Will AI replace security analysts?
Not on current evidence. AI is taking over repetitive triage, enrichment and drafting, which changes what analysts spend time on. Judgment, accountability and handling novel incidents remain human work, and AI output still needs verification.
What is the biggest AI-related risk for most organizations?
For most, it is their own AI applications and agents rather than AI-powered attackers: prompt injection against agents with real access, sensitive data sent to models, and unvetted AI tools adopted by staff. These are exposures you control directly.