AI on Both Sides of the Fight

Machine learning earned its place in the SOC (Security Operations Center) by doing what humans can't: watching everything, all the time, without blinking. A few pillars do most of the work.

Anomaly detection is usually unsupervised: the model learns a baseline of "normal" — login times, data volumes, process trees — and flags deviations, no labeled attack examples required. User and Entity Behavior Analytics (UEBA) applies this per-account, so a finance user suddenly touching source-code repos at 3 a.m. lights up. Malware classification tends to be supervised: models train on millions of labeled good/bad files, learning from byte sequences, API call patterns, and structural features to catch never-before-seen samples that signatures would miss.

Then there's the grind. SOAR (Security Orchestration, Automation, and Response) and ML-driven triage rank and enrich the flood of alerts so analysts spend attention where it matters, auto-closing the obvious noise and auto-gathering context on the rest. In a world where a mid-size enterprise generates billions of security events a day, this is not a luxury — it's the only way humans stay in the loop at all.

The same capabilities cut the other way, and adversaries adopted them fast. AI lowers the cost and raises the polish of nearly every offensive step:

• Phishing at scale — LLMs write fluent, personalized lures in any language, erasing the broken-grammar tells defenders trained users to spot. Spear-phishing that once took an hour of research per target now runs in seconds against thousands.
• Vulnerability discovery — AI-assisted fuzzing and code analysis surface bugs faster, and models can help translate a vulnerability into working exploit logic.
• Malware & evasion — generative tools produce polymorphic variants and obfuscation on demand, mutating faster than signatures can be written.
• Deepfake social engineering — cloned voices and faces turn "verify by calling them back" into a trap (the focus of the next module).

Criminal markets responded with purpose-built offerings. In mid-2023, researchers documented tools marketed as WormGPT and FraudGPT — LLMs stripped of safety guardrails and sold on underground forums specifically to craft business-email-compromise lures and malicious code. The barrier to competent cybercrime dropped hard.

Put both toolboxes on the table and you get a feedback loop. Defenders train a classifier; attackers probe it until they find inputs that slip past, then package that evasion. Defenders retrain on the new samples; attackers adapt again. This is a genuine adversarial arms race, and it never reaches a steady state.

The dynamic is asymmetric in the attacker's favor in one crucial way: the defender has to be right everywhere, while the attacker needs one working path. AI amplifies both the defender's coverage and the attacker's iteration speed — but it amplifies iteration more cheaply. An attacker can generate ten thousand phishing variants and A/B test which evades filters; the model that filters them must generalize to lures it has never seen.

Automation favors whoever can tolerate more failed attempts. For attackers, that number is nearly unlimited.

There's also a poisoning angle to the race: if an attacker can influence the data a defensive model learns from — feeding crafted "benign" samples into a feed the vendor scrapes — they can bend the model's future behavior. Which is exactly the subject of the next module: attacking the AI itself.

The most important thing a specialist understands about defensive AI is why it can't run unattended. Meet the base-rate fallacy. Imagine a detector that's 99% accurate on 10 million events a day, where only 0.1% are truly malicious. Even that excellent model produces a flood of false alarms — tens of thousands of false positives can swamp the few thousand real hits, and analysts drown. High accuracy on a rare event still yields low precision. This is not a hypothetical; it's the daily reality of alert fatigue.

Models also drift: the world changes, "normal" shifts, and a model trained last quarter quietly rots. They can be brittle against inputs their training never covered, and their reasoning is often opaque, which matters when an analyst must justify blocking a customer.