Skip to content
SELECTED WORKCASE_03 // PROJECT AEGIS

Train an AI defender
against an AI attacker.

Project Aegis was a research lab with a single goal: stop ransomware in hospitals. It pitted an offensive AI against a defensive one in an isolated arena, fine-tuned the defender on what it learned, then tested it against real attacks on a live decoy. The most valuable thing it produced was not a model. It was an honest measure of what AI detection can and cannot do.

PROJECT
Project Aegis, AI ransomware defence for hospitals
DOMAIN
AI security · model training · red vs blue
TEAM
John Hess
STATUS
Research complete, paused in 2026
01 // THE PROBLEM

Hospitals drown in alerts and still miss the attack.

Ransomware against hospitals delays care. The attack follows a familiar path: get in, harvest credentials, move sideways over file shares and remote desktop, stage, then encrypt everything at once. Each step leaves traces on the network. Few hospitals have the staff to watch for them.

Most detection tools answer with volume. Small teams end up ignoring a console that never stops shouting. The plan was an appliance built for that reality: few alerts, the right ones, explained in plain English, and offered to hospitals at the cost of the hardware.

A detector that cries wolf all day is a detector nobody is watching.

02 // THE ARENA

Red attacks. Blue learns. A judge keeps score.

RED ATTACKTELEMETRYBLUE VERDICTJUDGE

R_01

Red

A fixed offensive model running curated attack playbooks shaped like the ransomware kill chain: recon, credential abuse, lateral movement, staging. It never learns in the loop, on purpose, so the defender trains against a stable, diverse opponent.

R_02

The arena

An isolated network with a deliberately vulnerable target. Every round records what Red did, what the wire saw, what Blue said, and the score. Nothing in it can reach the real network.

R_03

Blue

The defender being trained: an open-weight model that reads network telemetry and decides whether to raise an alarm, and why.

R_04

The judge

A frontier model labels every round against a fixed menu of 13 attack techniques, calibrated against human-labelled gold rounds and capped by a spend kill-switch.

03 // TRAINING AND WHAT IT TAUGHT US

Every result was checked. Some of them broke.

F_01

Training closed a blind spot

Fine-tuning a 70-billion-parameter model on arena rounds took credential-abuse detection from 0% to 71%. A second round cut false alarms on those rounds from 47% to 6%.

F_02

Then real traffic humbled it

On real telemetry, every single-model defender alarmed on about 99% of everything, at roughly 4% precision. Excellent recall, undeployable noise.

F_03

The judge was wrong too

A human audit of the judge’s positive labels found all 24 were benign. The labeller was inventing attacks, which had quietly poisoned part of the evaluation set. Negatives were reliable; positives were not.

F_04

The fix: two tiers

A deterministic rule engine answers in under a millisecond and clears the obvious, then a retrieval-augmented model reviews what is left, with threat intelligence and no fine-tune. Together they stayed quiet on 94.3% of benign rounds, against 0.3% for the noisiest fine-tuned defender.

04 // THE RED TEAM

Stop asking models. Attack it and count.

Labels from another model had already proven unreliable, so recall was measured the hard way. A dedicated attack box ran scripted techniques against a live, internet-facing decoy workstation, and logged each one: what ran, from where, to where, and exactly when. The defender’s alarms were scored against that log. No model was asked to grade anything.

  1. 34 / 35 CAUGHT Real attacks captured off the wire: scanning, brute force, exploitation, lateral movement, command-and-control check-ins.
  2. HONEST CAVEAT That 97% was the high-recall single model, the noisy one. The quiet two-tier stack caught 31 of 35 on its own, missing the low-and-slow beacons. Its full recall in a production layout was never proven.
  3. OUT OF SCOPE Attacks that never touch the network, such as credential dumping on the host, are invisible to a network sensor. We never claimed them.

Detection pipelines fail silently. A filter scoped one subnet too wide once dropped every test attack while showing clean, empty output. Every layer now gets a test that proves it can still see.

05 // THE PRINCIPLES

Built for a network that is already hostile.

P_01

Security first

Never trade security posture for speed or convenience. It is a defence appliance in a hostile network, so security is the foundation, not a retrofit.

P_02

First, do no harm

Never leave a hospital network less safe than it would be without us. Active probing is opt-in, throttled, audited and bounded to declared subnets.

P_03

Default deny

Anything not explicitly enabled is off. No telemetry, no phone-home, no outbound call except to endpoints the operator configured.

P_04

The attack side stays private

The defender may be published once it is vetted. The offensive corpus never will be: releasing it would make hospitals less safe.

P_05

Ground truth, not opinion

Recall is measured against attacks we launched ourselves and logged second by second, not against another model’s guess.

P_06

Ownership before shape

Ask who traffic is from and to before what it looks like. The same blind spot caused false alarms in four separate layers.

06 // BY THE NUMBERS
97%per-technique recall on real attacks captured off the wire (34 of 35)
94.3%of benign rounds kept quiet by the two-tier stack
0 → 71%credential-abuse detection after fine-tuning
610rounds in the specificity test set, plus a human-checked gold set
07 // ENGINEERING

A 70B model, fine-tuned on one workstation.

All training ran on owned hardware: QLoRA adapters on a 70-billion-parameter open model, on a single 96 GB workstation GPU. Adapters were converted and served locally for side-by-side testing, so every defender version stayed available for comparison. One candidate architecture was dropped when its experts would not fit the memory budget. The 70B dense model replaced it and proved the better call.

The appliance side was held to the same bar as the models: hash-pinned dependencies, a lockfile guard in CI, and a control-by-control threat-model audit, validated end to end across 161 automated tests.

  • Llama 3.3 70B
  • Foundation-Sec-8B
  • QLoRA · PEFT
  • PyTorch
  • vLLM
  • llama.cpp
  • Ollama
  • Suricata
  • Kali Linux
  • SQLite · sqlite-vec
  • fastembed
  • MITRE ATT&CK
  • Claude (judge)
WHAT IT MEANS FOR YOU

Prove your AI security tools before you trust them.

We can red-team the detection you already run, measure its recall against real attacks instead of a vendor’s claims, and tell you plainly where it is blind.

START A PROJECT