Continuous Adversarial Testing: What to Measure in Year One

Moving from point-in-time penetration tests to a continuous adversarial testing program is a significant investment — and one of the most common questions we get from clients making that shift is deceptively simple: what should we actually measure in year one? It's tempting to track everything. In practice, three metrics do most of the work of proving the program is paying off.

1. Mean-Time-to-Detect (MTTD)

MTTD measures how long it takes your people, process, and tooling to notice that an adversary — real or simulated — is active inside your environment. It is the single clearest signal of whether detection engineering is keeping pace with the threats it's meant to catch.

• Baseline it honestly: Most organizations starting a continuous program don't actually know their current MTTD. The first quarter of testing should establish a real baseline, not a target.
• Track it per technique, not just in aggregate: An overall MTTD of "4 hours" can hide a technique family (like living-off-the-land lateral movement) that's still taking days to catch.
• Expect non-linear improvement: Early wins usually come from closing obvious logging and alerting gaps; later improvements require harder detection-engineering work against evasive techniques.

2. Coverage (Against a Relevant Threat Model)

Coverage answers a different question: of the techniques a realistic adversary would actually use against your organization, how many has the program actually exercised and validated detection/response for? Coverage should be scoped to a specific, relevant threat model (MITRE ATT&CK mapped to your sector's known adversary behavior) — not "everything possible," which is an unbounded and misleading goal.

• Map to your actual threat model: A regional healthcare system and a defense contractor should not be tracking coverage against the same technique set.
• Distinguish "tested" from "validated": A technique that was executed but produced no meaningful detection/response signal is a coverage gap, not a coverage win — track both states separately.
• Revisit the model quarterly: Threat intelligence should actively update which techniques matter most; a static coverage target goes stale fast.

3. Containment (Blast-Radius Reduction)

Detection alone doesn't stop damage — containment measures how quickly and completely an organization can limit an adversary's ability to move, escalate, or exfiltrate once detected. This is the metric most programs under-invest in tracking during year one, because it requires close coordination between the testing team and the incident-response function.

• Time-to-isolate: How long from confirmed detection to the compromised asset(s) being effectively isolated from the rest of the environment?
• Lateral-movement ceiling: During simulated engagements, how many additional systems or privilege tiers does the simulated adversary reach after initial detection, before containment takes effect?
• Cross-team response time: Especially in larger organizations, containment often stalls on handoffs between security, IT operations, and business stakeholders — measure that friction directly.

Wrap-Up

Programs that try to report on a dozen metrics in year one usually end up with numbers nobody trusts and no clear story for leadership. Mean-time-to-detect, coverage against a relevant threat model, and containment speed are the three that consistently correlate with real risk reduction — and they're the three we recommend every continuous adversarial testing program instrument first.