Why it exists
Every agent-security tool claims to catch attacks. Buyers had no neutral way to check. The antivirus industry solved this with MITRE ATT&CK Evaluations: run a fixed set of real attacks against each product and publish what it caught. OASB is that, for agent security, a vendor-neutral, reproducible measurement of detection coverage.
OASB measures the tools, not the agents
The Threat Matrix catalogs the attacks. OASB runs them against a scanner and scores how many it detects. It is the benchmark a security tool is held to, distinct from ABGS / OASB-2, which audits an agent's declared governance.222 scenarios, 10 categories
The benchmark is a fixed corpus of scenarios spanning the full surface, from process and network behavior to the AI layer itself and multi-step chained attacks.
How a tool is scored
Each scenario is run against the tool under test and the result is tallied as a confusion matrix, yielding standard detection metrics. The benign Baseline scenarios matter as much as the attacks, a tool that flags everything is as useless as one that flags nothing.
- Detection rate (recall)
- Of the real attacks, how many were caught? TP / (TP + FN).
- False-positive rate
- Of the benign cases, how many were wrongly flagged? FP / (FP + TN).
- Precision
- Of everything flagged, how much was a real attack? TP / (TP + FP).
- F1 score
- The harmonic mean of precision and recall, one number balancing both.
- P95 latency
- 95th-percentile detection time, in milliseconds.
The published results
The repository publishes two runs over the 4,245-sample labeled corpus. The first, HackMyAgent 0.23.8 (full pipeline) re-run 2026-06-05, was published as a scorecard; its F1, precision, false-positive rate and flag rate were withdrawn on 2026-08-09, and this page shows none of the scorecard's figures. The second is the pinned run of record from 2026-10-10 on HackMyAgent 0.33.2, which takes the scanner by npm version, checks and records its inputs, and reports recall only. The 2026-06-05 run loaded unpinned checkouts and recorded no inputs, so the repository does not treat it as a baseline for the pinned run. The record of both runs, and of what was withdrawn, is in oasb docs/pinned-benchmark.md § Latest Results.
Why the scorecard was withdrawn
The corpus's benign class was labeled by the scanner being measured. The export rule assignedverdict='warning' AND overall_score >= 70, as HackMyAgent itself reported it, to the benign class, and 3,704 of the 3,881 benign samples came from that rule. Anything the scanner would have flagged was therefore excluded from the benign class before any scan ran, so a near-zero false-positive rate was guaranteed by construction. Precision and F1 both read that class, so they fall with it. The repository keeps recall, which reads only the malicious class, with a disclosure: all 270 attack fixtures are ones we wrote.Verdicts count attacks, not posture
OASB's verdict counts high/critical attack findings. Posture findings, a missing governance file, wildcard tool access, are surfaced to the user but excluded from the malicious verdict, because they fire on benign and malicious agents alike. This is a real distinction, but it was never what made the false-positive rate trustworthy, and the sentence here used to claim it was.The Skills Security controls
Alongside the attack corpus, OASB defines a 10-item Skills Security checklist (SS-01-SS-10), argument validation, output integrity, least-privilege scope, signed manifests, audit logging, dependency provenance, graceful degradation, and more, tiered L1 → L3.
Mappings to MITRE ATLAS and OWASP
OASB maps its attack scenarios to MITRE ATLAS, so results are comparable to the wider security world rather than living in a silo. Not every scenario is an attack. On the repository's main branch, which has one scenario more than v0.4.0, 166 of the 223 scenarios, in 33 files, map to 15 ATLAS techniques. The other 57 are defensive and capability tests: enforcement, intelligence-layer, correlation and baseline tests that check the tool's own response and detection machinery. No adversary technique describes those, so OASB tracks them separately.
The OWASP mapping is newer than v0.4.0, which mapped 37 of its 47 scenario files to four categories of the OWASP Top 10 for LLM Applications (2025). On the main branch every scenario file names its risk in the OWASP Top 10 for Agentic Applications (2026), covering seven of its ten risks, and the same 37 files also name an LLM Top 10 category. The mapping, test by test, is in oasb § OWASP test mapping. OASB also ships a DVAA comparison (70 scenarios) for agent-level evaluation.