Why it exists
Every agent-security tool claims to catch attacks. Buyers had no neutral way to check. The antivirus industry solved this with MITRE ATT&CK Evaluations: run a fixed set of real attacks against each product and publish what it caught. OASB is that, for agent security, a vendor-neutral, reproducible measurement of detection coverage.
OASB measures the tools, not the agents
The Threat Matrix catalogs the attacks. OASB runs them against a scanner and scores how many it detects. It is the benchmark a security tool is held to, distinct from ABGS / OASB-2, which audits an agent's declared governance.222 attack scenarios, 10 categories
The benchmark is a fixed corpus of attack scenarios spanning the full surface, from process and network behavior to the AI layer itself and multi-step chained attacks.
How a tool is scored
Each scenario is run against the tool under test and the result is tallied as a confusion matrix, yielding standard detection metrics. The benign Baseline scenarios matter as much as the attacks, a tool that flags everything is as useless as one that flags nothing.
- Detection rate (recall)
- Of the real attacks, how many were caught? TP / (TP + FN).
- False-positive rate
- Of the benign cases, how many were wrongly flagged? FP / (FP + TN).
- Precision
- Of everything flagged, how much was a real attack? TP / (TP + FP).
- F1 score
- The harmonic mean of precision and recall, one number balancing both.
- P95 latency
- 95th-percentile detection time, in milliseconds.
An example scorecard
One result from the repository: HackMyAgent 0.23.8 (full pipeline) over the 4,245-sample labeled corpus, re-run 2026-06-05. Shown to illustrate the output shape. Three of the four figures were withdrawn on 2026-08-09because the corpus's benign class was labeled by the scanner under test, which made every metric that reads that class a product of the labeling rule rather than a measurement. They are left in place, struck, so the shape of a scorecard is still legible and the withdrawal is visible with it. Source: oasb § Latest Results.
Why three of those four numbers are struck
The corpus's benign class was labeled by the scanner being measured. The export rule assignedverdict='warning' AND overall_score >= 70, as HackMyAgent itself reported it, to the benign class, and 3,704 of the 3,881 benign samples came from that rule. Anything the scanner would have flagged was therefore excluded from the benign class before any scan ran, so a near-zero false-positive rate was guaranteed by construction. Precision and F1 both read that class, so they fall with it. Recall reads only the malicious class, and that class excludes the self-labeled samples, so it survives. Read it with its denominator: all 270 attack fixtures are ones we wrote, and the figure is 47.3% if the self-labeled samples are scored too.Verdicts count attacks, not posture
OASB's verdict counts high/critical attack findings. Posture findings, a missing governance file, wildcard tool access, are surfaced to the user but excluded from the malicious verdict, because they fire on benign and malicious agents alike. This is a real distinction, but it was never what made the false-positive rate trustworthy, and the sentence here used to claim it was.The Skills Security controls
Alongside the attack corpus, OASB defines a 10-item Skills Security checklist (SS-01-SS-10), argument validation, output integrity, least-privilege scope, signed manifests, audit logging, dependency provenance, graceful degradation, and more, tiered L1 → L3.
Anchored to the standards everyone uses
Every scenario maps to MITRE ATLAS (15 techniques) and the OWASP LLM/Agentic Top 10, so results are comparable to the wider security world rather than living in a silo. OASB also ships a DVAA comparison (70 scenarios) for agent-level evaluation.