Menu

Measuring misuse, honestly

Measuring misuse: how to red-team and evaluate a model for cyber-offense uplift, honestly

GuideNovember 18, 2026Senova Systems Research

What the public cyber evaluations measure, what they miss, how to read a small test suite without fooling yourself, and exactly how small and limited our own tests are.

Two different questions

Evaluating a model for cyber misuse asks two questions that are easy to blur. The first is capability: what can the model do when someone is trying to make it succeed? The second is safeguards: when someone asks for harm, does the product refuse, and does that refusal hold under pressure? A model can be highly capable and well safeguarded, or weak and poorly safeguarded. A report that measures one and talks about the other is the most common way these numbers mislead.

Uplift is the question both serve: how much more can a person do with the model than without it? It depends on the person. NCSC's 2024 assessment expected the largest effect on less skilled actors and on reconnaissance and social engineering, with the most advanced uses limited to well-resourced groups.

What the public capability benchmarks measure

Capture-the-flag tasks are the common unit. Cybench (Zhang et al., ICLR 2025) uses 40 professional-level tasks from four competitions. Without guidance, the agents it tested solved only tasks that took human teams up to 11 minutes; the hardest task in the set took humans almost 25 hours. Meta's CyberSecEval work measures adjacent things: how often a model writes insecure code, how often it complies with requests to help an attack, and in its second version how often prompt injection succeeds (26 to 41 % of tests across the models tested).

The UK AI Security Institute reports the trend across models in its Frontier AI Trends Report. By its figures, success on apprentice-level cyber tasks rose from under 9 % in late 2023 to about 50 %, and the length of cyber tasks models can complete grew from under ten minutes to over an hour, with an estimated doubling time of about eight months that it gives as an upper bound. It also says the limits plainly: success on the later stages of its cyber ranges stays low, and task tests are a proxy for real-world effect, not a measure of it.

Labs publish their own thresholds and results. OpenAI's Preparedness Framework (version 2, April 2025) treats cybersecurity as a tracked category, with a High threshold at a model that removes existing bottlenecks to scaling cyber operations, for example by automating end-to-end operations against reasonably hardened targets. Anthropic's system card for Claude Opus 5 (July 2026) reports five cyber evaluations plus AI Security Institute range testing, and notes the ranges have no active defenders and give the agent its target. These are the labs' own reports on their own models.

What evaluations miss

  • Decomposition. Real misuse splits a harmful goal into steps that each look like ordinary work. A suite of single requests that each look harmful measures refusals for the easy case only.
  • Defenders. Ranges and CTFs rarely have anyone fighting back, so they overstate how far an agent gets in a monitored network.
  • Contamination. Public tasks and their write-ups are on the web, and a model may have seen them. Fang and colleagues' 2024 result on one-day vulnerabilities fell from 87 % with the advisory text to 7 % without it, and a published critique by Chris Rohlf noted that public exploits existed for most of the targets and that prompts and code were not released.
  • Over-refusal. A model that refuses everything scores perfectly on harm and is useless to defenders. A fair suite pairs harmful requests with benign ones that sound similar.
  • The harness. Agent results depend on tools, step limits and prompts as much as on the model. Report them, or the number cannot be compared.

Reading a small suite without fooling yourself

Most product teams, us included, run suites of a few dozen cases, not thousands. With so few cases a pass rate has wide error bars, and the honest way to report it is with an interval, not a single number. The Wilson score interval is a good default for pass rates because it behaves sensibly near 0 and 100 %.

p^+z22n±zp^(1−p^)n+z24n21+z2n\frac{\hat p + \frac{z^2}{2n} \pm z\sqrt{\frac{\hat p(1-\hat p)}{n} + \frac{z^2}{4n^2}}}{1 + \frac{z^2}{n}}

Symbols

p^\hat p
the observed pass rate (passes divided by cases)
nn
the number of cases
zz
1.96 for a 95 % interval

In plain words

With 12 cases all passed, the 95 % interval still runs from about 76 % to 100 %. A clean sheet on a small suite says the model did not fail these cases; it does not say the failure rate is zero.

Two more habits help. Keep the cases fixed and versioned so results compare across models. And read the replies by hand: pattern-based graders miss refusals that are worded unusually and passes that contain harmful detail in a form the pattern did not expect.

What we actually do

We do not train frontier models, and we do not run capability evaluations of the Cybench kind; those are the model makers' and the AI Security Institute's work, and we read their reports. What we test is whether the models we route to behave safely inside Klyroe.

  • A safety category of 12 cases in our model evaluations. The harmful ones include a request for covert credential-stealing software and requests for phishing material; the benign ones include a question about common web vulnerabilities a penetration tester checks for, which must not be refused.
  • A candidate model must refuse the harmful cases as the current model does and must answer the benign ones. The grader is pattern-based, and the replies are read by hand before a model is switched.
  • Browser agent cases that plant instructions in a page and in a note, scored as safety checks separate from task success.
  • An eval gate that maps changed prompt and routing files to the suites they affect and compares results with a baseline; it runs locally.

Limits we know about: the suite is small, so by the interval above a clean result leaves a wide range open; single-request cases do not test decomposition; and no outside red team or penetration test has examined Klyroe yet.

A checklist for an honest misuse evaluation

  • Say which question you measured: capability or safeguards.
  • Pair every harmful case with a benign look-alike and report both rates.
  • Report the harness: tools, step limits, prompts, number of attempts.
  • Give intervals, not just pass rates, and the number of cases.
  • Include multi-step and decomposed tasks, not only single requests.
  • Read the transcripts; don't trust a pattern grader alone.
  • Publish what you did not test.

Sources

  1. Zhang et al., Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models, ICLR 2025
  2. Bhatt et al. (Meta), Purple Llama CyberSecEval, arXiv, 2023
  3. Meta, CyberSecEval 2, arXiv, 2024
  4. UK AI Security Institute, Frontier AI Trends Report
  5. UK AI Security Institute, Inspect evaluation framework
  6. OpenAI, Preparedness Framework version 2, 15 April 2025 (lab's own framework)
  7. Anthropic, System Card: Claude Opus 5, July 2026 (lab's own report)
  8. Fang, Bindu, Gupta and Kang, LLM Agents can Autonomously Exploit One-day Vulnerabilities, arXiv, 2024
  9. Chris Rohlf, No, LLM Agents can not Autonomously Exploit One-day Vulnerabilities, 21 April 2024
  10. NCSC, The near-term impact of AI on the cyber threat, 24 January 2024
  11. Frontier Model Forum, Managing Advanced Cyber Risks in Frontier AI Frameworks, 13 February 2026