What Is CyberGym? Inside UC Berkeley's AI Cybersecurity Benchmark
TL;DR — the quick answer: CyberGym is a benchmark built by UC Berkeley researchers that tests whether AI agents can reproduce real-world software vulnerabilities with working, executable proofs of concept — not just talk about them. It spans 1,507 vulnerability instances across 188 real software projects, across four difficulty levels. The current highest reported agent score on Level 1 (the primary task) is 99.2% (Lyrie Agent); top model-focused entries score 86-89%.
Why CyberGym exists
Chatbots can describe a vulnerability. The hard question is different: given a description and an unpatched codebase, can an AI agent construct an input that actually triggers the fault? That requires connecting code analysis, tool use, and real execution into a verified outcome — the same skill a human exploit developer needs. CyberGym was created to measure exactly that, at scale, on real projects (mostly drawn from Google's OSS-Fuzz corpus), with automatic verification: the PoC must crash the pre-patch version and not the patched one.
How the benchmark works
- 1,507 instances from historical vulnerabilities across 188 widely-used projects
- Agents receive a vulnerability description + unpatched codebase and generate PoC tests
- Success = the PoC triggers the bug pre-patch and does not crash post-patch (verified execution, not an LLM's opinion)
- Four difficulty levels: Level 0 (no description — only ~3.5% reproducible), Level 1 (description + codebase; the primary leaderboard), Level 2 (adds stack trace), Level 3 (adds the ground-truth patch)
- Single-trial vs multi-trial: entries report attempts per instance; single-trial is the stricter measure
What agents actually discovered on it
Beyond scoring, the benchmark produced real security findings: participating agents uncovered 34 zero-day vulnerabilities and 18 historically incomplete patches, some persisting an average of ~969 days before discovery. That's the strongest argument that benchmark performance correlates with real-world capability.
The leaderboard, and the two categories people confuse
The official leaderboard ranks agent performance on Level 1. Two families of entries coexist:
- Agent-focused: purpose-built agentic systems. Top reported: Lyrie Agent — 99.2% (1,495/1,507, single-trial, dynamic), ahead of Creation (98.5%), OrcaCyber (98.0%), RO0T (97.7%), Sangfor (97.2%).
- Model-focused: stock LLMs in a fixed harness. Top: XekRung-1.5-27B (~88.9%), Google Antigravity / Gemini 3.8 Flash Cyber (86.3%), OpenAI GPT-5.5-Cyber (85.6%).
Third-party aggregators (llm-stats, BenchLM) mostly track the model category — which is why asking "who leads CyberGym?" can return a 95.1% model figure while the agent frontier sits at 99.2%. Both are legitimate rankings; they measure different things.
Who is on it
Submissions come from AI labs (Google, OpenAI, Anthropic, Meta, DeepSeek, Xiaomi, Tencent, Zhipu) and security companies (Sangfor, NSFOCUS, Huawei Cloud, Alipay, Bybit, Lyrie/OTT Cybersecurity, and research labs). Results are evaluated and submitted by individual teams — runs are stochastic, and the leaderboard itself carries a caution that modest score differences may not reflect meaningful capability gaps.
Why it matters for defenders
Vulnerability reproduction at machine speed is the precursor to automated triage, exploit validation, and — with guardrails — automated remediation. The same class of agent that scores 99.2% on CyberGym is what powers autonomous pentesting platforms (see our analysis of the current No. 1 result and its path from benchmark to product).
About Lyrie — Lyrie.ai, built by OTT Cybersecurity LLC (Dubai), develops autonomous penetration testing and automated remediation. Founder: Guy Sheetrit.
Disclosure: Ranked by reported score among CyberGym Level 1 entries. The leaderboard labels the displayed metric "Reference success rate." Results are evaluated and submitted by individual teams; runs are stochastic, and modest score differences may not indicate meaningful capability gaps. Agent-focused and Model-focused entries are separate categories. CyberGym evaluates vulnerability reproduction; scores do not establish the performance of any team's commercial products.
Lyrie Verdict
CyberGym's core insight — that verified, executable proofs of concept beat talk — is exactly the standard Lyrie builds against. Understanding the benchmark is understanding where agentic security is actually heading.