LLMs made it cheap to flood bug bounty programs with submissions. Here's how Elastic built an AI triage agent that matches human decisions 85% of the time, including the architecture, threat model and calibration against 3,300 real reports

The problem: Triage doesn't scale linearly

AI triage architecture: Two VMs, two skills, one orchestrator

Why not a single VM?

Why ephemeral VMs instead of containers on a long-running host?

Vulnerability analysis pipeline: Eight stages with built-in skepticism

Eight stages of AI-powered vulnerability triage

Elastic-specific triage rules

How the AI assesses realistic exploitability

The adversarial review: Challenging your own AI

Calibrating against 3,317 historical reports

Reproducing vulnerabilities in a sandbox

Threat model for AI systems processing untrusted input

Defense in depth for AI vulnerability triage

Six Docker Compose topologies for Elastic Stack reproduction

Orchestrating AI triage with Elastic Workflows

Lessons from building AI-powered bug bounty triage

Report framing is the hardest bias to counter

Calibration data matters more than prompt engineering

Reproduction is expensive but decisive

Security of the triage system is itself a security problem

What's next for AI vulnerability triage at Elastic

Elastic built an AI agentic workflow that triages bug bounty reports for ~$2 each, matching human analysts 85% of the time. Architecture, threat model and results.

Our system agrees with human triage decisions 85% of the time, validated against 764 known-outcome HackerOne reports. We further calibrated the triage rules against our full corpus of over 3,300 historical reports, iteratively improving accuracy from 75% to 84% on a stratified sample. The disagreements tend to cluster around edge cases where exploitability depends on deployment-specific configuration. Every report still gets human review before any external action is taken, so accuracy affects efficiency, not safety.

How accurate is AI-powered vulnerability triage compared to human analysts?

Yes, with significant sandboxing. Our system provisions ephemeral Elastic Stack environments via Docker Compose, executes researcher-described steps in a resource-limited container with no internet access, and interprets the output. The main constraint is that the reproduction environment must defend against the same class of attacks the report describes, which requires network isolation, egress filtering, credential separation, and ephemeral infrastructure.

Can an AI agent reproduce vulnerability findings automatically?

Analysis costs ~$0.50 to $1.15 per report, mostly Claude API usage. Reproduction adds another ~$0.80 to $4.90, when needed, but roughly 70% of reports are rejected at analysis and never reach reproduction. The goal isn’t to replace human review but to deliver predigested analysis so that a security engineer can reach a decision in minutes rather than spending time on mechanical first-pass work.

What does it cost to triage a bug bounty report with AI?

Only with defense in depth. The primary threat is prompt injection via report text, rather than sandbox escape during reproduction. The most important mitigation is architectural: The agent cannot take external actions directly. All output goes to storage, where the Elastic Workflows orchestrator posts results as internal comments on the HackerOne report. A human reviews the analysis before taking any external action. High-value credentials never leave the orchestration layer. On top of that: ephemeral VMs that self-destruct after 30 minutes, proxy-filtered egress, Docker containers with no internet access and 512MB RAM limits, credentials in memory only, and read-only workspace mounts.

Is it safe to let an AI system process and reproduce untrusted vulnerability reports?

Framing bias is the single biggest source of incorrect triage for both humans and AI. We added explicit detection in the analysis and adversarial review stages: The system checks whether it mirrored the report's language, anchored CVSS scores to claimed severity, or accepted prerequisites without independently evaluating them. Historical acceptance rates per category provide a statistical check against overgenerous assessments.

How do you stop AI from adopting the researcher's severity framing?

Not even close. Acceptance rates vary wildly by category. Some categories have rates in the single digits because the reported behavior is often a feature working as designed or falls outside the program's scope. Other categories have meaningfully higher acceptance rates. These rates are program-specific, but most bug bounty programs see a similar pattern: A small number of categories account for the majority of valid findings, while the long tail of submissions has a low acceptance rate.

Do all vulnerability categories get accepted at the same rate?

The primary defense is architectural rather than detection-based: The agent cannot take any external action directly. All output flows to storage, and a human reviews it before any message is posted to HackerOne or any issue is created. High-value credentials, like the HackerOne API token, never leave the orchestration layer, so even a fully compromised analysis cannot act on its own. The adversarial review adds a second-pass check on manipulated output.

How do you prevent prompt injection in an AI agent that processes untrusted input?

The architecture generalizes: Structured multistage analysis, adversarial review, sandboxed reproduction, and human-in-the-loop decisions work with any product. What don’t generalize are the triage rules. The rules encoding which vulnerabilities are actually features, what constitutes valid DoS versus volumetric abuse, and how to assess exploitability given your product's access model are specific to your product and your program's policies. That institutional knowledge is the hardest part to encode, and it’s where the accuracy comes from.

Author

Ioannis Kakavas

Articles

Agents vs. agents: how we triage HackerOne reports for $2 each, 85% as well as a human