How AI Agents Triage Security Alerts, and What the First Tests Show
Security teams receive far more alerts than analysts can read, and vendors now sell AI that sorts them. This explains how alert-triage agents are built, what a named operator, a vendor and two research benchmarks each report, and why the evidence still ranges from production to prototype.
Checked against Google Cloud's reference architecture (reviewed April 8, 2026), Rapid7's April 2025 announcement, RSM's case study, and the ALERT-BENCH and SIR-Bench preprints
Role scorecard
Information Security Analysts SOC 15-1212.00
- Core tasks AI does in production0 of 2
- Most mature evidenceIn production
- CostRapid7: included at no additional cost for InsightIDR users (vendor-stated); other prices not found
- BenefitRSM estimates $2M a year saved (RSM, as reported on its vendor's site)
As of · Tasks: O*NET 31.01
| Core task (O*NET) | What AI or robots do | Extent | Evidence grade | Example case | Source |
|---|---|---|---|---|---|
| Review violations of computer security procedures and discuss procedures with violators to ensure violations are not repeated. | AI investigates each alert and returns a verdict with its reasoning for an analyst to check; the discussion with the people involved stays with the analyst | Assists | In production | RSM Defense runs Intezer on about 500,000 alerts a quarter, with analysts checking the results (RSM-stated, vendor-hosted case study) | 4 |
| Perform risk assessments and execute tests of data processing system to ensure functioning of data processing activities and security measures. | Google's reference architecture has an agent look up asset misconfigurations in a third-party posture-management tool; no operator is named | Not yet | Vendor claim | Google Cloud "Agentic AI use case: Orchestrate security operations workflows", last reviewed April 8, 2026 | 2 |
Extent and grade follow our evidence grades. The first row is "Assists", not "Does it", because RSM's analysts check the verdicts and the task also includes talking to the people involved. Alert triage is not a listed O*NET task by name, so we mapped it to the closest one; other readers might choose another. We show two of the 11 core tasks O*NET lists for the occupation; the rest (encryption and firewall work, policy documents, user training and similar) have no AI case in the sources we opened.1
Costs and benefits
| Price | Rapid7 says AI Alert Triage will be available to InsightIDR users "at no additional cost" (Rapid7-stated, April 2025); no price for the other products was found in the sources we opened3 |
|---|---|
| Savings | RSM estimates about $2 million a year, according to Intezer's case study, which is hosted by the vendor4 |
| Speed | Average triage time of 2 minutes 30 seconds per alert at RSM, and about 500,000 alerts ingested a quarter (as stated by the vendor-hosted case study)4 |
| Misses in testing | In the ALERT-BENCH test every approach missed at least 40.4% of attack-related alerts; the authors' own redesign cut that to 3.1% while escalating 18.4% of alerts to analysts (preprint, not peer reviewed)5 |
As of · Sources: the operator, vendor and research pages cited in each row23456
| Source | Who states it | Evidence grade | What was measured | Result as stated | Note | Source |
|---|---|---|---|---|---|---|
| RSM Defense with Intezer | Operator, vendor-hosted | In production | Alerts ingested and investigated per quarter; average triage time; wrong verdicts found by RSM analysts | About 500,000; 2 min 30 s; none found; est. $2M saved a year | RSM-stated, as published by Intezer; analysts check results | 4 |
| Rapid7 AI Alert Triage | Vendor | Vendor claim | Share of benign alerts closed across its managed detection and response service | 99.93% benign alert closure rate; over 8 trillion alerts a week processed | Rapid7-stated, April 2025; AI suggests a disposition and the analyst decides | 3 |
| Google Cloud reference architecture | Vendor documentation | Vendor claim | Multi-agent design for triage and investigation with human approval | No performance figures given | Last reviewed April 8, 2026; no operator named | 2 |
| ALERT-BENCH | Research preprint | Prototype | Five agent approaches on 1,247 alerts from one multi-stage attack scenario replayed through a live SIEM | Every approach missed at least 40.4% of attack-related alerts; F1 0.371 to 0.744 | Authors also propose AIDA: F1 0.958, false-negative rate 3.1%, 18.4% escalated | 5 |
| SIR-Bench | Research preprint | Prototype | 794 test cases from 129 incident patterns, scored by an LLM judge | The authors' own SIR agent: 97.1% true positives detected, 73.4% false positives rejected | Authors built both benchmark and agent; "baseline" for future work | 6 |
The figures come from different measures and are not comparable: closure rate, triage time, test-set recall and F1 answer different questions. Nothing here is independent field data; the RSM and Rapid7 figures are stated by the organisations that sell or run the product, and the two benchmarks use different alerts and scoring. We did not average or rank them.
How it works
The problem: more alerts than people can read
A security operations center (SOC) is the team that watches an organisation's systems for attacks. Tools flag suspicious activity as alerts, and an analyst must decide for each one whether it is harmless, needs investigation or must be escalated. Rapid7 cites a survey in which SOC teams received an average of 4,484 alerts a day and 67% were ignored because of alert fatigue and false positives; that is Rapid7's use of a third-party survey, which we have not opened.3 Sorting alerts, called alert triage, is the first job that vendors aim AI at.3
Design 1: a classifier that proposes a verdict
The simplest design adds a field to each alert. Rapid7 says its AI Alert Triage engine "suggests an initial disposition (benign or malicious)" with the reason it chose and supporting information from the investigation, and that its model was trained and tested by its managed detection and response service.3 The analyst still decides: the field is there "to assist" them.3 Its headline figure, a 99.93% benign alert closure rate, counts how many benign alerts were closed. It does not say how many real attacks were wrongly closed, which is the figure the research below puts at the centre.3
Design 2: an agent that investigates
The second design lets an AI agent run its own searches. Google's reference architecture, reviewed April 8, 2026, has a root "SOC manager" agent that reads the analyst's request, fetches matching runbooks and the incident response plan from a retrieval database (see retrieval-augmented generation), then splits the job between specialised agents: a "Tier 1 analyst" that retrieves alert details, affected assets and user context, and a threat-intelligence researcher that links indicators to known attacker behaviour.2 The agents reach the log system, threat intelligence and other tools through Model Context Protocol servers, and the architecture lists "human-in-the-loop approval" among the actions the system can perform.2 The document gives no performance figures and names no operator.2
A named operator: RSM Defense
RSM's case study, published on its vendor Intezer's site, is the only operator account we found. It reports about 500,000 alerts ingested and investigated per quarter, an average triage time of 2 minutes 30 seconds, and no instance in which RSM's analysts found an incorrect verdict; RSM's Willoughby estimates the saving at roughly $2 million a year.4 RSM's analysts check the results, and the page says the quality assurance explains why they have found none.4 Because the page is hosted and written by the vendor, we treat the figures as RSM's statements as reported, not as an independent audit.
What the first tests show
ALERT-BENCH (arXiv, October 7, 2026) replays enterprise telemetry through a live SIEM, so each system must find its own evidence. In 1,247 alerts from one multi-stage attack scenario, "every approach missed at least 40.4% of attack-related alerts"; attack alerts were more likely to be dismissed when a search returned no records; and reviewing a decision within the same context made more correct decisions wrong than it repaired.5 The authors then built a multi-agent design, AIDA, that must propose a decision before an independent challenge and needs stronger evidence before dismissing an alert. On the same alerts they report an F1 score of 0.958 against 0.371 to 0.744 for the five approaches, and a false-negative rate of 3.1% with 18.4% of alerts escalated to analysts.5 That result is the authors' own design tested on their own benchmark, and it is a preprint.
SIR-Bench (arXiv, April 13, 2026) takes a different angle: 794 test cases from 129 anonymised incident patterns, built to tell real investigation from "alert parroting", scored on triage accuracy, novel findings and tool use by an "adversarial LLM-as-Judge".6 The authors' own SIR agent reached 97.1% true-positive detection and 73.4% false-positive rejection, which they describe as a baseline for future agents.6 Note the gap in that result: the agent rejected under three quarters of the harmless alerts, so, by the authors' measure, a quarter or more would still reach a person.
What people still do
In every source above a person remains in the loop: RSM's analysts check results,4 Rapid7's field assists an analyst,3 Google recommends a human-in-the-loop flow so supervisors can "approve, override, or pause agent actions",2 and the best ALERT-BENCH design still escalates 18.4% of alerts.5 Our scorecard therefore counts the cases as "Assists". For how we grade evidence, see how we measure AI in roles.
Sources
- O*NET OnLine, 15-1212.00 – Information Security Analysts (O*NET 31.0 tasks; 2025 employment 192,900 and median annual wage $129,180 per its summary), retrieved 2026-10-25.
- Google Cloud, Agentic AI use case: Orchestrate security operations workflows, Cloud Architecture Center, last reviewed 2026-04-08.
- Rapid7 (Chris Wraight), InsightIDR AI Alert Triage Automatically Classifies Alerts with 99.93% Benign Alert Accuracy, blog, April 29, 2025 (updated May 2, 2025).
- Intezer, RSM scales alert triage with AI, saving $2M annually, customer case study (vendor-hosted).
- Tsegai, Kantchelian, Yao and Gao, From Investigation Failures to Reliable SOC Agents: Understanding and Improving LLM-Based Alert Triage, arXiv:2610.10608, preprint, October 7, 2026.
- Begimher, Leo, Huang, Gaw and Zheng, SIR-Bench: Evaluating Investigation Depth in Security Incident Response Agents, arXiv:2604.12040, preprint, April 13, 2026.