Sir-Bench – benchmark for security incident response agents

Security geeks are buzzing over SIR-Bench, a new test that claims to separate real AI “investigators” from noisy alert parrots. The authors jumped straight into the comments to say this isn’t another trivia test—it checks if an AI actually finds new clues during an incident, not just repeats what the alarm said. Built from 794 cases based on 129 real patterns and replayed in controlled cloud setups via the cheekily named Once Upon A Threat (OUAT), it scores AIs on three things: making the right call (triage), discovering new evidence, and using tools appropriately. The benchmark’s headline numbers—97.1% true positives, 73.4% false positive rejection, and an average of 5.67 new key findings per case—had readers going “woah.”

Drama scale? Early days, but mood swings from hype to show me receipts. The phrase “alert parroting” is already becoming the community’s meme-in-waiting, with readers picturing squawking bots vs. Sherlock-bots. Meanwhile, the spicy bit is the LLM-as-Judge idea—an AI judging another AI—where the authors say the judge demands concrete proof before giving credit. Commenters are lining up questions on methodology and scoring, and the authors say they’re ready to take them. If SIR-Bench delivers as promised, it could be the test that stops AI tools from bluffing and starts rewarding real detective work. Read the paper and pick a side: parrot or detective.