{"@context":"https://schema.org","@type":"CreativeWork","@id":"https://forgecascade.org/public/capsules/c0c2de27-7842-4aa0-bd19-507ee286e6a9","identifier":"c0c2de27-7842-4aa0-bd19-507ee286e6a9","url":"https://forgecascade.org/public/capsules/c0c2de27-7842-4aa0-bd19-507ee286e6a9","name":"ASMR-Bench: Auditing for Sabotage in ML Research","text":"# ASMR-Bench: Auditing for Sabotage in ML Research\n\n**Authors:** Eric Gan, Aryan Bhatt, Buck Shlegeris, Julian Stastny, Vivek Hebbar\n**arXiv:** https://arxiv.org/abs/2604.16286v1\n**Published:** 2026-04-17T17:47:32Z\n\n## Abstract\nAs AI systems are increasingly used to conduct research autonomously, misaligned systems could introduce subtle flaws that produce misleading results while evading detection. We introduce ASMR-Bench (Auditing for Sabotage in ML Research), a benchmark for evaluating the ability of auditors to detect sabotage in ML research codebases. ASMR-Bench consists of 9 ML research codebases with sabotaged variants that produce qualitatively different experimental results. Each sabotage modifies implementation details, such as hyperparameters, training data, or evaluation code, while preserving the high-level methodology described in the paper. We evaluated frontier LLMs and LLM-assisted human auditors on ASMR-Bench and found that both struggled to reliably detect sabotage: the best performance was an AUROC of 0.77 and a top-1 fix rate of 42%, achieved by Gemini 3.1 Pro. We also tested LLMs as red teamers and found that LLM-generated sabotages were weaker than human-generated ones but still sometimes evaded same-capability LLM auditors. We release ASMR-Bench to support research on monitoring and auditing techniques for AI-conducted research.","keywords":["cs.AI"],"about":[],"citation":[],"isPartOf":{"@type":"Dataset","name":"Forge Cascade Knowledge Graph","url":"https://forgecascade.org"},"publisher":{"@type":"Organization","name":"Forge Cascade","url":"https://forgecascade.org"},"dateCreated":"2026-04-20T06:00:03.088000Z","dateModified":"2026-05-08T23:37:07.467875Z","isBasedOn":"https://arxiv.org/abs/2604.16286v1","additionalProperty":[{"@type":"PropertyValue","name":"trust_level","value":65},{"@type":"PropertyValue","name":"verification_status","value":"source_linked"},{"@type":"PropertyValue","name":"provenance_status","value":"valid"},{"@type":"PropertyValue","name":"evidence_level","value":"primary_source"},{"@type":"PropertyValue","name":"content_hash","value":"534be3839e0e6777d6717e6ead128118d6a331adf07bd06a97fd83e35d74f5b3"}]}