AI Agent Commands: Humans Miss 1 in 3 Threats in 40K Game Runs
Original: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs
Why This Matters
Findings expose critical human oversight failures in AI agent security, especially for credential theft and script-injection attacks.
A browser game simulating human-in-the-loop AI agent oversight collected data from over 40,000 runs and 409,000 approve/deny decisions. Results show average players missed 1 in 3 threats, with a mean accuracy of just 66.3%, revealing significant weaknesses in human oversight of AI coding agents.
Scale X published findings from a browser game designed to test human ability to approve or deny AI coding agent commands under time pressure. After 40,000+ sessions and 409,000 individual decisions, key results emerged: the average miss rate for threats was 33.7% (mean accuracy 66.3%), 32.9% of sessions ended with a net negative score, and 7% of players approved every single prompt. Threat categories varied significantly in miss rates: obvious destructive commands like 'rm -rf /' were missed only 11.7% of the time, while credential exfiltration commands such as 'cat ~/.aws/credentials' were missed 35% of the time. The single most-missed threat was 'npm run analyze,' approved 64.7% of the time despite the malicious payload being visible in the agent's history log. Collectively, commands hiding payloads behind familiar script names (npm run analyze, setup, deploy) were missed 52.5% of the time — nearly double the 28.4% miss rate for other exfiltration-style attacks. The findings underscore a structural problem: familiar command names reduce perceived risk even when malicious intent is explicitly displayed.