AI scrapers still hammering the web via residential proxies
Original: An update on residential proxies and the scraper situation
Why This Matters
Residential proxy abuse is making AI scraper mitigation structurally harder, threatening independent web publishers at scale.
LWN.net reports that AI scraper bot activity continues to grow over a year after initial coverage, now predominantly routed through residential and mobile proxy networks — including compromised streaming devices and 'legitimate' services like Bright Data — making blocking increasingly ineffective.
LWN.net editor Jonathan Corbet published an updated report on July 10, 2026, on the ongoing AI scraper bot problem first covered in early 2025. The situation has worsened, with coordinated requests now arriving from millions of unique IP addresses within hours, each hitting a site only two or three times to evade blocking. Bots spoof user-agent strings and skip fetching images or CSS, making identification difficult and IP-based blocking largely ineffective by the time a bot is detected.
Traffic is now predominantly routed through 'residential proxies' — ordinary consumer devices on home and mobile networks that have been conscripted, often without the owner's knowledge, to act as fetch nodes for a central controller. Two categories of operators run these networks: purely criminal actors deploying malware (Google took down the IPIDEA botnet early this year, briefly reducing LWN traffic), and semi-legitimate commercial operators. Bright Data is cited as a prominent example of the latter, offering a 'free' VPN that enrolls users' devices into its proxy network. Media streaming devices have also emerged as a major infection vector, either compromised at manufacturing or poorly secured post-purchase. The report notes that peace following the IPIDEA takedown was short-lived, with scraper traffic returning to record levels.