AI Scrapers Surge Wikimedia Commons Bandwidth by 50%—A Growing Threat to Open Knowledge
AI crawlers are straining Wikimedia Commons' bandwidth. Here's why it matters.
Matilda
AI Scrapers Surge Wikimedia Commons Bandwidth by 50%—A Growing Threat to Open Knowledge
AI-powered web crawlers are rapidly reshaping the internet, and not always for the better. Wikimedia Commons, the open repository of images, videos, and audio files under the Wikimedia Foundation, has seen its bandwidth demands skyrocket by 50% since January 2024. Unlike the organic rise in human users, this spike is largely attributed to AI scrapers harvesting massive amounts of data for machine learning models. Image:Google How AI Scrapers Are Overloading Wikimedia Commons The Wikimedia Foundation recently disclosed that nearly 65% of its most resource-intensive traffic stems from bots, even though they account for just 35% of total page views. Unlike human visitors, these scrapers indiscriminately access bulk content—including rarely viewed pages—forcing Wikimedia's servers to fetch data from its core infrastructure. This process is costly and unsustainable. "Our infrastructure is built to sustain sudden traffic spikes from humans during high-interest events, but the amount of…
