Michael Weinberg
Executive Director, Engelberg Center on Innovation Law & Policy
New York University
Martin Klein
Research Data Group Lead
Pacific Northwest National Laboratory
Tim Lloyd
CEO
LibLynx
Dispatches from the Great AI Scraping Bot War
In recent years, publicly available digital collections have experienced a significant, and at times debilitating, increase in traffic. The source of this traffic is bots scraping pages in order to build data sets to train artificial intelligence (AI) models. Regardless of one’s opinion about the legitimacy of training AI on information scraped from the internet, the volume and behavior of these bots has created a series of technical challenges for the teams tasked with keeping these collections online. This briefing will provide an overview of the current status of this cat-and-mouse conflict, and present resources designed to spread actionable, up-to-date information about countermeasures in a way that is useful to the community without revealing those measures to the bots.
The Good Bot – Bad Bot Conundrum in the Open Science Ecosystem
Open access repositories serve as a foundational pillar of global research infrastructure, together stewarding an estimated 340 million records designed for barrier-free access by both humans and machines. However, this “open by design” philosophy is currently under threat. A recent survey conducted by the Confederation of Open Access Repositories (COAR) revealed that aggressive, automated bot traffic is placing an unsustainable strain on repository infrastructures. Many institutions reported serious service disruptions, leading some to implement “blanket blocks” on all bot activity. While these defensive measures are understandable, they risk undermining the core principles of open science by disrupting the very flow of information and speed of innovation that open access was intended to enable. Trying to identify mitigation strategies, the COAR Taskforce reached a critical realization: there is currently no “magic bullet” solution to manage aggressive bots without also hindering legitimate machine use. Without urgent, coordinated action, the repository community risks adopting fragmented, defensive measures that will ultimately diminish the value of the entire open science ecosystem.
This presentation will highlight the severity of this challenge and outline a roadmap for developing shared good practices. The aim is to provide the international repository community with the tools to manage AI traffic while preserving the openness essential for modern research.
The Bot Taxonomy: A Collaborative Project to Develop a Taxonomy for Knowledge Platforms
Bot traffic is exploding across knowledge platforms (publishers, repositories, libraries), with a growing ecosystem of good and bad bots that platforms need to either block or optimize for. However, corporate bot lists are difficult to engage with or automate because they require technical expertise and often ignore niche bots that information providers rely on, such as archival bots. This limits the ability of platform managers to make informed decisions on what bots to allow onto their platforms and how to configure an appropriate experience. The Bot Taxonomy Project is collaborating with thought leaders from across the community to develop a taxonomy that empowers all organizations to manage bots regardless of technical expertise or size. An effective taxonomy will make bot information accessible to non-experts, so that non-technical staff tasked with managing platforms can make effective decisions on bot management, and enable scalable automation of bot management, so that ongoing management is cost-effective. This presentation will share information about the project background, the rationale for a community-driven taxonomy, an update on the project’s status, and lessons learned to date.
https://www.glamelab.org/products/are-ai-bots-knocking-cultural-heritage-offline/
https://coar-repositories.org/what-we-do/ai-and-repositories/
https://dealing-with-bots.coar-repositories.org/
https://scholarlykitchen.sspnet.org/2026/05/07/guest-post-love-death-robots-scholarly-edition/
