Of AI Training Bots and Cultural Heritage

In recent months, the information community has become increasingly aware of a growing problem: a rise in AI training bot activity online. AI bots crawling websites is not a new phenomenon, but the activity appears to have spiked over the past year, at times disrupting access to online scholarly content. Todd Carpenter took note of this issue in his May Letter from the Executive Director, and a June 2 story from Nature also called attention to the impact of AI bots on science content online. Seeking to understand the scale and impact of bot activity on online collections hosted by galleries, libraries, archives, and museums, the GLAM-E Lab conducted a survey earlier this year. 

The resulting report by Michael Weinberg, "Are AI Bots Knocking Cultural Heritage Offline?," presents results from a survey of 43 institutions from around the world. Thirty-nine reported a recent increase in traffic in the past year, and 27 of these had identified AI bots as the cause (several others suspected bots were responsible, but were not certain). Several respondents reported a disruption in service to users as a result of the activity:

Many respondents did not realize they were experiencing a growth in bot traffic until the traffic reached the point where it overwhelmed the service and knocked online collections offline.

The in-depth report also covers patterns in bot traffic, how institutions are identifying (or not identifying) the source of increases in website traffic, and the methods that are most effective in blocking bot activity. It also outlines concerns about balancing solutions to the problem with ensuring (human) user access and considers the long-term costs tied to prevention methods. Read the full report.