Volunteer Saves 50M-View LGBT Archive From AI Scrapers

A volunteer-run website preserving decades of LGBT history survived an unexpected threat that nearly took it offline forever.

Jonathan Harborne founded the LGBT History Project 15 years ago to ensure the struggles of past generations wouldn't be forgotten. The London-based creator wanted future generations to understand the fights for recognition and respect, including the devastating impact of the AIDS pandemic.

The site recently celebrated 50 million views and earned a place in the British Library's permanent archives. But when Harborne upgraded to a new server expecting better performance, something went terribly wrong.

AI bots from major tech companies were hammering his site with relentless automated requests. On a single day, Meta's crawler made roughly 26,000 requests, downloading about 12 gigabytes of content including years of editing histories and backend pages.

The problem hit Harborne's wallet hard. He funds the entire project himself and was forced to double his server capacity just to keep the site running while bots scraped his data.

"I was paying the cost for people like Meta to train their AIs," Harborne said. Multiple companies including Alibaba also sent crawlers that ignored his site's requests to stay away.

Harborne spent sleepless nights teaching himself advanced server protection techniques. He learned to configure security systems, write blocking rules, and create checks to separate human visitors from automated bots.

The Bright Side

The archive is now protected and accessible again to the people who need it most. Harborne's quick thinking and determination saved a crucial historical resource that documents struggles many want to forget but everyone should remember.

His work ensures that 50 million views worth of LGBT history, pain, triumph, and progress remains available for generations seeking to understand where they came from. One volunteer's late nights of learning turned a potential disaster into a testament to the power of preservation.

The LGBT History Project stands strong, protected and ready for its next 50 million views.