How to stop excessive bot traffic and AI scrapers
Massive volumes of automated scrapers overwhelm sites with tens of thousands of public pages, driving hosting costs up significantly. This overwhelming bot traffic ruins the reliability of web data and forces owners to look for specialized blocking services to regain control.
What people tried
Every workaround mentioned in the threads below. We haven’t tested any of them — and nobody here is claiming they worked.
- 1using Vercel and Cloudflare to block traffic
- 2putting the app behind an auth gate with cloudflare turnstile
- 3switching to hcaptcha
- 4using truesign.ai to block crawlers
In their words
Unedited, most upvoted first, each linked to the thread it came from.
“My concern is the contact information.”source ↗
“If someone can simply scrape the pages and collect both the listings and all of their contact details, they could essentially recreate a large part of the directory elsewhere.”source ↗
“What I’m trying to understand is what practical measures experienced WordPress developers use to make large-scale scraping of contact information significantly harder without making the site frustrating for legitimate users or hurting SEO.”source ↗
“I’m currently quite lost on the best solution for this, so I’d appreciate practical advice from people who have dealt with this problem.”source ↗
“We host a web app which has around 50,000 statically generated public pages, and the amount of bot traffic is insane.”source ↗
“I’ve never seen anything like it before, our hosting costs are up like crazy, and besides all of this, I can’t trust anything I read or see on this web anymore.”source ↗
“I occasionally suffer this same problem when happy crawlers find my site and use truesign.ai to block them, so far successfully.”source ↗
Where this came up
People with this problem also raised
- 4Why are automated bot attacks and hacking attempts suddenly surging?
- 3How to stop an email spam bomb attacking a user inbox
- 10Why does a WordPress site need a plugin for everything?
- 2How to stop cache warming from crashing the server
- 3Why is my website traffic suddenly spiking with no sales?
- 3Why is security software generating so many false alerts?