How to stop excessive bot traffic and AI scrapers
Massive volumes of automated scrapers overwhelm sites with tens of thousands of public pages, driving hosting costs up significantly. This overwhelming bot traffic ruins the reliability of web data and forces owners to look for specialized blocking services to regain control.
What the numbers show
Counted from the threads quoted below. How we count
- The most recent one is from October 2026.
- This comes up 104.2× more often in r/Wordpress than across the site as a whole.
What people tried
Every workaround mentioned in the threads below. We haven’t tested any of them — and nobody here is claiming they worked.
Found another way around this?
Share your steps and trade-offs so fellow engineers don't have to reinvent the wheel. No account required.
- 1using Vercel and Cloudflare to block trafficDid this help you?
- 2putting the app behind an auth gate with cloudflare turnstileDid this help you?
- 3switching to hcaptchaDid this help you?
- 4using truesign.ai to block crawlersDid this help you?
- 5Introducing a WAF rule in Cloudflare to block visits via wildcard patternsDid this help you?
- 6Adding nginx rules to return a 404 status on pagination URLsDid this help you?
- 7Disabling archives and feeds from being indexed or crawlable using SEO plugins like Yoast or Rank MathDid this help you?
- 8Checking access logs by user agent to analyze bot behaviorDid this help you?
- 9Locking down IT access and trying to secure company dataDid this help you?
- 10Pre-anonymizing data before letting AI systems process itDid this help you?
- 11Manually applying the human or bot session filter in the dashboardDid this help you?
- 12Using Cloudflare WAF and custom WAF rules to block trafficDid this help you?
- 13Setting up rules and cache to manage server loadDid this help you?
- 14blocking entire countries or geographical areas using CloudflareDid this help you?
- 15asking the web host to block traffic from certain regionsDid this help you?
- 16setting up Cloudflare security rules to block common threatsDid this help you?
- 17Setting up Cloudflare with Bot Fight Mode / Block AI Bots enabledDid this help you?
- 18Upgrading hosting plans or increasing PHP workersDid this help you?
- 19Migrating to better hosting companies (e.g., Kinsta, GreenGeeks)Did this help you?
- 20Setting rate limiting or Crawl-delay in robots.txt / server configDid this help you?
- 21Blocking specific IP ranges or countries (Geo-blocking)Did this help you?
- 22Asking support to temporarily filter abusive traffic or apply rate limiting at the server levelDid this help you?
- 23Outright dropping the single IP address instead of trying to overcomplicate rate limiting.Did this help you?
- 24Using fail2ban to scan logs and automate drops.Did this help you?
- 25Using ipset to block known bogus IPs.Did this help you?
- 26Checking the IP's geolocation and blocking it if it's outside the target audience country.Did this help you?
- 27Leaving 'Under Attack' mode on permanently, despite it showing a loading screen to real users.Did this help you?
- 28Creating custom Web Application Firewall (WAF) rules to block or challenge specific IPs, countries, or paths.Did this help you?
- 29Caching HTML at the CDN level so bot requests never reach the origin server running PHP.Did this help you?
- 30Adding specific search engine bot blocks directly to the .htaccess file.Did this help you?
- 31Disabling XML-RPC to prevent brute force attempts.Did this help you?
- 32Using bot throttling plugins to temporarily block scraping behavior.Did this help you?
- 33Setting up Web Application Firewall (WAF) rules to block specific crawler user agents.Did this help you?
- 34Disabling plugins that inject sitewide JavaScript to reduce main thread blocking.Did this help you?
- 35Checking server access logs to identify the exact behavior of the bots.Did this help you?
- 36Looking into adding a txt file (robots.txt) to block crawler access.Did this help you?
In their words
Unedited, grouped by where they were said, most upvoted first within each place, each linked to the thread it came from.
“these last months, maybe the last year, the sessions on my store went crazy : 100-200 simultaneous sessions non-stop, day and night.”source ↗
“My conversion rate in August : 0.66% with the bots, 1.57% without.”source ↗
“Since it's my business I of course need to link my shop, but f*** AI slop and them making use of my stuff.”source ↗
“Is there a way I can block them off Shopify?”source ↗
“I'm not a coder or anything, saw a short Info of a txt that can be added to block the "friendly" bots?”source ↗
“My concern is the contact information.”source ↗
“If someone can simply scrape the pages and collect both the listings and all of their contact details, they could essentially recreate a large part of the directory elsewhere.”source ↗
“What I’m trying to understand is what practical measures experienced WordPress developers use to make large-scale scraping of contact information significantly harder without making the site frustrating for legitimate users or hurting SEO.”source ↗
“I’m currently quite lost on the best solution for this, so I’d appreciate practical advice from people who have dealt with this problem.”source ↗
“I noticed Ive been getting a high load of Traffic from China, its my top traffic source and im located in the USA. I read that I should block China traffic, but how do I do that on Wordpress? Also is it safe for SEO to block traffic from China?”source ↗
“My website crashes every other hour for an hour so down like 50% of the time UNLESS I have under attack mode on in Cloudflare which blocks bots. If I turn it off and the website crashes, it immediately comes back online a minute after turning it on.”source ↗
“Are these number normal? Is there a way to fix this without having under attack mode on 24/7? I feel like it annoys the viewers with that 5 second loading screen and also potentially effect google analytics.”source ↗
“Woocommerce site on godaddy managed wp throwing constant 503s even with plugins disabled”source ↗
“In access log it shows dozens of different IPs from all over the world per second. Possible bot problem?”source ↗
“We're experiencing a lot of bot traffic going to those addresses and are not sure why.”source ↗
“I put the Cloudflare rule in place this morning (2-3 hours before writing this) and according to Cloudflare the rule has been triggered 8,430 times since I created it!”source ↗
“My WP site for my self-published books has very low traffic counts, but these last two days, it's seeing hits from all over China.”source ↗
“probably bots scraping content, happens to my art portfolio all the time”source ↗
“Really? I’ve been trying to sort my server with rules and cache.”source ↗
“I had this problem. It was low-rent AI bots hammering my shop. When bot-hammering uses up your server quotas GoDaddy’s throws 503s to your audience.”source ↗
“Had similar resource usage on my server with a very low traffic site. Moved it to a new server. Monitored requests with cloudflare and Google. Noticed a metric ton of search requests for non-existent locations under the domain.”source ↗
“I used the block anthropic function and set up all the waf rules so the site isn't going offline anymore. However after doing that everything slowed down and it says 10 duplicate queries and 10 slow queries and 40 second wait time with query monitor.”source ↗
“Blocking anthropic suddenly triggered a huge mess and I don't know why. Also the bot hits were 700k before blocking now they are 2 million. Though in the events pane it says blocked but they are still served data of 20gb.”source ↗
“We host a web app which has around 50,000 statically generated public pages, and the amount of bot traffic is insane.”source ↗
“I’ve never seen anything like it before, our hosting costs are up like crazy, and besides all of this, I can’t trust anything I read or see on this web anymore.”source ↗
“I occasionally suffer this same problem when happy crawlers find my site and use truesign.ai to block them, so far successfully.”source ↗
“I looked in my logs and noticed one bot (with the same IP and same user agent) has been making requests to my server at the rate of one request per second for about 5 minutes.”source ↗
“I thought about using the recent module of iptables but I'm not sure if that is the best because if I pick too low of numbers then I block legit users and if I pick too high of numbers then my system might go to a crawl.”source ↗
“I'm doing what I can to secuure my company's data, but I feel at this point it is going to be pointless because it feels like legislation is going to change to make trademark/copyright not a real enforceable thing.”source ↗
“Banning an ip may be an option but what if a legitimate user also uses that IP as well at a later time?”source ↗
Where this came up
People with this problem also raised
- 4Why are automated bot attacks and hacking attempts suddenly surging?
- 3How to stop an email spam bomb attacking a user inbox
- 10Why does a WordPress site need a plugin for everything?
- 2How to stop cache warming from crashing the server
- 6Why is my website traffic suddenly spiking with no sales?
- 9Why is security software generating so many false alerts?