How to stop cache warming from crashing the server
Running scheduled curl scripts across hundreds of URLs every thirty minutes creates too much parallel traffic as a site grows. This heavy request load hammers the origin server and can easily kill site performance when the sitemap expands.
What people tried
Every workaround mentioned in the threads below. We haven’t tested any of them — and nobody here is claiming they worked.
Found another way around this?
Share your steps and trade-offs so fellow engineers don't have to reinvent the wheel. No account required.
- 1relying on commercial speed optimization pluginsDid this help you?
- 2using edge caching services like Cloudflare APODid this help you?
- 3triggering full purges only on specific events like updates rather than running warm repeatedlyDid this help you?
In their words
Unedited, grouped by where they were said, most upvoted first within each place, each linked to the thread it came from.
“solid approach, but watch your server load if the sites grow. warming every 30 min is fine for smaller sitemaps but can hammer your origin if you're hitting hundreds of URLs in parallel.”source ↗
“Don’t curl your entire site every 30 minutes - that will kill your server with a decent number of pages.”source ↗
Where this came up
People with this problem also raised
- 16How to stop excessive bot traffic and AI scrapers
- 9How to stop software updates from breaking your production website
- 10Why does a WordPress site need a plugin for everything?
- 5Why do gas stations put huge temporary holds on my card?
- 3How to stop back sweating while dancing at my wedding
- 7Why did page load times suddenly spike to 20+ seconds?