rate limiting googlebot?

Hey James,

I’m sorry about the delay in getting back to you, and thank you for your patience while I was out last week. When Wordfence throttles Googlebot, it returns a 503. As you said, Googlebot should understand this response and slow its crawl rate.

Can you send me your access logs from around the time the site is being crawled too heavily, please? Your host should be able to provide those if you don’t regularly have access to them. You can email them to wftest @ wordfence . com. Please use your forum username as the subject and let me know here once you’ve sent those!

Thanks,
Margaret

Hey James,

Thanks for sending the logs over and for the well wishes! I was out for positive reasons, thankfully!

In the section you highlighted, the IP address 34.174.*** accessed the site successfully only 2 times before the 508 response started. Since a 508 response indicates the server has run out of resources, it’s sent by the server itself before the request reaches Wordfence, which is why the rate limiting didn’t kick in. The hostname on the IP is *.bc.googleusercontent.com. This is often confused for Googlebot, but it’s actually the hostname used by Google-hosted cloud servers for users, rather than being used by Google itself.

Checking through the access logs, I also see a few times where there are bots spoofing the user agent Googlebot. For example, on 9/25 at 5:30 am, the IP address 79.127.*** uses the Googlebot user agent, but is not a legitimate Googlebot. I see they had approximately 60 accesses over the course of 2 minutes before the server started to return 508 responses. This would not have been throttled by the rate-limiting settings in the original diagnostics I received.

If you’d like to verify legitimate Googlebot traffic in the future, you can use reverse DNS lookup or check against Google’s published IP ranges. Google has some solid details on that here: https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot

If you often see a variety of IP addresses before the resource limits have been hit, it might be worth using a service with DDoS protection, such as Cloudflare, to add an extra layer of security to the site.

If your server’s resource limits are fairly low, you might consider setting the rate limits as low as 60 requests per minute, but please keep an eye on your Live Traffic log to ensure that regular visitors aren’t being blocked.

It may also be a good idea to check with your host. They should be able to help review for any potential issues leading to higher-than-normal resource usage.

Please let me know if you have any questions!

Thanks,
Margaret

Hey JB,

Since one of the bots is spoofing the Googlebot user agent, I suspect they wouldn’t be well-behaved on the site. It did look like, at several points, the 508 errors started very early into the bots’ access attempts, so it could be time to move to hosting with more resources available.

We don’t normally recommend permanent blocks, as IP addresses can change pretty often, but if you see the same IP addresses attacking the site over and over, it might be worthwhile in your case to permanently block the biggest offenders. Since your firewall was optimized, they’d be served the block page before any other plugins load, which would help you save server resources.

I’m going to go ahead and resolve this ticket, but please let me know here if you have any other questions, and I’ll be happy to help!

Thanks,
Margaret