Hi @ildomandatore, thanks for your question.
Wordfence will usually treat verified crawlers that are known to observe robots.txt like GoogleBot, Bing etc. favourably, unless your own Rate Limiting settings are quite strict. Requests can be made to look at face-value like legitimate bots when they’re actually from other IPs or origins, so there is a chance Wordfence is blocking these based on its list of bad IPs, referers etc.
Just ensure your rules already don’t limit Verified Google crawlers, and throttle instead of block. Throttling is generally better than blocking with crawlers because any good search engine understands what has happened if it is mistakenly blocked and your site isn’t penalized because of it. You can see my Rate Limiting settings if you click that link.
I hope that helps you out!
Peter.
I’ve started a reset of Wordfece and used WordPress Assistent plugin. I have done a migration 10 months ago and all was stopped to crawl. I hope have solved every trouble. I’ll back if have still troubles.
Thanks, I appreciate the update and hope that resolves any issues you were seeing.
Feel free to start a new topic any time highlighting steps you’ve already tried (such as already resetting/reinstalling the plugin, config changes, etc.) so that we look into any issues that remain.
Peter.
I still have problems. I found a url blocked by you.
View post on imgur.com
Hi @ildomandatore,
That IP doesn’t match any of Googlebot’s IP ranges, which don’t begin with 118 in their records. The IP resolves to an ISP in Vietnam currently rather than a bot from another provider too by the looks of things.
Thanks,
Peter.
