Update:
After I didn’t have a solution yesterday and this morning, I’ve come a little closer to the matter. Apparently Wordfence is building a blocking list for different IPs during the tests. For example because of “non-existent page” or “blocked for Exceeded the maximum number of page not found errors per minute for humans”. That was not the case before.
Plugin Support
WFAdam
(@wfadam)
Hello @detoris and thanks for reaching out to us!
If these Lighthouse requests are being blocked by Wordfence, you should see these requests in the Tools > Live Traffic page. Are you seeing any blocks here? If so, you could whitelist them directly from here.
Let me know what you find!
Thanks!
Thank you for the feedback.
Yes, I see the blocking in live traffic. From there, however, I can only execute unblock directly and not include it in a whitelist. As far as I know, I can only manage and feed different entries in the various whitelists in the options.
Can I feed a whitelist directly from the live traffic page?
Or in which whitelist would I have to make which entries (Options / Whitelisted IP addresses that bypass all rules?)?
Thanks
Plugin Support
WFAdam
(@wfadam)
You could add all of Lighthouse’s IPs to the Whitelisted IP addresses that bypass all rules if the option to whitelist is not available in the Live Traffic.
I was browsing Pagespeed support and I read a post about there not being a set group of IPs that they use, so whitelisting these might be hard. Unless you have the information I don’t, in that case, I would love for you to share!
Thanks again!
Summary of my results:
Due to the different tests by PageSpeed Insights with and without cache etc. PageSpeed Insights wants to have access to many files with a shortened invalid path at once. Due to the number of failed accesses (tried to access a non-existent page) in a very short time, Wordfence blocks the access of PageSpeed Insights according to my narrowly set rule. Hence the above error message.
After the observations in Live Traffic, I added 3 IPs to the whitelist:
66.249.93.220
66.249.93.221
66.249.93.222
After that, PageSpeed Insights can run the tests again without any problems. Alternatively, I could have extended the rule, but that would have been too uncertain for me.
For information, the complete information about the blocking again
IP: 66.249.93.222 Hostname: google-proxy-66-249-93-222.google.com
Human/Bot: Bot
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4143.7 Safari/537.36 Chrome-Lighthouse
IP: 66.249.93.221 Hostname: google-proxy-66-249-93-221.google.com
Human/Bot: Bot
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4143.7 Safari/537.36 Chrome-Lighthouse
IP: 66.249.93.220 Hostname: google-proxy-66-249-93-220.google.com
Human/Bot: Bot
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4143.7 Safari/537.36 Chrome-Lighthouse
I don’t know why PageSpeed Insights shows this strange behavior and searches for non-existent pages, but I can now manage the behavior in Wordfence.
So I put this thread on resolved.
Thanks for your support
Plugin Support
WFAdam
(@wfadam)
Thanks @detoris for the information!
Glad you could find a resolution to this!
Let us know if you need anything else!
Thanks again for your support!
I just submitted my new website in Google Pagespeed Insights for a website speed audit. But it is returning an error as follows:
Lighthouse returned error: FAILED_DOCUMENT_REQUEST. Lighthouse was unable to reliably load the page you requested. Make sure you are testing the correct URL and that the server is properly responding to all requests. (Details: net::ERR_FAILED).
The website https://blueheronhealthnewsreviews.com/ absolutely has no issue & load perfectly at present but still returning the above error.
Anybody having idea to fix the issue?
