Google Can't Access Site – Robot.txt Error

  • sdatrny

    (@sdatrny)


    11 years, 3 months ago

    I began receiving error messages from Google letting me know it stopped indexing my site since it was having trouble accessing my Robot.txt file. After much ado with tech support with my host, the problem was ultimately narrowed down to the Wordfence plugin. When it’s active, Google can’t access my site. As soon as I deactivate it, all is well. Please correct this, as I’d really like to continue using the plugin.

    https://ww.wp.xz.cn/plugins/wordfence/

Viewing 5 replies - 1 through 5 (of 5 total)
  • WFBrian

    (@wfbrian)

    11 years, 3 months ago

    Hi,

    Are you using Cloudflare?

    Brian

    Thread Starter sdatrny

    (@sdatrny)

    11 years, 3 months ago

    Hi,
    No, I am not, just Wordfence.

    WFBrian

    (@wfbrian)

    11 years, 3 months ago

    How do you have “How should we treat Google’s crawlers” in Wordfence Options? Try the anyone claiming to be Google option as that is the most lenient Google setting. If that doesn’t help, let me know and I’ll do some more digging with the development team.

    -Brian

    Thread Starter sdatrny

    (@sdatrny)

    11 years, 3 months ago

    Hi Brian,

    I tried the “Anyone claiming to be Google” option in the settings that you suggested. The Robot.txt test in Google Webmaster Tools seemed to be operating OK today, but the “Google Fetch” function results in a “Temporarily Unavailable” message. When I leave the “How to Treat Google” setting unchanged (full blown), the Fetch message yields “Unavailable” (without the “temporarily”). Both the Robot.txt test and Goggle Fetch work fine if I disable Wordfence.

    Thanks.

    WFBrian

    (@wfbrian)

    11 years, 3 months ago

    OK. Thanks for testing. I’ll pass this off and have it looked into.

    -Brian

    (Ref FB684)

Viewing 5 replies - 1 through 5 (of 5 total)

The topic ‘Google Can't Access Site – Robot.txt Error’ is closed to new replies.