Unblocking blocked bots

  • Resolved rickysmithy

    (@rickysmithy)


    8 years, 3 months ago

    I am trying to unblock some blocked bots in the live traffic view and it does not seem to work.

    I have the option to block this bot (crawl-66-249-79-19.googlebot.com) however it is already blocked (type:blocked) in the GUI. And it returns a 503 response to the bot, which I think with interfere with page indexing.

    How can I bulk unblock these bots?

    Thank you

Viewing 6 replies - 1 through 6 (of 6 total)
  • mountainguy2

    (@mountainguy2)

    8 years, 3 months ago

    According to this https://support.google.com/webmasters/answer/80553?hl=en you are indeed blocking a google crawl. There are a couple of things to do. Sadly, I have no idea where these options are in the recent UI downgrade, perhaps you have time to find them. For starters, there used to be Wordfence option to allow verified Google crawlers. Perhaps that’s still in there somewhere, but without a search function on the Wordfence UI (a patently ridiculous omission for a UI this broken up) who knows how long this will take to find. Perhaps our human Wordfence search engine folks will chime in and let us know. In the meantime, try to ascertain why the crawler was blocked, I suppose by clicking on the block in Live Traffic. If blocked due to rate limiting and you want to leave your rate limiting as is, set your Google crawl rate slower. I believe this can be done simply in your robots.txt file, but can also be done in Webmaster Tools. http://support.hostgator.com/articles/telling-google-how-often-to-crawl-your-website

    MTN

    Thread Starter rickysmithy

    (@rickysmithy)

    8 years, 3 months ago

    Hi there, I did manage to bulk unblock the google crawl bots, in the firewall tab and under blocking. However the unblocking seemed to take a few hours to enact.
    I could not find the allow verified google bots option, would be great if that was accessible…
    I do agree the UI is not the best to navigate, the recent update seems to have made this worse

    mountainguy2

    (@mountainguy2)

    8 years, 3 months ago

    The Google bot stuff used to be under “Rate Limiting Rules” on the “Options” page, Ver 6. If you find the mysterious beast let us know. I’m busy actually making content rather than learning new pointless UI. MTN

    Thread Starter rickysmithy

    (@rickysmithy)

    8 years, 3 months ago

    Its now under ‘all firewall options’ which is under the firewall tab.

    mountainguy2

    (@mountainguy2)

    8 years, 3 months ago

    Thanks!

    wfalaa

    (@wfalaa)

    8 years, 3 months ago

    Hi @rickysmithy
    I hope you tried the new “Search All Options” feature in “All Options” page in the recent version 7.0.4, this should be helpful in findings other options locations.

    Thanks.

Viewing 6 replies - 1 through 6 (of 6 total)

The topic ‘Unblocking blocked bots’ is closed to new replies.