Not blocking legitimate bots

  • Resolved WordPressFan

    (@micoots)


    7 years, 6 months ago

    Hi. I’ve just started testing this plugin and have found that when Google bot generates 404’s it gets blocked by the plugin.

    The following should never ever be blocked:

    AOL.com
    Baidu
    Bingbot/MSN
    DuckDuckGo
    Googlebot
    Teoma
    Yahoo!
    Yandex

    So there’s got to be a way to have an inbuilt whitelist for the above, so even when monitoring 404’s or anything else, the above don’t get blocked.

    Can you please implement or advise how we can implement easily?

    When adding Google bot manually to the whitelist:

    https://evert.meulie.net/faqwd/googlebot-ip-ranges/

    there should also be a way to add CIDR’s:

    64.18.0.0/20

    instead of ranges:

    64.18.0.0 – 64.18.15.255

    It will just make it much easier.

    A better list looks to be here:

    https://www.techarp.com/guides/search-engine-robot-ip-addresses/

    Thanks.

    • This topic was modified 7 years, 6 months ago by WordPressFan.
    • This topic was modified 7 years, 6 months ago by WordPressFan.
Viewing 2 replies - 1 through 2 (of 2 total)
  • Plugin Author Jeff Sterup

    (@foomagoo)

    7 years, 6 months ago

    You just need to add their IP ranges. A CIDR is an IP range. Math is done on the IP ranges to determine if the visitor is within a range. So CIDR notation wont work for that. You just need to convert the CIDR to an IP range. I have known bots on my Whitelist. I also have some known bots on my blacklist. So it is up to the administrator of the site whether they want the bots to be blocked or not. Baidu is never allowed on any of my sites. With a little research you can figure out the IP ranges currently used by the bots you want to whitelist.

    Thread Starter WordPressFan

    (@micoots)

    7 years, 6 months ago

    Hi Jeff. Yes thanks, I found this site:

    https://www.ip2location.com/free/robot-whitelist

    to have current addresses but they are a lot 😉

    I have turned off the 404 check features for now, since it’s too much manual overhead at the moment to get that part working automatically.

    Michael.

Viewing 2 replies - 1 through 2 (of 2 total)

The topic ‘Not blocking legitimate bots’ is closed to new replies.

Tags