Allow only Googlebots and Bingbots?

  • Resolved youaresam2

    (@youaresam2)


    4 years, 1 month ago

    I am using your plugin IP Location Block.

    I was constantly in pain because of various attacks from foreign countries.
    But all my problems have been solved since using your plugin.

    Truly the best plugin!

    But there are some problems.
    I want to allow bingbot, but I don’t want to allow other strange bots.
    But I can’t figure out how to allow only bingbot.

    The only option I found regarding bingbot is bot:HOST .
    I would be very grateful if you could tell me how to fix this.

    Once again, thank you very much for making such a good program.

    ps.
    First of all I can’t find the option to automatically disable the bot on the plugin introduction page
    I almost missed the opportunity to use this great plugin.
    It would be nice if the plugin introduction page could be slightly modified.
    Also, it would be good to treat bingbot (ex: msnbot-157-55-39-43.search.msn.com) separately like Google.

Viewing 3 replies - 1 through 3 (of 3 total)
  • Plugin Author Darko G.

    (@darkog)

    4 years, 1 month ago

    Hey @youaresam2

    What string you tried for blocking Bingbot so far?

    I suggest to try the following:

    1. Faster, but less checks:

    bingbot/2.0:*
    or simply:
    bingbot:*

    2. Better but slower, good thing the dns lookup is cached so it shouldn’t be a huge problem

    bingbot/2.0:HOST=msnbot
    or simply:
    bingbot:HOST=msnbot

    Let me know if that works.

    You can find more information here:
    https://iplocationblock.com/codex/ua-string-and-qualification/

    Bingbot user agents can be found here:
    https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0

    I am also doing some tests to verify this functionality.

    Best,
    Darko

    • This reply was modified 4 years, 1 month ago by Darko G..
    • This reply was modified 4 years, 1 month ago by Darko G..
    • This reply was modified 4 years, 1 month ago by Darko G..
    Plugin Author Darko G.

    (@darkog)

    4 years, 1 month ago

    Hey,

    I just tested the default conditions and it looks like this condition bot:HOST allows any other crawlers including googlebot and bingbot.

    Default conditions:

    Google:HOST,bot:HOST,slurp:HOST
spider:HOST,archive:HOST,*:FEED
embed.ly:HOST,Twitterbot:US,Facebot:US

    So if you only want Googlebot and Bingbot, best fit will be the following:

    Google:HOST,bingbot/2.0:*,archive:HOST,*:FEED,Twitterbot:US,Facebot:US

    This will allow any crawlers from Google, the Bingbot labeled one, Wayback Archive, Twitter and Facebook.

    If you just want strictly Googlebot and Bingbot, do the following:

    Googlebot:HOST=google,bingbot/2.0:HOST=msnbot

    Other notes:

    • slurp is Yahoo

    Best Regards,
    Darko

    • This reply was modified 4 years, 1 month ago by Darko G..
    Thread Starter youaresam2

    (@youaresam2)

    4 years ago

    Thanks for the great and kind reply.

Viewing 3 replies - 1 through 3 (of 3 total)

The topic ‘Allow only Googlebots and Bingbots?’ is closed to new replies.