Rate Limiting settings

  • Resolved Andrea

    (@licoreo)


    6 months, 1 week ago

    I’m writing to ask for clarification about the Rate Limiting settings in the free version of Wordfence.

    On our site, we would like to set the limit to 60 requests per minute with the “Block it” action.
    We also plan to set “How should we treat Google’s crawlers” to “Verified Google crawlers will not be rate-limited”.

    Before applying these settings, we want to be sure that they will not cause any issues for other legitimate crawlers such as the ones used by OpenAI (ChatGPT) or Bing.

    Could you please confirm whether these crawlers might be blocked or rate-limited under this configuration?
    Any guidance or best practices would be greatly appreciated.

    Thank you for your help.

Viewing 1 replies (of 1 total)
  • Plugin Support wfpeter

    (@wfpeter)

    6 months, 1 week ago

    Hi @licoreo, thanks for getting in touch with us!

    On a new installation, I generally set my Rate Limiting rules to these values to start with:
    Rate Limiting Screenshot

    • Verified Google crawlers will not be rate-limited
    • If anyone’s requests exceed – 240 per minute
    • If a crawler’s page views exceed – 120 per minute
    • If a crawler’s pages not found (404s) exceed – 60 per minute
    • If a human’s page views exceed – 120 per minute
    • If a human’s pages not found (404s) exceed – 60 per minute
    • How long is an IP address blocked when it breaks a rule – 30 minutes

    I also always recommend setting the rule to “throttle it” instead of “block it”. Throttling is generally better than blocking because any good search engine understands what happened if it is mistakenly blocked and your site won’t be penalized.

    Make sure and set your Rate Limiting Rules realistically and set the value for how long an IP is blocked to 30 minutes or more.

    You can read about our Rate Limiting settings in our help documentation if you need to know a little more. Remember there is no hard and fast, one-size-fits-all set of rules for every site. This is just a good place to start. During an attack you may want to make those rules stricter. If you do see search engine/AI crawlers getting blocked too often, you might want to loosen them up a little.

    Many thanks,
    Peter.

Viewing 1 replies (of 1 total)

The topic ‘Rate Limiting settings’ is closed to new replies.