AI scraper bots: block all

Hi Justin,

The [?] next to the “Block AI crawlers?” setting links to this page: https://kb.theseoframework.com/kb/robots-txt-blocks/#blocking-ai-crawlers.

At the top of that page, I highlighted important information:

Robots.txt is not a security measure, and it can’t be used to prevent access to your website. Bots can choose to ignore robots.txt, and some bots don’t even look for it at all.

Blocking bots is something a WordPress plugin cannot—or rather, should not—do. TSF only steers them using recognized protocols.

So, if your site struggles with bot traffic, consider talking to your hosting provider. They can help you implement measures to limit or block bot traffic outright. Still, no WordPress site should struggle with bot traffic if the server is properly configured.