• Clear Talk Anti-Spam was all I needed in conjunction with Imunify 360 to keep servers stable and free of abuse.

    This past year has had me fighting against all of these AI bots and scrapers that use hundreds and even thousands of different IP addresses to continually scrape every page of every website. They are frequently from the same network using the first two octets.

    I’ve had to write plugins that can track and detect abusive IP addresses up the chain and eventually block their entire IP address block of the most abusive networks (frequently Google Cloud, Microsoft Cloud and AWS server traffic) just to keep my sites available to real visitors. These bots will otherwise eat up every available connection slot and grind a site to a halt.

    It’d be a 5 star rating, and was a 5 star in my book, but this year it is only a 4 star because these abusive bots and scrapers need to be addressed in 2026.

Viewing 3 replies - 1 through 3 (of 3 total)
  • Plugin Support amagsumov

    (@amagsumov)

    Hello @webdesignxtc,

    Thank you for your review!

    Regarding the bot activity you’re seeing, I’d like to highlight that the Anti-Spam plugin includes two features designed to handle these types of crawlers: Anti-Crawler and Anti-Flood: https://cleantalk.org/help/anti-flood-and-anti-crawler

    Additionally, you can use the Anti-Crawler settings to block bots by their User-Agent: https://cleantalk.org/help/filter-ua

    If you have any questions or need help fine-tuning these settings for your specific site, please feel free to open a ticket in our support system or email us directly at [email protected].

    • This reply was modified 2 weeks, 3 days ago by amagsumov.
    Thread Starter Brian Bateman

    (@webdesignxtc)

    I will contact support to see if I can somehow figure out which User-Agents I could block, but usually these guys are sending false user agents to disguise themselves as chrome on windows or something similar.

    I’ve had Anti-Crawler on when I’ve seen 1000+ users from a small town in Iowa all within 30 minutes, all from rotating IP addresses on Google Cloud. Anti-Crawler was missing them because they learn to hit the site maybe 5 times with an IP before rotating to the next IP.

    I was able to write a plugin tha watches the entire IP address range and progressively go from giving a 403 to an IP for a short time, increasing the length on subsequent violations and then if it sees traffic from the same IP range violating the rules, then it starts blocking the range for increasingly longer times.

    I’ll share it with support.

    Plugin Support amagsumov

    (@amagsumov)

    Thank you for the additional details.

    We’ve recieved your email. let’s continue our troubleshooting within that email thread.

    Thank you.

    • This reply was modified 2 weeks, 3 days ago by amagsumov.
Viewing 3 replies - 1 through 3 (of 3 total)

You must be logged in to reply to this review.