Title: Allow Heritrix crawler
Last modified: November 7, 2022

---

# Allow Heritrix crawler

 *  Resolved [Sjors](https://wordpress.org/support/users/sjors-roelofs/)
 * (@sjors-roelofs)
 * [3 years, 5 months ago](https://wordpress.org/support/topic/allow-heritrix-crawler/)
 * Hi!
 * I have a site that needs to be crawled once a day. The crawler for this site 
   is Heritrix. Currently, the crawl fails because iThemes Security blocks the Heritrix
   user agent by a line in the htaccess file.
 * If I remove this line, it gets automatically added again after a while.
 * Is there a way to always allow this crawler/user-agent to crawl the website, 
   without completely disabling iThemes Security? I cannot find any settings within
   the plugin settings to allow this, only to block certain user agents.
 * Thanks!

Viewing 2 replies - 1 through 2 (of 2 total)

 *  [nlpro](https://wordpress.org/support/users/nlpro/)
 * (@nlpro)
 * [3 years, 5 months ago](https://wordpress.org/support/topic/allow-heritrix-crawler/#post-16176109)
 * Hee Sjors,
 * Navigate to:
 * Security > Settings > CONFIGURE > Lockouts > Ban Users
 * and disable the **Default Ban List** setting.
 * +++++ To prevent any confusion, I’m not iThemes +++++
 *  Plugin Support [chandelierrr](https://wordpress.org/support/users/shanedelierrr/)
 * (@shanedelierrr)
 * [3 years, 5 months ago](https://wordpress.org/support/topic/allow-heritrix-crawler/#post-16196385)
 * Hi [@sjors-roelofs](https://wordpress.org/support/users/sjors-roelofs/), I hope
   the information provided by nlpro helped resolve the issue. Since we haven’t 
   received a response, I’ll mark this post as resolved. If you still need some 
   assistance, feel free to open a new support topic, and we’d be happy to assist.
   Thank you!

Viewing 2 replies - 1 through 2 (of 2 total)

The topic ‘Allow Heritrix crawler’ is closed to new replies.

 * ![](https://ps.w.org/better-wp-security/assets/icon.svg?rev=2980272)
 * [Solid Security – Password, Two Factor Authentication, and Brute Force Protection](https://wordpress.org/plugins/better-wp-security/)
 * [Frequently Asked Questions](https://wordpress.org/plugins/better-wp-security/#faq)
 * [Support Threads](https://wordpress.org/support/plugin/better-wp-security/)
 * [Active Topics](https://wordpress.org/support/plugin/better-wp-security/active/)
 * [Unresolved Topics](https://wordpress.org/support/plugin/better-wp-security/unresolved/)
 * [Reviews](https://wordpress.org/support/plugin/better-wp-security/reviews/)

## Tags

 * [crawler](https://wordpress.org/support/topic-tag/crawler/)
 * [user-agent](https://wordpress.org/support/topic-tag/user-agent/)

 * 2 replies
 * 3 participants
 * Last reply from: [chandelierrr](https://wordpress.org/support/users/shanedelierrr/)
 * Last activity: [3 years, 5 months ago](https://wordpress.org/support/topic/allow-heritrix-crawler/#post-16196385)
 * Status: resolved