Title: Crawler setup
Last modified: April 25, 2018

---

# Crawler setup

 *  Resolved [Manuel22](https://wordpress.org/support/users/manuel22/)
 * (@manuel22)
 * [8 years, 1 month ago](https://wordpress.org/support/topic/crawler-setup/)
 * Hello,
    I searched yesterday all day long so I’m quite desperate:
 * I set up a new VM with Ubuntu 16.04 and installed OpenLitespeed and WordPress
   as dsecribed here: [http://open.litespeedtech.com/mediawiki/index.php/Help:1-Click_Install](http://open.litespeedtech.com/mediawiki/index.php/Help:1-Click_Install)(
   Didn’t change locations)
 * So far so good. Caching works when I go to the page manually, hit refresh and
   check the response headers and the sourcecode.
 * But I don’t get the crawler to preload the sites. After the crawler finished 
   its run, the response headers give me:
 *     ```
       Server: LiteSpeed
       Transfer-Encoding: chunked
       X-Litespeed-Cache-Control: public,max-age=600
       X-Litespeed-Tag: 7d51_PGSRP,7d51_URL.6666cd76f96956469e7be39d750cc7d9,7d51_F,7d51_Po.904,7d51_PGS,7d51_
       ```
   
 * The crawler itself runs as far as I can see in the “Watch Crawler Status” window.
 * I assume the problem is [enabeling the crawler](https://www.litespeedtech.com/support/wiki/doku.php/litespeed_wiki:cache:lscwp:configuration:enabling_the_crawler)
   with the crawler snippet:
 *     ```
       <IfModule Litespeed>
        CacheEngine on crawler
       </IfModule>
       ```
   
 * As I don’t use any control panel, the wiki tells me to “Add the Crawler Snippet
   and optional server variables to your server-level Apache config file”
 * **But how does this file look like? Where can I find it? **I googled all day 
   long yesterday but obviously didn’t find the right file. So any hint is welcome.
   
   Thanks for any help 😉
    -  This topic was modified 8 years, 1 month ago by [Manuel22](https://wordpress.org/support/users/manuel22/).

Viewing 2 replies - 1 through 2 (of 2 total)

 *  Plugin Support [qtwrk](https://wordpress.org/support/users/qtwrk/)
 * (@qtwrk)
 * [8 years, 1 month ago](https://wordpress.org/support/topic/crawler-setup/#post-10223222)
 * Hi,
 * Apologize for late response , we are checking this, and will get to you as soon
   as possible.
 * Best regards,
 *  Plugin Support [qtwrk](https://wordpress.org/support/users/qtwrk/)
 * (@qtwrk)
 * [8 years ago](https://wordpress.org/support/topic/crawler-setup/#post-10250669)
 * Hi,
 * Sorry for late response.
 * I just installed an OLS and tested , it seems enable crwaler cron doesn’t require
   additional step , the one you see , is for LiteSpeed enterprise.
 * by the way , the default role for crawler is guest user , so if you visit it 
   by login user (usually by admin user) you will not see crawled page.
 * if you wanna check this , you can try open a private tab in browser and access
   from there.
 * Please let us know if you have any further questions.
 * Best regards,

Viewing 2 replies - 1 through 2 (of 2 total)

The topic ‘Crawler setup’ is closed to new replies.

 * ![](https://ps.w.org/litespeed-cache/assets/icon-256x256.png?rev=2554181)
 * [LiteSpeed Cache](https://wordpress.org/plugins/litespeed-cache/)
 * [Frequently Asked Questions](https://wordpress.org/plugins/litespeed-cache/#faq)
 * [Support Threads](https://wordpress.org/support/plugin/litespeed-cache/)
 * [Active Topics](https://wordpress.org/support/plugin/litespeed-cache/active/)
 * [Unresolved Topics](https://wordpress.org/support/plugin/litespeed-cache/unresolved/)
 * [Reviews](https://wordpress.org/support/plugin/litespeed-cache/reviews/)

 * 2 replies
 * 2 participants
 * Last reply from: [qtwrk](https://wordpress.org/support/users/qtwrk/)
 * Last activity: [8 years ago](https://wordpress.org/support/topic/crawler-setup/#post-10250669)
 * Status: resolved