Title: Multisite crawler issue Last modified: July 26, 2023 --- # Multisite crawler issue * Resolved [dev](https://wordpress.org/support/users/devksec/) * (@devksec) * [2 years, 10 months ago](https://wordpress.org/support/topic/multisite-crawler-issue/) * Hello, * We’re on A2 hosting and they’ve been unable to get the crawler working on subsites. I’ve put a workaround of having the subsites sitemap URLS in the primary site which works, however the crawler isn’t running automatically and is stuck “Start watching…” * When run manually sometimes it does run without an issue run but others it gets a “stopped_reset”. It does seem to often be stuck in position 1, despite crawlers 1-4 being disabled (we don’t use the litespeed webp ones). The last interval doesn’t update on its own but CRON appears to show the “litespeed_task_crawler” next run time to be changing as expected. * ```wp-block-code Start watching... 25 Jul 2023 02:01:47 Size: 853 Crawler: #5 Position: 1 Threads: 3 Status: crawling, prepare running 26 Jul 2023 10:57:28 Size: 853 Crawler: #1 Position: 1 Threads: 3 Status: stopped_reset 26 Jul 2023 10:57:28 Size: 853 Crawler: #5 Position: 1 Threads: 3 Status: crawling, prepare running 26 Jul 2023 11:00:29 Size: 853 Crawler: #5 Position: 10 Threads: 3 Status: crawling, updated position 26 Jul 2023 11:00:43 Size: 853 Crawler: #5 Position: 16 Threads: 3 Status: crawling, updated position 26 Jul 2023 11:00:58 Size: 853 Crawler: #5 Position: 25 Threads: 3 Status: crawling, updated position 26 Jul 2023 11:01:13 Size: 853 Crawler: #5 Position: 34 Threads: 3 Status: crawling, updated position 26 Jul 2023 11:01:26 Size: 853 Crawler: #5 Position: 46 Threads: 3 Status: crawling, updated position 26 Jul 2023 11:01:37 Size: 853 Crawler: #5 Position: 58 Threads: 3 Status: crawling, updated position 26 Jul 2023 11:01:48 Size: 853 Crawler: #5 Position: 70 Threads: 3 Status: crawling, updated position 26 Jul 2023 11:02:01 Size: 853 Crawler: #5 Position: 85 Threads: 3 Status: crawling, updated position 26 Jul 2023 11:02:12 Size: 853 Crawler: #5 Position: 97 Threads: 3 Status: crawling, updated position 26 Jul 2023 11:02:23 Size: 853 Crawler: #5 Position: 109 Threads: 3 Status: crawling, updated position 26 Jul 2023 11:02:36 Size: 853 Crawler: #5 Position: 124 Threads: 3 Status: crawling, updated position 26 Jul 2023 11:02:47 Size: 853 Crawler: #5 Position: 136 Threads: 3 Status: crawling, updated position 26 Jul 2023 11:02:58 Size: 853 Crawler: #5 Position: 148 Threads: 3 Status: crawling, updated position 26 Jul 2023 11:03:09 Size: 853 Crawler: #5 Position: 160 Threads: 3 Status: crawling, updated position ``` * Reviewing the reset logs, there are some API queries for third-party integrations but unsure if any of these are the issue. Here are some examples * ```wp-block-code 07/24/23 14:35:22.541 [34.243.223.231:54066 4 xe8] 💓 ------POST HTTP/1.1 (HTTPS) / 07/24/23 14:35:22.541 [34.243.223.231:54066 4 xe8] Query String: wc-api=wc_shipstation&auth_key=REDACTED 07/24/23 14:35:22.541 [34.243.223.231:54066 4 xe8] User Agent: RestSharp/106.3.1.0 07/24/23 14:35:22.541 [34.243.223.231:54066 4 xe8] Accept: application/json, application/xml, text/json, text/x-json, text/javascript, text/xml 07/24/23 14:35:22.541 [34.243.223.231:54066 4 xe8] Accept Encoding: gzip 07/24/23 14:35:22.541 [34.243.223.231:54066 4 xe8] X-LSCACHE: true 07/24/23 14:35:22.541 [34.243.223.231:54066 4 xe8] X-LiteSpeed-Purge: public,1344_Po.25407,1344_W.recent-posts-4,1344_FD,1344_A.0,1344_F,1344_H,1344_PGS,1344_PGSRP,1344_D.202307,1344_REST => LiteSpeed\LSC->send_headers()@616 => WP_Hook->apply_filters(,ARRAY)@308 => WP_Hook->do_action(ARRAY)@332 => /home/ksc5577p/REDACTED.co.uk/wp-includes/load.php@517 07/24/23 12:40:32.965 [=CLI=ksc5577p 3 4BO] X-LiteSpeed-Purge: public,1343_Po.24879,1343_URL./,1343_W.recent-posts-4,1343_FD,1343_A.1,1343_F,1343_H,1343_PGS,1343_PGSRP,1343_D.202307,1343_REST => LiteSpeed\LSC->send_headers()@616 => WP_Hook->apply_filters(,ARRAY)@308 => WP_Hook->do_action(ARRAY)@332 => /home/ksc5577p/REDACTED.ksec.co.uk/wp-includes/load.php@517 07/24/23 12:33:46.651 [34.241.67.207:56998 3 j1l] 💓 ------GET HTTP/1.1 (HTTPS) / 07/24/23 12:33:46.651 [34.241.67.207:56998 3 j1l] Query String: wc-api=wc_shipstation&REDACTED 07/24/23 12:33:46.651 [34.241.67.207:56998 3 j1l] User Agent: ShipStation 07/24/23 12:33:46.651 [34.241.67.207:56998 3 j1l] Accept: 07/24/23 12:33:46.651 [34.241.67.207:56998 3 j1l] Accept Encoding: gzip 07/24/23 12:33:46.651 [34.241.67.207:56998 3 j1l] X-LSCACHE: true 07/24/23 12:33:46.652 [34.241.67.207:56998 3 j1l] X-LiteSpeed-Purge: public,1343_Po.24878,1343_W.recent-posts-4,1343_FD,1343_A.2629,1343_F,1343_H,1343_PGS,1343_PGSRP,1343_D.202307,1343_REST => LiteSpeed\LSC->send_headers()@616 => WP_Hook->apply_filters(,ARRAY)@308 => WP_Hook->do_action(ARRAY)@332 => /home/ksc5577p/REDACTED.co.uk/wp-includes/load.php@517 07/24/23 11:24:14.569 [185.62.138.109:33528 5 RXe] 💓 ------POST HTTP/1.1 (HTTPS) /wp-json/wc/v2/stock-sync-batch 07/24/23 11:24:14.569 [185.62.138.109:33528 5 RXe] Query String: consumer_key=REDACTED 07/24/23 11:24:14.569 [185.62.138.109:33528 5 RXe] User Agent: WooCommerce API Client-PHP/2.0.1 07/24/23 11:24:14.569 [185.62.138.109:33528 5 RXe] Accept: application/json 07/24/23 11:24:14.569 [185.62.138.109:33528 5 RXe] Accept Encoding: gzip 07/24/23 11:24:14.569 [185.62.138.109:33528 5 RXe] X-LSCACHE: true 07/24/23 11:24:14.569 [185.62.138.109:33528 5 RXe] X-LiteSpeed-Purge: public,1345_Po.162,1345_URL./product/REDACTED/,1345_W.recent-posts-1,1345_FD,1345_A.451,1345_F,1345_H,1345_PGS,1345_PGSRP,1345_D.202207,1345_REST,1345_Po.158,1345_T.4,1345_T.19,1345_T.52,1345_T.43,1345_T.23,1345_T.47,1345_T.28,1345_T.29,1345_T.51,1345_T.30,1345_T.31,1345_PT.product,1345_product => LiteSpeed\LSC->send_headers()@616 => WP_Hook->apply_filters(,ARRAY)@308 => WP_Hook->do_action(ARRAY)@332 => /home/ksc5577p/REDACTED.co.uk/wp-includes/load.php@517 ``` * Here is our Report number: **XLANDMHU** * We’re happy to work with the primary site being the crawler or to try resolve the subsite to crawl instead but unsure why its not automatically running. Viewing 11 replies - 1 through 11 (of 11 total) * Plugin Support [qtwrk](https://wordpress.org/support/users/qtwrk/) * (@qtwrk) * [2 years, 10 months ago](https://wordpress.org/support/topic/multisite-crawler-issue/#post-16925785) * if manual run works , then it’s most likely cron issue * please make sure your wp cron is always triggered in time , and I would suggest to set up a system cron to trigger it. * Thread Starter [dev](https://wordpress.org/support/users/devksec/) * (@devksec) * [2 years, 10 months ago](https://wordpress.org/support/topic/multisite-crawler-issue/#post-16925854) * Hello, * Our cron is working as expected and is a system cron. Is there a specific one we can run just for the crawler? WP Crons shows the crawler being scheduled as expected. We thought it could just be being reset rather than a cron issue as it wordpress cron doesn’t have any other issues. It runs every 10 minutes for the site. The primary management site is often in maintenance mode, subsites are not, would this disable the crawler? * Plugin Support [qtwrk](https://wordpress.org/support/users/qtwrk/) * (@qtwrk) * [2 years, 10 months ago](https://wordpress.org/support/topic/multisite-crawler-issue/#post-16925865) * please go to crawler setting , set crawler interval to 61 , then in sitemap setting, drop domain set to OFF , and make sure your server load is within limit * after you set interval to 61 , ideally , the crawler should start itself every 61 seconds if not running , give it an hour or two , see how it behaves. * Thread Starter [dev](https://wordpress.org/support/users/devksec/) * (@devksec) * [2 years, 10 months ago](https://wordpress.org/support/topic/multisite-crawler-issue/#post-16927820) * At the moment, due to the subsites not being able to run the crawler (A2 host can’t seem to resolve this, it shows as “crawler_disabled”). All of the sitemap URLS are within the multisite primary, if drop domain is added it then won’t have any pages to crawl. * We would need to resolve the crawler_disabled issue for subsites first. Any ideas on how to resolve that ? * Much appreciated for your support with this! * Plugin Support [qtwrk](https://wordpress.org/support/users/qtwrk/) * (@qtwrk) * [2 years, 10 months ago](https://wordpress.org/support/topic/multisite-crawler-issue/#post-16928748) * please relay to A2hosting support , that add * ```wp-block-code CacheEngine on crawler ``` * to your Apache virtualhost context will resolve that error. * Thread Starter [dev](https://wordpress.org/support/users/devksec/) * (@devksec) * [2 years, 10 months ago](https://wordpress.org/support/topic/multisite-crawler-issue/#post-16930484) * Hello, * They have done that which has enabled the multisite primary site to have the crawler but all subsites are still disabled. * Is there a multi-site specific config required? They’ve not been able to get it enabled on any subsite yet. * Plugin Support [qtwrk](https://wordpress.org/support/users/qtwrk/) * (@qtwrk) * [2 years, 10 months ago](https://wordpress.org/support/topic/multisite-crawler-issue/#post-16931230) * hmmm , no , this should work , you can create a phpinfo page for each of these subsites , you will be looking for `$_SERVER['X-LSCACHE'] ` and it should contain word `crawler` in it , to indicate that directive is added * Thread Starter [dev](https://wordpress.org/support/users/devksec/) * (@devksec) * [2 years, 10 months ago](https://wordpress.org/support/topic/multisite-crawler-issue/#post-16931742) * On the subsites litespeed toolbox settings, it shows it as enabled and has a config: * ```wp-block-code Server Variables SERVER_SOFTWARE = LiteSpeed DOCUMENT_ROOT = /home/ksc5577p/REDACTED/ X-LSCACHE = on,crawler ``` * There’s a config there from testing as well. * ```wp-block-code ) img_optm-webp_replace_srcset = true img_optm-jpg_quality = 82 crawler = true crawler-usleep = 500 crawler-run_duration = 400 crawler-run_interval = 600 crawler-crawl_interval = 302400 crawler-threads = 3 crawler-timeout = 30 crawler-load_limit = 3 crawler-sitemap = https://REDACTED/sitemap_index.xml crawler-drop_domain = true crawler-map_timeout = 120 crawler-roles = array ( ) ``` * Doesn’t show any recent activity on the subsite for crawling but does appear to be enabled yet if its manually run * ```wp-block-code Current sitemap crawl started at: 07/22/2023 01:09:30 The next complete sitemap crawl will start at: 08/08/2023 20:59:30 Last complete run time for all crawlers: 1237800 seconds Current crawler started at: 1s ago Current server load: 0.63 Last interval: 1s ago Ended reason: crawler_disabled ``` - Start watching… - 01 Jan 1970 01:00:00     Size: 473     Crawler: #5     Position: 1     Threads: 0     Status: crawler_disabled - .. * - This reply was modified 2 years, 10 months ago by [dev](https://wordpress.org/support/users/devksec/). * Plugin Support [qtwrk](https://wordpress.org/support/users/qtwrk/) * (@qtwrk) * [2 years, 10 months ago](https://wordpress.org/support/topic/multisite-crawler-issue/#post-16932571) * are you visiting the phpinfo page by subsite domain ? instead of going by main site domain ? * Thread Starter [dev](https://wordpress.org/support/users/devksec/) * (@devksec) * [2 years, 10 months ago](https://wordpress.org/support/topic/multisite-crawler-issue/#post-16933224) * This is from litespeeds settings/toolbox on the subsite, the crawler config etc is different from the main primary site. * Plugin Support [qtwrk](https://wordpress.org/support/users/qtwrk/) * (@qtwrk) * [2 years, 10 months ago](https://wordpress.org/support/topic/multisite-crawler-issue/#post-16936924) * please create a ticket by mail to `support at litespeedtech.com` with reference link to this topic , we will investigate further. Viewing 11 replies - 1 through 11 (of 11 total) The topic ‘Multisite crawler issue’ is closed to new replies. * ![](https://ps.w.org/litespeed-cache/assets/icon-256x256.png?rev=2554181) * [LiteSpeed Cache](https://wordpress.org/plugins/litespeed-cache/) * [Frequently Asked Questions](https://wordpress.org/plugins/litespeed-cache/#faq) * [Support Threads](https://wordpress.org/support/plugin/litespeed-cache/) * [Active Topics](https://wordpress.org/support/plugin/litespeed-cache/active/) * [Unresolved Topics](https://wordpress.org/support/plugin/litespeed-cache/unresolved/) * [Reviews](https://wordpress.org/support/plugin/litespeed-cache/reviews/) * 11 replies * 2 participants * Last reply from: [qtwrk](https://wordpress.org/support/users/qtwrk/) * Last activity: [2 years, 10 months ago](https://wordpress.org/support/topic/multisite-crawler-issue/#post-16936924) * Status: resolved