Title: Crawler and http auth Last modified: May 4, 2024 --- # Crawler and http auth * Resolved [Pixelwunder](https://wordpress.org/support/users/pixelwunder/) * (@pixelwunder) * [2 years, 1 month ago](https://wordpress.org/support/topic/crawler-and-http-auth/) * We have a live page (www.) and a stage (stage8.). The stage has a http auth in the htaccess. * However, we have configured the htaccess to allow local access. * I want to test the crawling in particular because it works on the WWW with 6.0, but not with 6.1 and 6.2, after a certain number of pages the crawler stops. * Despite http auth, I was able to read the sitemap correctly on the stage and start the crawler once. After about 500 pages it stopped and LS Cache no longer responded to Reset Position and Manually Run. Crawler status was Watching … * Only when I removed the changes that allow local access did Reset Position and Manually Run work again. * What are we missing here? If the sitemap can be read in, why are crawling, reset position and manual run not working? Viewing 15 replies - 1 through 15 (of 19 total) 1 [2](https://wordpress.org/support/topic/crawler-and-http-auth/page/2/?output_format=md) [→](https://wordpress.org/support/topic/crawler-and-http-auth/page/2/?output_format=md) * Plugin Support [qtwrk](https://wordpress.org/support/users/qtwrk/) * (@qtwrk) * [2 years, 1 month ago](https://wordpress.org/support/topic/crawler-and-http-auth/#post-17735487) * please do me a full screenshot of crawler summary page , and also share the directive in .htaccess related to block/allow * Thread Starter [Pixelwunder](https://wordpress.org/support/users/pixelwunder/) * (@pixelwunder) * [2 years, 1 month ago](https://wordpress.org/support/topic/crawler-and-http-auth/#post-17735533) * ![](https://i0.wp.com/www.prad.de/images/1.png?ssl=1) * ![](https://i0.wp.com/www.prad.de/images/2.png?ssl=1) * ```wp-block-code Deny from all Allow from 127.0.0.1 Allow from ::1 Allow from localhost Allow from XXX.147.219.XX (Website Server) Allow from 10.10.52.247 satisfy any ``` * I have started the crawler manually, but it is terminated immediately. * Thread Starter [Pixelwunder](https://wordpress.org/support/users/pixelwunder/) * (@pixelwunder) * [2 years, 1 month ago](https://wordpress.org/support/topic/crawler-and-http-auth/#post-17735666) * ![](https://i0.wp.com/www.prad.de/images/3.png?ssl=1) * The rawler has now started once and then stops. * It is generally possible to run two instances of LS Cache on one server, isn’t it? - This reply was modified 2 years, 1 month ago by [Pixelwunder](https://wordpress.org/support/users/pixelwunder/). * Plugin Support [qtwrk](https://wordpress.org/support/users/qtwrk/) * (@qtwrk) * [2 years, 1 month ago](https://wordpress.org/support/topic/crawler-and-http-auth/#post-17735706) * it says `stopped_reset` this indicates there was a purge all happened , * try lower the crawler interval to 61 or similar for test , it should start the crawler every 61 seconds * and also , in your `allow` add server public IPv6 if any , and all other IPs if server has multiple IPs * Thread Starter [Pixelwunder](https://wordpress.org/support/users/pixelwunder/) * (@pixelwunder) * [2 years, 1 month ago](https://wordpress.org/support/topic/crawler-and-http-auth/#post-17735722) * The curious thing is that the value is already set to 61 ms. Only under Summary it says: * Current sitemap crawl started at: 24m 5s ago The next complete sitemap crawl will start at: 05/06/2024 16:27:37 * Are two LS Cache instances technically possible on one server? Or do the crawlers of www. and stage8. get in each other’s way with the identical domain? * Plugin Support [qtwrk](https://wordpress.org/support/users/qtwrk/) * (@qtwrk) * [2 years, 1 month ago](https://wordpress.org/support/topic/crawler-and-http-auth/#post-17735728) * that value can’t be 61 ms , please share the report number , let me double check * you can have multiple crawler on one server * Thread Starter [Pixelwunder](https://wordpress.org/support/users/pixelwunder/) * (@pixelwunder) * [2 years, 1 month ago](https://wordpress.org/support/topic/crawler-and-http-auth/#post-17735738) * **JBPBJAUI** * ![](https://i0.wp.com/www.prad.de/images/4.png?ssl=1) * Plugin Support [qtwrk](https://wordpress.org/support/users/qtwrk/) * (@qtwrk) * [2 years, 1 month ago](https://wordpress.org/support/topic/crawler-and-http-auth/#post-17735751) * please try wait for 2 minutes, click the manunal run few times, you should see it running * Thread Starter [Pixelwunder](https://wordpress.org/support/users/pixelwunder/) * (@pixelwunder) * [2 years, 1 month ago](https://wordpress.org/support/topic/crawler-and-http-auth/#post-17735770) * ![](https://i0.wp.com/www.prad.de/images/5.png?ssl=1) * Although the WP-Cron is running correctly and “Manually run” also shows me that Async Crawling has been started, nothing changes at all under Summary: * Current sitemap crawl started at: 48m 25s ago The next complete sitemap crawl will start at: 05/06/2024 16:27:37Last complete run time for all crawlers: 5134 secondsRun time for previous crawler: 5134 secondsCurrent crawler started at: 48m 25s ago * Plugin Support [qtwrk](https://wordpress.org/support/users/qtwrk/) * (@qtwrk) * [2 years, 1 month ago](https://wordpress.org/support/topic/crawler-and-http-auth/#post-17735792) * please try keep click manual start button few times * Thread Starter [Pixelwunder](https://wordpress.org/support/users/pixelwunder/) * (@pixelwunder) * [2 years, 1 month ago](https://wordpress.org/support/topic/crawler-and-http-auth/#post-17735803) * I have already done this several times. It does nothing. * The Crawler has now been at position 436 for an hour. * Plugin Support [qtwrk](https://wordpress.org/support/users/qtwrk/) * (@qtwrk) * [2 years, 1 month ago](https://wordpress.org/support/topic/crawler-and-http-auth/#post-17735812) * ```wp-block-code 100, 'blocking' => true, 'sslverify' => false, // 'cookies' => $_COOKIE, ); $url = 'https://domain.com/wp-admin/admin-ajax.php?action=async_litespeed&nonce=123123123&litespeed_type=crawler'; $result = wp_remote_post(esc_url_raw($url), $args); echo '
';
       var_dump($result);
       echo '
'; ``` * please create a php file with code above, put it at same as your wp-config.php, open it in browser, see what it returns. * Thread Starter [Pixelwunder](https://wordpress.org/support/users/pixelwunder/) * (@pixelwunder) * [2 years, 1 month ago](https://wordpress.org/support/topic/crawler-and-http-auth/#post-17735882) * Composer detected issues in your platform: Your Composer dependencies require a PHP version “>= 8.1.0”. * The website runs php 8.1. * Thread Starter [Pixelwunder](https://wordpress.org/support/users/pixelwunder/) * (@pixelwunder) * [2 years, 1 month ago](https://wordpress.org/support/topic/crawler-and-http-auth/#post-17735895) * ```wp-block-code array(6) { ["headers"]=> object(WpOrg\Requests\Utility\CaseInsensitiveDictionary)#19080 (1) { ["data":protected]=> array(19) { ["date"]=> string(29) "Mon, 06 May 2024 14:28:46 GMT" ["content-type"]=> string(24) "text/html; charset=UTF-8" ["accept-ch"]=> string(277) "Sec-CH-UA-Bitness, Sec-CH-UA-Arch, Sec-CH-UA-Full-Version, Sec-CH-UA-Mobile, Sec-CH-UA-Model, Sec-CH-UA-Platform-Version, Sec-CH-UA-Full-Version-List, Sec-CH-UA-Platform, Sec-CH-UA, UA-Bitness, UA-Arch, UA-Full-Version, UA-Mobile, UA-Model, UA-Platform-Version, UA-Platform, UA" ["cross-origin-embedder-policy"]=> string(12) "require-corp" ["cross-origin-opener-policy"]=> string(11) "same-origin" ["cross-origin-resource-policy"]=> string(11) "same-origin" ["origin-agent-cluster"]=> string(2) "?1" ["permissions-policy"]=> string(267) "accelerometer=(),autoplay=(),browsing-topics=(),camera=(),clipboard-read=(),clipboard-write=(),geolocation=(),gyroscope=(),hid=(),interest-cohort=(),magnetometer=(),microphone=(),payment=(),publickey-credentials-get=(),screen-wake-lock=(),serial=(),sync-xhr=(),usb=()" ["referrer-policy"]=> string(11) "same-origin" ["x-frame-options"]=> string(10) "SAMEORIGIN" ["cf-mitigated"]=> string(9) "challenge" ["cf-chl-out"]=> string(177) "9oCgUQcUhfTHOD/djByAEMXIVgFF4m5Y6r4s80gGWIJbk87HMnHbEES4K0l6091mPsV8k6f3cumELzY+vUVFo+cbh9uyFOceakdqVEdO0spTYBFA9ZT+IP9TVxhJHRxYIjzdP6NlyI4dzkPZgnPB9Q==$0uMQsXiS3aj68bSTq0UTBg==" ["cache-control"]=> string(82) "private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0" ["expires"]=> string(29) "Thu, 01 Jan 1970 00:00:01 GMT" ["set-cookie"]=> string(255) "__cf_bm=ZfmCfpiLGHk2E4aHIgTEdlwdUjdZaTliApiJuxGNqD0-1715005726-1.0.1.1-CV5SjLIl4ZuWiUdsgyDE8lMirx9RlqLWlazy.umxr_JmLMZft7E3vf0MfROVxMvVdeiW54kBsfGdwug9HA.oAQ; path=/; expires=Mon, 06-May-24 14:58:46 GMT; domain=.domain.com; HttpOnly; Secure; SameSite=None" ["vary"]=> string(15) "Accept-Encoding" ["server"]=> string(10) "cloudflare" ["cf-ray"]=> string(20) "87f9a89b7eb51d84-FRA" ["content-encoding"]=> string(4) "gzip" } } ["body"]=> string(16595) " " ["response"]=> array(2) { ["code"]=> int(403) ["message"]=> string(9) "Forbidden" } ["cookies"]=> array(1) { [0]=> object(WP_Http_Cookie)#19088 (7) { ["name"]=> string(7) "__cf_bm" ["value"]=> string(149) "ZfmCfpiLGHk2E4aHIgTEdlwdUjdZaTliApiJuxGNqD0-1715005726-1.0.1.1-CV5SjLIl4ZuWiUdsgyDE8lMirx9RlqLWlazy.umxr_JmLMZft7E3vf0MfROVxMvVdeiW54kBsfGdwug9HA.oAQ" ["expires"]=> int(1715007526) ["path"]=> string(1) "/" ["domain"]=> string(10) "domain.com" ["port"]=> NULL ["host_only"]=> bool(false) } } ["filename"]=> NULL ["http_response"]=> object(WP_HTTP_Requests_Response)#19079 (5) { ["data"]=> NULL ["headers"]=> NULL ["status"]=> NULL ["response":protected]=> object(WpOrg\Requests\Response)#19081 (10) { ["body"]=> string(16595) " " ["raw"]=> string(18258) "HTTP/1.1 403 Forbidden Date: Mon, 06 May 2024 14:28:46 GMT Content-Type: text/html; charset=UTF-8 Transfer-Encoding: chunked Connection: close Accept-CH: Sec-CH-UA-Bitness, Sec-CH-UA-Arch, Sec-CH-UA-Full-Version, Sec-CH-UA-Mobile, Sec-CH-UA-Model, Sec-CH-UA-Platform-Version, Sec-CH-UA-Full-Version-List, Sec-CH-UA-Platform, Sec-CH-UA, UA-Bitness, UA-Arch, UA-Full-Version, UA-Mobile, UA-Model, UA-Platform-Version, UA-Platform, UA Cross-Origin-Embedder-Policy: require-corp Cross-Origin-Opener-Policy: same-origin Cross-Origin-Resource-Policy: same-origin Origin-Agent-Cluster: ?1 Permissions-Policy: accelerometer=(),autoplay=(),browsing-topics=(),camera=(),clipboard-read=(),clipboard-write=(),geolocation=(),gyroscope=(),hid=(),interest-cohort=(),magnetometer=(),microphone=(),payment=(),publickey-credentials-get=(),screen-wake-lock=(),serial=(),sync-xhr=(),usb=() Referrer-Policy: same-origin X-Frame-Options: SAMEORIGIN cf-mitigated: challenge cf-chl-out: 9oCgUQcUhfTHOD/djByAEMXIVgFF4m5Y6r4s80gGWIJbk87HMnHbEES4K0l6091mPsV8k6f3cumELzY+vUVFo+cbh9uyFOceakdqVEdO0spTYBFA9ZT+IP9TVxhJHRxYIjzdP6NlyI4dzkPZgnPB9Q==$0uMQsXiS3aj68bSTq0UTBg== Cache-Control: private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Expires: Thu, 01 Jan 1970 00:00:01 GMT Set-Cookie: __cf_bm=ZfmCfpiLGHk2E4aHIgTEdlwdUjdZaTliApiJuxGNqD0-1715005726-1.0.1.1-CV5SjLIl4ZuWiUdsgyDE8lMirx9RlqLWlazy.umxr_JmLMZft7E3vf0MfROVxMvVdeiW54kBsfGdwug9HA.oAQ; path=/; expires=Mon, 06-May-24 14:58:46 GMT; domain=.domain.com; HttpOnly; Secure; SameSite=None Vary: Accept-Encoding Server: cloudflare CF-RAY: 87f9a89b7eb51d84-FRA Content-Encoding: gzip " ["headers"]=> object(WpOrg\Requests\Response\Headers)#19082 (1) { ["data":protected]=> array(19) { ["date"]=> array(1) { [0]=> string(29) "Mon, 06 May 2024 14:28:46 GMT" } ["content-type"]=> array(1) { [0]=> string(24) "text/html; charset=UTF-8" } ["accept-ch"]=> array(1) { [0]=> string(277) "Sec-CH-UA-Bitness, Sec-CH-UA-Arch, Sec-CH-UA-Full-Version, Sec-CH-UA-Mobile, Sec-CH-UA-Model, Sec-CH-UA-Platform-Version, Sec-CH-UA-Full-Version-List, Sec-CH-UA-Platform, Sec-CH-UA, UA-Bitness, UA-Arch, UA-Full-Version, UA-Mobile, UA-Model, UA-Platform-Version, UA-Platform, UA" } ["cross-origin-embedder-policy"]=> array(1) { [0]=> string(12) "require-corp" } ["cross-origin-opener-policy"]=> array(1) { [0]=> string(11) "same-origin" } ["cross-origin-resource-policy"]=> array(1) { [0]=> string(11) "same-origin" } ["origin-agent-cluster"]=> array(1) { [0]=> string(2) "?1" } ["permissions-policy"]=> array(1) { [0]=> string(267) "accelerometer=(),autoplay=(),browsing-topics=(),camera=(),clipboard-read=(),clipboard-write=(),geolocation=(),gyroscope=(),hid=(),interest-cohort=(),magnetometer=(),microphone=(),payment=(),publickey-credentials-get=(),screen-wake-lock=(),serial=(),sync-xhr=(),usb=()" } ["referrer-policy"]=> array(1) { [0]=> string(11) "same-origin" } ["x-frame-options"]=> array(1) { [0]=> string(10) "SAMEORIGIN" } ["cf-mitigated"]=> array(1) { [0]=> string(9) "challenge" } ["cf-chl-out"]=> array(1) { [0]=> string(177) "9oCgUQcUhfTHOD/djByAEMXIVgFF4m5Y6r4s80gGWIJbk87HMnHbEES4K0l6091mPsV8k6f3cumELzY+vUVFo+cbh9uyFOceakdqVEdO0spTYBFA9ZT+IP9TVxhJHRxYIjzdP6NlyI4dzkPZgnPB9Q==$0uMQsXiS3aj68bSTq0UTBg==" } ["cache-control"]=> array(1) { [0]=> string(82) "private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0" } ["expires"]=> array(1) { [0]=> string(29) "Thu, 01 Jan 1970 00:00:01 GMT" } ["set-cookie"]=> array(1) { [0]=> string(255) "__cf_bm=ZfmCfpiLGHk2E4aHIgTEdlwdUjdZaTliApiJuxGNqD0-1715005726-1.0.1.1-CV5SjLIl4ZuWiUdsgyDE8lMirx9RlqLWlazy.umxr_JmLMZft7E3vf0MfROVxMvVdeiW54kBsfGdwug9HA.oAQ; path=/; expires=Mon, 06-May-24 14:58:46 GMT; domain=.domain.com; HttpOnly; Secure; SameSite=None" } ["vary"]=> array(1) { [0]=> string(15) "Accept-Encoding" } ["server"]=> array(1) { [0]=> string(10) "cloudflare" } ["cf-ray"]=> array(1) { [0]=> string(20) "87f9a89b7eb51d84-FRA" } ["content-encoding"]=> array(1) { [0]=> string(4) "gzip" } } } ["status_code"]=> int(403) ["protocol_version"]=> float(1.100000000000000088817841970012523233890533447265625) ["success"]=> bool(false) ["redirects"]=> int(0) ["url"]=> string(104) "https://domain.com/wp-admin/admin-ajax.php?action=async_litespeed&nonce=123123123&litespeed_type=crawler" ["history"]=> array(0) { } ["cookies"]=> object(WpOrg\Requests\Cookie\Jar)#19078 (1) { ["cookies":protected]=> array(1) { ["__cf_bm"]=> object(WpOrg\Requests\Cookie)#19086 (5) { ["name"]=> string(7) "__cf_bm" ["value"]=> string(149) "ZfmCfpiLGHk2E4aHIgTEdlwdUjdZaTliApiJuxGNqD0-1715005726-1.0.1.1-CV5SjLIl4ZuWiUdsgyDE8lMirx9RlqLWlazy.umxr_JmLMZft7E3vf0MfROVxMvVdeiW54kBsfGdwug9HA.oAQ" ["attributes"]=> object(WpOrg\Requests\Utility\CaseInsensitiveDictionary)#19085 (1) { ["data":protected]=> array(6) { ["path"]=> string(1) "/" ["expires"]=> int(1715007526) ["domain"]=> string(10) "domain.com" ["httponly"]=> bool(true) ["secure"]=> bool(true) ["samesite"]=> string(4) "None" } } ["flags"]=> array(4) { ["creation"]=> int(1715005726) ["last-access"]=> int(1715005726) ["persistent"]=> bool(false) ["host-only"]=> bool(false) } ["reference_time"]=> int(1715005726) } } } } ["filename":protected]=> NULL } } ``` * Sorry, that was my mistake. * Plugin Support [qtwrk](https://wordpress.org/support/users/qtwrk/) * (@qtwrk) * [2 years, 1 month ago](https://wordpress.org/support/topic/crawler-and-http-auth/#post-17735931) * ```wp-block-code ["code"]=> int(403) ["message"]=> string(9) "Forbidden" } ``` * something blocked the admin-ajax.php , which is needed for crawler Viewing 15 replies - 1 through 15 (of 19 total) 1 [2](https://wordpress.org/support/topic/crawler-and-http-auth/page/2/?output_format=md) [→](https://wordpress.org/support/topic/crawler-and-http-auth/page/2/?output_format=md) The topic ‘Crawler and http auth’ is closed to new replies. * ![](https://ps.w.org/litespeed-cache/assets/icon-256x256.png?rev=2554181) * [LiteSpeed Cache](https://wordpress.org/plugins/litespeed-cache/) * [Frequently Asked Questions](https://wordpress.org/plugins/litespeed-cache/#faq) * [Support Threads](https://wordpress.org/support/plugin/litespeed-cache/) * [Active Topics](https://wordpress.org/support/plugin/litespeed-cache/active/) * [Unresolved Topics](https://wordpress.org/support/plugin/litespeed-cache/unresolved/) * [Reviews](https://wordpress.org/support/plugin/litespeed-cache/reviews/) * 19 replies * 2 participants * Last reply from: [qtwrk](https://wordpress.org/support/users/qtwrk/) * Last activity: [2 years, 1 month ago](https://wordpress.org/support/topic/crawler-and-http-auth/page/2/#post-17736156) * Status: resolved