Crawler and http auth
-
We have a live page (www.) and a stage (stage8.). The stage has a http auth in the htaccess.
However, we have configured the htaccess to allow local access.
I want to test the crawling in particular because it works on the WWW with 6.0, but not with 6.1 and 6.2, after a certain number of pages the crawler stops.
Despite http auth, I was able to read the sitemap correctly on the stage and start the crawler once. After about 500 pages it stopped and LS Cache no longer responded to Reset Position and Manually Run. Crawler status was Watching …
Only when I removed the changes that allow local access did Reset Position and Manually Run work again.
What are we missing here? If the sitemap can be read in, why are crawling, reset position and manual run not working?
-
please do me a full screenshot of crawler summary page , and also share the directive in .htaccess related to block/allow


Deny from all Allow from 127.0.0.1 Allow from ::1 Allow from localhost Allow from XXX.147.219.XX (Website Server) Allow from 10.10.52.247 satisfy anyI have started the crawler manually, but it is terminated immediately.

The rawler has now started once and then stops.
It is generally possible to run two instances of LS Cache on one server, isn’t it?
-
This reply was modified 2 years, 1 month ago by
Pixelwunder.
it says
stopped_resetthis indicates there was a purge all happened ,try lower the crawler interval to 61 or similar for test , it should start the crawler every 61 seconds
and also , in your
allowadd server public IPv6 if any , and all other IPs if server has multiple IPsThe curious thing is that the value is already set to 61 ms. Only under Summary it says:
Current sitemap crawl started at: 24m 5s ago
The next complete sitemap crawl will start at: 05/06/2024 16:27:37Are two LS Cache instances technically possible on one server? Or do the crawlers of www. and stage8. get in each other’s way with the identical domain?
that value can’t be 61 ms , please share the report number , let me double check
you can have multiple crawler on one server
JBPBJAUI

please try wait for 2 minutes, click the manunal run few times, you should see it running

Although the WP-Cron is running correctly and “Manually run” also shows me that Async Crawling has been started, nothing changes at all under Summary:
Current sitemap crawl started at: 48m 25s ago
The next complete sitemap crawl will start at: 05/06/2024 16:27:37
Last complete run time for all crawlers: 5134 seconds
Run time for previous crawler: 5134 seconds
Current crawler started at: 48m 25s agoplease try keep click manual start button few times
I have already done this several times. It does nothing.
The Crawler has now been at position 436 for an hour.
<?php require( './wp-load.php' ); $args = array( 'timeout' => 100, 'blocking' => true, 'sslverify' => false, // 'cookies' => $_COOKIE, ); $url = 'https://domain.com/wp-admin/admin-ajax.php?action=async_litespeed&nonce=123123123&litespeed_type=crawler'; $result = wp_remote_post(esc_url_raw($url), $args); echo '<pre>'; var_dump($result); echo '</pre>';please create a php file with code above, put it at same as your wp-config.php , open it in browser, see what it returns.
Composer detected issues in your platform: Your Composer dependencies require a PHP version “>= 8.1.0”.
The website runs php 8.1.
array(6) { ["headers"]=> object(WpOrg\Requests\Utility\CaseInsensitiveDictionary)#19080 (1) { ["data":protected]=> array(19) { ["date"]=> string(29) "Mon, 06 May 2024 14:28:46 GMT" ["content-type"]=> string(24) "text/html; charset=UTF-8" ["accept-ch"]=> string(277) "Sec-CH-UA-Bitness, Sec-CH-UA-Arch, Sec-CH-UA-Full-Version, Sec-CH-UA-Mobile, Sec-CH-UA-Model, Sec-CH-UA-Platform-Version, Sec-CH-UA-Full-Version-List, Sec-CH-UA-Platform, Sec-CH-UA, UA-Bitness, UA-Arch, UA-Full-Version, UA-Mobile, UA-Model, UA-Platform-Version, UA-Platform, UA" ["cross-origin-embedder-policy"]=> string(12) "require-corp" ["cross-origin-opener-policy"]=> string(11) "same-origin" ["cross-origin-resource-policy"]=> string(11) "same-origin" ["origin-agent-cluster"]=> string(2) "?1" ["permissions-policy"]=> string(267) "accelerometer=(),autoplay=(),browsing-topics=(),camera=(),clipboard-read=(),clipboard-write=(),geolocation=(),gyroscope=(),hid=(),interest-cohort=(),magnetometer=(),microphone=(),payment=(),publickey-credentials-get=(),screen-wake-lock=(),serial=(),sync-xhr=(),usb=()" ["referrer-policy"]=> string(11) "same-origin" ["x-frame-options"]=> string(10) "SAMEORIGIN" ["cf-mitigated"]=> string(9) "challenge" ["cf-chl-out"]=> string(177) "9oCgUQcUhfTHOD/djByAEMXIVgFF4m5Y6r4s80gGWIJbk87HMnHbEES4K0l6091mPsV8k6f3cumELzY+vUVFo+cbh9uyFOceakdqVEdO0spTYBFA9ZT+IP9TVxhJHRxYIjzdP6NlyI4dzkPZgnPB9Q==$0uMQsXiS3aj68bSTq0UTBg==" ["cache-control"]=> string(82) "private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0" ["expires"]=> string(29) "Thu, 01 Jan 1970 00:00:01 GMT" ["set-cookie"]=> string(255) "__cf_bm=ZfmCfpiLGHk2E4aHIgTEdlwdUjdZaTliApiJuxGNqD0-1715005726-1.0.1.1-CV5SjLIl4ZuWiUdsgyDE8lMirx9RlqLWlazy.umxr_JmLMZft7E3vf0MfROVxMvVdeiW54kBsfGdwug9HA.oAQ; path=/; expires=Mon, 06-May-24 14:58:46 GMT; domain=.domain.com; HttpOnly; Secure; SameSite=None" ["vary"]=> string(15) "Accept-Encoding" ["server"]=> string(10) "cloudflare" ["cf-ray"]=> string(20) "87f9a89b7eb51d84-FRA" ["content-encoding"]=> string(4) "gzip" } } ["body"]=> string(16595) " " ["response"]=> array(2) { ["code"]=> int(403) ["message"]=> string(9) "Forbidden" } ["cookies"]=> array(1) { [0]=> object(WP_Http_Cookie)#19088 (7) { ["name"]=> string(7) "__cf_bm" ["value"]=> string(149) "ZfmCfpiLGHk2E4aHIgTEdlwdUjdZaTliApiJuxGNqD0-1715005726-1.0.1.1-CV5SjLIl4ZuWiUdsgyDE8lMirx9RlqLWlazy.umxr_JmLMZft7E3vf0MfROVxMvVdeiW54kBsfGdwug9HA.oAQ" ["expires"]=> int(1715007526) ["path"]=> string(1) "/" ["domain"]=> string(10) "domain.com" ["port"]=> NULL ["host_only"]=> bool(false) } } ["filename"]=> NULL ["http_response"]=> object(WP_HTTP_Requests_Response)#19079 (5) { ["data"]=> NULL ["headers"]=> NULL ["status"]=> NULL ["response":protected]=> object(WpOrg\Requests\Response)#19081 (10) { ["body"]=> string(16595) " " ["raw"]=> string(18258) "HTTP/1.1 403 Forbidden Date: Mon, 06 May 2024 14:28:46 GMT Content-Type: text/html; charset=UTF-8 Transfer-Encoding: chunked Connection: close Accept-CH: Sec-CH-UA-Bitness, Sec-CH-UA-Arch, Sec-CH-UA-Full-Version, Sec-CH-UA-Mobile, Sec-CH-UA-Model, Sec-CH-UA-Platform-Version, Sec-CH-UA-Full-Version-List, Sec-CH-UA-Platform, Sec-CH-UA, UA-Bitness, UA-Arch, UA-Full-Version, UA-Mobile, UA-Model, UA-Platform-Version, UA-Platform, UA Cross-Origin-Embedder-Policy: require-corp Cross-Origin-Opener-Policy: same-origin Cross-Origin-Resource-Policy: same-origin Origin-Agent-Cluster: ?1 Permissions-Policy: accelerometer=(),autoplay=(),browsing-topics=(),camera=(),clipboard-read=(),clipboard-write=(),geolocation=(),gyroscope=(),hid=(),interest-cohort=(),magnetometer=(),microphone=(),payment=(),publickey-credentials-get=(),screen-wake-lock=(),serial=(),sync-xhr=(),usb=() Referrer-Policy: same-origin X-Frame-Options: SAMEORIGIN cf-mitigated: challenge cf-chl-out: 9oCgUQcUhfTHOD/djByAEMXIVgFF4m5Y6r4s80gGWIJbk87HMnHbEES4K0l6091mPsV8k6f3cumELzY+vUVFo+cbh9uyFOceakdqVEdO0spTYBFA9ZT+IP9TVxhJHRxYIjzdP6NlyI4dzkPZgnPB9Q==$0uMQsXiS3aj68bSTq0UTBg== Cache-Control: private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Expires: Thu, 01 Jan 1970 00:00:01 GMT Set-Cookie: __cf_bm=ZfmCfpiLGHk2E4aHIgTEdlwdUjdZaTliApiJuxGNqD0-1715005726-1.0.1.1-CV5SjLIl4ZuWiUdsgyDE8lMirx9RlqLWlazy.umxr_JmLMZft7E3vf0MfROVxMvVdeiW54kBsfGdwug9HA.oAQ; path=/; expires=Mon, 06-May-24 14:58:46 GMT; domain=.domain.com; HttpOnly; Secure; SameSite=None Vary: Accept-Encoding Server: cloudflare CF-RAY: 87f9a89b7eb51d84-FRA Content-Encoding: gzip " ["headers"]=> object(WpOrg\Requests\Response\Headers)#19082 (1) { ["data":protected]=> array(19) { ["date"]=> array(1) { [0]=> string(29) "Mon, 06 May 2024 14:28:46 GMT" } ["content-type"]=> array(1) { [0]=> string(24) "text/html; charset=UTF-8" } ["accept-ch"]=> array(1) { [0]=> string(277) "Sec-CH-UA-Bitness, Sec-CH-UA-Arch, Sec-CH-UA-Full-Version, Sec-CH-UA-Mobile, Sec-CH-UA-Model, Sec-CH-UA-Platform-Version, Sec-CH-UA-Full-Version-List, Sec-CH-UA-Platform, Sec-CH-UA, UA-Bitness, UA-Arch, UA-Full-Version, UA-Mobile, UA-Model, UA-Platform-Version, UA-Platform, UA" } ["cross-origin-embedder-policy"]=> array(1) { [0]=> string(12) "require-corp" } ["cross-origin-opener-policy"]=> array(1) { [0]=> string(11) "same-origin" } ["cross-origin-resource-policy"]=> array(1) { [0]=> string(11) "same-origin" } ["origin-agent-cluster"]=> array(1) { [0]=> string(2) "?1" } ["permissions-policy"]=> array(1) { [0]=> string(267) "accelerometer=(),autoplay=(),browsing-topics=(),camera=(),clipboard-read=(),clipboard-write=(),geolocation=(),gyroscope=(),hid=(),interest-cohort=(),magnetometer=(),microphone=(),payment=(),publickey-credentials-get=(),screen-wake-lock=(),serial=(),sync-xhr=(),usb=()" } ["referrer-policy"]=> array(1) { [0]=> string(11) "same-origin" } ["x-frame-options"]=> array(1) { [0]=> string(10) "SAMEORIGIN" } ["cf-mitigated"]=> array(1) { [0]=> string(9) "challenge" } ["cf-chl-out"]=> array(1) { [0]=> string(177) "9oCgUQcUhfTHOD/djByAEMXIVgFF4m5Y6r4s80gGWIJbk87HMnHbEES4K0l6091mPsV8k6f3cumELzY+vUVFo+cbh9uyFOceakdqVEdO0spTYBFA9ZT+IP9TVxhJHRxYIjzdP6NlyI4dzkPZgnPB9Q==$0uMQsXiS3aj68bSTq0UTBg==" } ["cache-control"]=> array(1) { [0]=> string(82) "private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0" } ["expires"]=> array(1) { [0]=> string(29) "Thu, 01 Jan 1970 00:00:01 GMT" } ["set-cookie"]=> array(1) { [0]=> string(255) "__cf_bm=ZfmCfpiLGHk2E4aHIgTEdlwdUjdZaTliApiJuxGNqD0-1715005726-1.0.1.1-CV5SjLIl4ZuWiUdsgyDE8lMirx9RlqLWlazy.umxr_JmLMZft7E3vf0MfROVxMvVdeiW54kBsfGdwug9HA.oAQ; path=/; expires=Mon, 06-May-24 14:58:46 GMT; domain=.domain.com; HttpOnly; Secure; SameSite=None" } ["vary"]=> array(1) { [0]=> string(15) "Accept-Encoding" } ["server"]=> array(1) { [0]=> string(10) "cloudflare" } ["cf-ray"]=> array(1) { [0]=> string(20) "87f9a89b7eb51d84-FRA" } ["content-encoding"]=> array(1) { [0]=> string(4) "gzip" } } } ["status_code"]=> int(403) ["protocol_version"]=> float(1.100000000000000088817841970012523233890533447265625) ["success"]=> bool(false) ["redirects"]=> int(0) ["url"]=> string(104) "https://domain.com/wp-admin/admin-ajax.php?action=async_litespeed&nonce=123123123&litespeed_type=crawler" ["history"]=> array(0) { } ["cookies"]=> object(WpOrg\Requests\Cookie\Jar)#19078 (1) { ["cookies":protected]=> array(1) { ["__cf_bm"]=> object(WpOrg\Requests\Cookie)#19086 (5) { ["name"]=> string(7) "__cf_bm" ["value"]=> string(149) "ZfmCfpiLGHk2E4aHIgTEdlwdUjdZaTliApiJuxGNqD0-1715005726-1.0.1.1-CV5SjLIl4ZuWiUdsgyDE8lMirx9RlqLWlazy.umxr_JmLMZft7E3vf0MfROVxMvVdeiW54kBsfGdwug9HA.oAQ" ["attributes"]=> object(WpOrg\Requests\Utility\CaseInsensitiveDictionary)#19085 (1) { ["data":protected]=> array(6) { ["path"]=> string(1) "/" ["expires"]=> int(1715007526) ["domain"]=> string(10) "domain.com" ["httponly"]=> bool(true) ["secure"]=> bool(true) ["samesite"]=> string(4) "None" } } ["flags"]=> array(4) { ["creation"]=> int(1715005726) ["last-access"]=> int(1715005726) ["persistent"]=> bool(false) ["host-only"]=> bool(false) } ["reference_time"]=> int(1715005726) } } } } ["filename":protected]=> NULL } }Sorry, that was my mistake.
["code"]=> int(403) ["message"]=> string(9) "Forbidden" }something blocked the admin-ajax.php , which is needed for crawler
-
This reply was modified 2 years, 1 month ago by
The topic ‘Crawler and http auth’ is closed to new replies.