Crawler and http auth
-
We have a live page (www.) and a stage (stage8.). The stage has a http auth in the htaccess.
However, we have configured the htaccess to allow local access.
I want to test the crawling in particular because it works on the WWW with 6.0, but not with 6.1 and 6.2, after a certain number of pages the crawler stops.
Despite http auth, I was able to read the sitemap correctly on the stage and start the crawler once. After about 500 pages it stopped and LS Cache no longer responded to Reset Position and Manually Run. Crawler status was Watching …
Only when I removed the changes that allow local access did Reset Position and Manually Run work again.
What are we missing here? If the sitemap can be read in, why are crawling, reset position and manual run not working?
-
I will have this checked by my programmer and get back to you. Thanks for the hint.
no sorry , wait , I forgot to say , you need to replace
domain.comto your actual domain and open the fileSorry, I had forgotten to adjust the php file.
array(6) { ["headers"]=> object(WpOrg\Requests\Utility\CaseInsensitiveDictionary)#19080 (1) { ["data":protected]=> array(21) { ["set-cookie"]=> string(58) "PHPSESSID=2b9dc1529e77aac34f16a06a44311165; path=/; secure" ["pragma"]=> string(8) "no-cache" ["content-type"]=> string(24) "text/html; charset=UTF-8" ["x-robots-tag"]=> string(7) "noindex" ["x-content-type-options"]=> string(7) "nosniff" ["referrer-policy"]=> string(31) "strict-origin-when-cross-origin" ["x-frame-options"]=> string(21) "SAMEORIGIN,SAMEORIGIN" ["x-litespeed-tag"]=> string(12) "9ca_HTTP.200" ["expires"]=> string(29) "Wed, 11 Jan 1984 05:00:00 GMT" ["x-litespeed-cache-control"]=> string(8) "no-cache" ["cache-control"]=> string(46) "no-cache, no-store, must-revalidate, max-age=0" ["content-length"]=> string(1) "5" ["content-encoding"]=> string(2) "br" ["vary"]=> string(15) "Accept-Encoding" ["date"]=> string(29) "Mon, 06 May 2024 15:55:49 GMT" ["server"]=> string(9) "LiteSpeed" ["www-authenticate"]=> string(35) "Basic realm="Restricted Admin-Area"" ["x-xss-protection"]=> string(13) "1; mode=block" ["strict-transport-security"]=> string(16) "max-age=17280000" ["permissions-policy"]=> string(52) "camera=(), geolocation=(), microphone=(), payment=()" ["alt-svc"]=> string(167) "h3=":443"; ma=2592000, h3-29=":443"; ma=2592000, h3-Q050=":443"; ma=2592000, h3-Q046=":443"; ma=2592000, h3-Q043=":443"; ma=2592000, quic=":443"; ma=2592000; v="43,46"" } } ["body"]=> string(0) "" ["response"]=> array(2) { ["code"]=> int(200) ["message"]=> string(2) "OK" } ["cookies"]=> array(1) { [0]=> object(WP_Http_Cookie)#19088 (7) { ["name"]=> string(9) "PHPSESSID" ["value"]=> string(32) "2b9dc1529e77aac34f16a06a44311165" ["expires"]=> NULL ["path"]=> string(1) "/" ["domain"]=> string(14) "stage8.prad.de" ["port"]=> NULL ["host_only"]=> bool(true) } } ["filename"]=> NULL ["http_response"]=> object(WP_HTTP_Requests_Response)#19079 (5) { ["data"]=> NULL ["headers"]=> NULL ["status"]=> NULL ["response":protected]=> object(WpOrg\Requests\Response)#19081 (10) { ["body"]=> string(0) "" ["raw"]=> string(991) "HTTP/1.1 200 OK Connection: close set-cookie: PHPSESSID=2b9dc1529e77aac34f16a06a44311165; path=/; secure pragma: no-cache content-type: text/html; charset=UTF-8 x-robots-tag: noindex x-content-type-options: nosniff referrer-policy: strict-origin-when-cross-origin x-frame-options: SAMEORIGIN,SAMEORIGIN x-litespeed-tag: 9ca_HTTP.200 expires: Wed, 11 Jan 1984 05:00:00 GMT x-litespeed-cache-control: no-cache cache-control: no-cache, no-store, must-revalidate, max-age=0 content-length: 5 content-encoding: br vary: Accept-Encoding date: Mon, 06 May 2024 15:55:49 GMT server: LiteSpeed www-authenticate: Basic realm="Restricted Admin-Area" x-xss-protection: 1; mode=block strict-transport-security: max-age=17280000 permissions-policy: camera=(), geolocation=(), microphone=(), payment=() alt-svc: h3=":443"; ma=2592000, h3-29=":443"; ma=2592000, h3-Q050=":443"; ma=2592000, h3-Q046=":443"; ma=2592000, h3-Q043=":443"; ma=2592000, quic=":443"; ma=2592000; v="43,46" 0" ["headers"]=> object(WpOrg\Requests\Response\Headers)#19082 (1) { ["data":protected]=> array(21) { ["set-cookie"]=> array(1) { [0]=> string(58) "PHPSESSID=2b9dc1529e77aac34f16a06a44311165; path=/; secure" } ["pragma"]=> array(1) { [0]=> string(8) "no-cache" } ["content-type"]=> array(1) { [0]=> string(24) "text/html; charset=UTF-8" } ["x-robots-tag"]=> array(1) { [0]=> string(7) "noindex" } ["x-content-type-options"]=> array(1) { [0]=> string(7) "nosniff" } ["referrer-policy"]=> array(1) { [0]=> string(31) "strict-origin-when-cross-origin" } ["x-frame-options"]=> array(1) { [0]=> string(21) "SAMEORIGIN,SAMEORIGIN" } ["x-litespeed-tag"]=> array(1) { [0]=> string(12) "9ca_HTTP.200" } ["expires"]=> array(1) { [0]=> string(29) "Wed, 11 Jan 1984 05:00:00 GMT" } ["x-litespeed-cache-control"]=> array(1) { [0]=> string(8) "no-cache" } ["cache-control"]=> array(1) { [0]=> string(46) "no-cache, no-store, must-revalidate, max-age=0" } ["content-length"]=> array(1) { [0]=> string(1) "5" } ["content-encoding"]=> array(1) { [0]=> string(2) "br" } ["vary"]=> array(1) { [0]=> string(15) "Accept-Encoding" } ["date"]=> array(1) { [0]=> string(29) "Mon, 06 May 2024 15:55:49 GMT" } ["server"]=> array(1) { [0]=> string(9) "LiteSpeed" } ["www-authenticate"]=> array(1) { [0]=> string(35) "Basic realm="Restricted Admin-Area"" } ["x-xss-protection"]=> array(1) { [0]=> string(13) "1; mode=block" } ["strict-transport-security"]=> array(1) { [0]=> string(16) "max-age=17280000" } ["permissions-policy"]=> array(1) { [0]=> string(52) "camera=(), geolocation=(), microphone=(), payment=()" } ["alt-svc"]=> array(1) { [0]=> string(167) "h3=":443"; ma=2592000, h3-29=":443"; ma=2592000, h3-Q050=":443"; ma=2592000, h3-Q046=":443"; ma=2592000, h3-Q043=":443"; ma=2592000, quic=":443"; ma=2592000; v="43,46"" } } } ["status_code"]=> int(200) ["protocol_version"]=> float(1.100000000000000088817841970012523233890533447265625) ["success"]=> bool(true) ["redirects"]=> int(0) ["url"]=> string(108) "https://stage8.prad.de/wp-admin/admin-ajax.php?action=async_litespeed&nonce=123123123&litespeed_type=crawler" ["history"]=> array(0) { } ["cookies"]=> object(WpOrg\Requests\Cookie\Jar)#19078 (1) { ["cookies":protected]=> array(1) { ["PHPSESSID"]=> object(WpOrg\Requests\Cookie)#19086 (5) { ["name"]=> string(9) "PHPSESSID" ["value"]=> string(32) "2b9dc1529e77aac34f16a06a44311165" ["attributes"]=> object(WpOrg\Requests\Utility\CaseInsensitiveDictionary)#19085 (1) { ["data":protected]=> array(3) { ["path"]=> string(1) "/" ["secure"]=> bool(true) ["domain"]=> string(14) "stage8.prad.de" } } ["flags"]=> array(4) { ["creation"]=> int(1715010949) ["last-access"]=> int(1715010949) ["persistent"]=> bool(false) ["host-only"]=> bool(true) } ["reference_time"]=> int(1715010949) } } } } ["filename":protected]=> NULL } }alright , this looks alright
please create a ticket by mail to
support at litespeedtech.comwith reference link to this topic, we will investigate further
The topic ‘Crawler and http auth’ is closed to new replies.