• Resolved Pixelwunder

    (@pixelwunder)


    We have a live page (www.) and a stage (stage8.). The stage has a http auth in the htaccess.

    However, we have configured the htaccess to allow local access.

    I want to test the crawling in particular because it works on the WWW with 6.0, but not with 6.1 and 6.2, after a certain number of pages the crawler stops.

    Despite http auth, I was able to read the sitemap correctly on the stage and start the crawler once. After about 500 pages it stopped and LS Cache no longer responded to Reset Position and Manually Run. Crawler status was Watching …

    Only when I removed the changes that allow local access did Reset Position and Manually Run work again.

    What are we missing here? If the sitemap can be read in, why are crawling, reset position and manual run not working?

Viewing 15 replies - 1 through 15 (of 19 total)
  • Plugin Support qtwrk

    (@qtwrk)

    please do me a full screenshot of crawler summary page , and also share the directive in .htaccess related to block/allow

    Thread Starter Pixelwunder

    (@pixelwunder)

    Deny from all
    Allow from 127.0.0.1
    Allow from ::1
    Allow from localhost
    Allow from XXX.147.219.XX (Website Server)
    Allow from 10.10.52.247
    satisfy any

    I have started the crawler manually, but it is terminated immediately.

    Thread Starter Pixelwunder

    (@pixelwunder)

    The rawler has now started once and then stops.

    It is generally possible to run two instances of LS Cache on one server, isn’t it?

    • This reply was modified 2 years, 1 month ago by Pixelwunder.
    Plugin Support qtwrk

    (@qtwrk)

    it says stopped_reset this indicates there was a purge all happened ,

    try lower the crawler interval to 61 or similar for test , it should start the crawler every 61 seconds

    and also , in your allow add server public IPv6 if any , and all other IPs if server has multiple IPs

    Thread Starter Pixelwunder

    (@pixelwunder)

    The curious thing is that the value is already set to 61 ms. Only under Summary it says:

    Current sitemap crawl started at: 24m 5s ago
    The next complete sitemap crawl will start at: 05/06/2024 16:27:37

    Are two LS Cache instances technically possible on one server? Or do the crawlers of www. and stage8. get in each other’s way with the identical domain?

    Plugin Support qtwrk

    (@qtwrk)

    that value can’t be 61 ms , please share the report number , let me double check

    you can have multiple crawler on one server

    Thread Starter Pixelwunder

    (@pixelwunder)

    JBPBJAUI

    Plugin Support qtwrk

    (@qtwrk)

    please try wait for 2 minutes, click the manunal run few times, you should see it running

    Thread Starter Pixelwunder

    (@pixelwunder)

    Although the WP-Cron is running correctly and “Manually run” also shows me that Async Crawling has been started, nothing changes at all under Summary:

    Current sitemap crawl started at: 48m 25s ago
    The next complete sitemap crawl will start at: 05/06/2024 16:27:37
    Last complete run time for all crawlers: 5134 seconds
    Run time for previous crawler: 5134 seconds
    Current crawler started at: 48m 25s ago

    Plugin Support qtwrk

    (@qtwrk)

    please try keep click manual start button few times

    Thread Starter Pixelwunder

    (@pixelwunder)

    I have already done this several times. It does nothing.

    The Crawler has now been at position 436 for an hour.

    Plugin Support qtwrk

    (@qtwrk)

    <?php
    
    require( './wp-load.php' );
    $args = array(
      'timeout'   => 100,
      'blocking'  => true,
      'sslverify' => false,
      // 'cookies'   => $_COOKIE,
    );
    
    $url = 'https://domain.com/wp-admin/admin-ajax.php?action=async_litespeed&nonce=123123123&litespeed_type=crawler';
    $result = wp_remote_post(esc_url_raw($url), $args);
    echo '<pre>';
    var_dump($result);
    echo '</pre>';

    please create a php file with code above, put it at same as your wp-config.php , open it in browser, see what it returns.

    Thread Starter Pixelwunder

    (@pixelwunder)

    Composer detected issues in your platform: Your Composer dependencies require a PHP version “>= 8.1.0”.

    The website runs php 8.1.

    Thread Starter Pixelwunder

    (@pixelwunder)

    array(6) {
      ["headers"]=>
      object(WpOrg\Requests\Utility\CaseInsensitiveDictionary)#19080 (1) {
        ["data":protected]=>
        array(19) {
          ["date"]=>
          string(29) "Mon, 06 May 2024 14:28:46 GMT"
          ["content-type"]=>
          string(24) "text/html; charset=UTF-8"
          ["accept-ch"]=>
          string(277) "Sec-CH-UA-Bitness, Sec-CH-UA-Arch, Sec-CH-UA-Full-Version, Sec-CH-UA-Mobile, Sec-CH-UA-Model, Sec-CH-UA-Platform-Version, Sec-CH-UA-Full-Version-List, Sec-CH-UA-Platform, Sec-CH-UA, UA-Bitness, UA-Arch, UA-Full-Version, UA-Mobile, UA-Model, UA-Platform-Version, UA-Platform, UA"
          ["cross-origin-embedder-policy"]=>
          string(12) "require-corp"
          ["cross-origin-opener-policy"]=>
          string(11) "same-origin"
          ["cross-origin-resource-policy"]=>
          string(11) "same-origin"
          ["origin-agent-cluster"]=>
          string(2) "?1"
          ["permissions-policy"]=>
          string(267) "accelerometer=(),autoplay=(),browsing-topics=(),camera=(),clipboard-read=(),clipboard-write=(),geolocation=(),gyroscope=(),hid=(),interest-cohort=(),magnetometer=(),microphone=(),payment=(),publickey-credentials-get=(),screen-wake-lock=(),serial=(),sync-xhr=(),usb=()"
          ["referrer-policy"]=>
          string(11) "same-origin"
          ["x-frame-options"]=>
          string(10) "SAMEORIGIN"
          ["cf-mitigated"]=>
          string(9) "challenge"
          ["cf-chl-out"]=>
          string(177) "9oCgUQcUhfTHOD/djByAEMXIVgFF4m5Y6r4s80gGWIJbk87HMnHbEES4K0l6091mPsV8k6f3cumELzY+vUVFo+cbh9uyFOceakdqVEdO0spTYBFA9ZT+IP9TVxhJHRxYIjzdP6NlyI4dzkPZgnPB9Q==$0uMQsXiS3aj68bSTq0UTBg=="
          ["cache-control"]=>
          string(82) "private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0"
          ["expires"]=>
          string(29) "Thu, 01 Jan 1970 00:00:01 GMT"
          ["set-cookie"]=>
          string(255) "__cf_bm=ZfmCfpiLGHk2E4aHIgTEdlwdUjdZaTliApiJuxGNqD0-1715005726-1.0.1.1-CV5SjLIl4ZuWiUdsgyDE8lMirx9RlqLWlazy.umxr_JmLMZft7E3vf0MfROVxMvVdeiW54kBsfGdwug9HA.oAQ; path=/; expires=Mon, 06-May-24 14:58:46 GMT; domain=.domain.com; HttpOnly; Secure; SameSite=None"
          ["vary"]=>
          string(15) "Accept-Encoding"
          ["server"]=>
          string(10) "cloudflare"
          ["cf-ray"]=>
          string(20) "87f9a89b7eb51d84-FRA"
          ["content-encoding"]=>
          string(4) "gzip"
        }
      }
      ["body"]=>
      string(16595) "
    "
      ["response"]=>
      array(2) {
        ["code"]=>
        int(403)
        ["message"]=>
        string(9) "Forbidden"
      }
      ["cookies"]=>
      array(1) {
        [0]=>
        object(WP_Http_Cookie)#19088 (7) {
          ["name"]=>
          string(7) "__cf_bm"
          ["value"]=>
          string(149) "ZfmCfpiLGHk2E4aHIgTEdlwdUjdZaTliApiJuxGNqD0-1715005726-1.0.1.1-CV5SjLIl4ZuWiUdsgyDE8lMirx9RlqLWlazy.umxr_JmLMZft7E3vf0MfROVxMvVdeiW54kBsfGdwug9HA.oAQ"
          ["expires"]=>
          int(1715007526)
          ["path"]=>
          string(1) "/"
          ["domain"]=>
          string(10) "domain.com"
          ["port"]=>
          NULL
          ["host_only"]=>
          bool(false)
        }
      }
      ["filename"]=>
      NULL
      ["http_response"]=>
      object(WP_HTTP_Requests_Response)#19079 (5) {
        ["data"]=>
        NULL
        ["headers"]=>
        NULL
        ["status"]=>
        NULL
        ["response":protected]=>
        object(WpOrg\Requests\Response)#19081 (10) {
          ["body"]=>
          string(16595) "
    "
          ["raw"]=>
          string(18258) "HTTP/1.1 403 Forbidden
    Date: Mon, 06 May 2024 14:28:46 GMT
    Content-Type: text/html; charset=UTF-8
    Transfer-Encoding: chunked
    Connection: close
    Accept-CH: Sec-CH-UA-Bitness, Sec-CH-UA-Arch, Sec-CH-UA-Full-Version, Sec-CH-UA-Mobile, Sec-CH-UA-Model, Sec-CH-UA-Platform-Version, Sec-CH-UA-Full-Version-List, Sec-CH-UA-Platform, Sec-CH-UA, UA-Bitness, UA-Arch, UA-Full-Version, UA-Mobile, UA-Model, UA-Platform-Version, UA-Platform, UA
    Cross-Origin-Embedder-Policy: require-corp
    Cross-Origin-Opener-Policy: same-origin
    Cross-Origin-Resource-Policy: same-origin
    Origin-Agent-Cluster: ?1
    Permissions-Policy: accelerometer=(),autoplay=(),browsing-topics=(),camera=(),clipboard-read=(),clipboard-write=(),geolocation=(),gyroscope=(),hid=(),interest-cohort=(),magnetometer=(),microphone=(),payment=(),publickey-credentials-get=(),screen-wake-lock=(),serial=(),sync-xhr=(),usb=()
    Referrer-Policy: same-origin
    X-Frame-Options: SAMEORIGIN
    cf-mitigated: challenge
    cf-chl-out: 9oCgUQcUhfTHOD/djByAEMXIVgFF4m5Y6r4s80gGWIJbk87HMnHbEES4K0l6091mPsV8k6f3cumELzY+vUVFo+cbh9uyFOceakdqVEdO0spTYBFA9ZT+IP9TVxhJHRxYIjzdP6NlyI4dzkPZgnPB9Q==$0uMQsXiS3aj68bSTq0UTBg==
    Cache-Control: private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0
    Expires: Thu, 01 Jan 1970 00:00:01 GMT
    Set-Cookie: __cf_bm=ZfmCfpiLGHk2E4aHIgTEdlwdUjdZaTliApiJuxGNqD0-1715005726-1.0.1.1-CV5SjLIl4ZuWiUdsgyDE8lMirx9RlqLWlazy.umxr_JmLMZft7E3vf0MfROVxMvVdeiW54kBsfGdwug9HA.oAQ; path=/; expires=Mon, 06-May-24 14:58:46 GMT; domain=.domain.com; HttpOnly; Secure; SameSite=None
    Vary: Accept-Encoding
    Server: cloudflare
    CF-RAY: 87f9a89b7eb51d84-FRA
    Content-Encoding: gzip
    
    "
          ["headers"]=>
          object(WpOrg\Requests\Response\Headers)#19082 (1) {
            ["data":protected]=>
            array(19) {
              ["date"]=>
              array(1) {
                [0]=>
                string(29) "Mon, 06 May 2024 14:28:46 GMT"
              }
              ["content-type"]=>
              array(1) {
                [0]=>
                string(24) "text/html; charset=UTF-8"
              }
              ["accept-ch"]=>
              array(1) {
                [0]=>
                string(277) "Sec-CH-UA-Bitness, Sec-CH-UA-Arch, Sec-CH-UA-Full-Version, Sec-CH-UA-Mobile, Sec-CH-UA-Model, Sec-CH-UA-Platform-Version, Sec-CH-UA-Full-Version-List, Sec-CH-UA-Platform, Sec-CH-UA, UA-Bitness, UA-Arch, UA-Full-Version, UA-Mobile, UA-Model, UA-Platform-Version, UA-Platform, UA"
              }
              ["cross-origin-embedder-policy"]=>
              array(1) {
                [0]=>
                string(12) "require-corp"
              }
              ["cross-origin-opener-policy"]=>
              array(1) {
                [0]=>
                string(11) "same-origin"
              }
              ["cross-origin-resource-policy"]=>
              array(1) {
                [0]=>
                string(11) "same-origin"
              }
              ["origin-agent-cluster"]=>
              array(1) {
                [0]=>
                string(2) "?1"
              }
              ["permissions-policy"]=>
              array(1) {
                [0]=>
                string(267) "accelerometer=(),autoplay=(),browsing-topics=(),camera=(),clipboard-read=(),clipboard-write=(),geolocation=(),gyroscope=(),hid=(),interest-cohort=(),magnetometer=(),microphone=(),payment=(),publickey-credentials-get=(),screen-wake-lock=(),serial=(),sync-xhr=(),usb=()"
              }
              ["referrer-policy"]=>
              array(1) {
                [0]=>
                string(11) "same-origin"
              }
              ["x-frame-options"]=>
              array(1) {
                [0]=>
                string(10) "SAMEORIGIN"
              }
              ["cf-mitigated"]=>
              array(1) {
                [0]=>
                string(9) "challenge"
              }
              ["cf-chl-out"]=>
              array(1) {
                [0]=>
                string(177) "9oCgUQcUhfTHOD/djByAEMXIVgFF4m5Y6r4s80gGWIJbk87HMnHbEES4K0l6091mPsV8k6f3cumELzY+vUVFo+cbh9uyFOceakdqVEdO0spTYBFA9ZT+IP9TVxhJHRxYIjzdP6NlyI4dzkPZgnPB9Q==$0uMQsXiS3aj68bSTq0UTBg=="
              }
              ["cache-control"]=>
              array(1) {
                [0]=>
                string(82) "private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0"
              }
              ["expires"]=>
              array(1) {
                [0]=>
                string(29) "Thu, 01 Jan 1970 00:00:01 GMT"
              }
              ["set-cookie"]=>
              array(1) {
                [0]=>
                string(255) "__cf_bm=ZfmCfpiLGHk2E4aHIgTEdlwdUjdZaTliApiJuxGNqD0-1715005726-1.0.1.1-CV5SjLIl4ZuWiUdsgyDE8lMirx9RlqLWlazy.umxr_JmLMZft7E3vf0MfROVxMvVdeiW54kBsfGdwug9HA.oAQ; path=/; expires=Mon, 06-May-24 14:58:46 GMT; domain=.domain.com; HttpOnly; Secure; SameSite=None"
              }
              ["vary"]=>
              array(1) {
                [0]=>
                string(15) "Accept-Encoding"
              }
              ["server"]=>
              array(1) {
                [0]=>
                string(10) "cloudflare"
              }
              ["cf-ray"]=>
              array(1) {
                [0]=>
                string(20) "87f9a89b7eb51d84-FRA"
              }
              ["content-encoding"]=>
              array(1) {
                [0]=>
                string(4) "gzip"
              }
            }
          }
          ["status_code"]=>
          int(403)
          ["protocol_version"]=>
          float(1.100000000000000088817841970012523233890533447265625)
          ["success"]=>
          bool(false)
          ["redirects"]=>
          int(0)
          ["url"]=>
          string(104) "https://domain.com/wp-admin/admin-ajax.php?action=async_litespeed&nonce=123123123&litespeed_type=crawler"
          ["history"]=>
          array(0) {
          }
          ["cookies"]=>
          object(WpOrg\Requests\Cookie\Jar)#19078 (1) {
            ["cookies":protected]=>
            array(1) {
              ["__cf_bm"]=>
              object(WpOrg\Requests\Cookie)#19086 (5) {
                ["name"]=>
                string(7) "__cf_bm"
                ["value"]=>
                string(149) "ZfmCfpiLGHk2E4aHIgTEdlwdUjdZaTliApiJuxGNqD0-1715005726-1.0.1.1-CV5SjLIl4ZuWiUdsgyDE8lMirx9RlqLWlazy.umxr_JmLMZft7E3vf0MfROVxMvVdeiW54kBsfGdwug9HA.oAQ"
                ["attributes"]=>
                object(WpOrg\Requests\Utility\CaseInsensitiveDictionary)#19085 (1) {
                  ["data":protected]=>
                  array(6) {
                    ["path"]=>
                    string(1) "/"
                    ["expires"]=>
                    int(1715007526)
                    ["domain"]=>
                    string(10) "domain.com"
                    ["httponly"]=>
                    bool(true)
                    ["secure"]=>
                    bool(true)
                    ["samesite"]=>
                    string(4) "None"
                  }
                }
                ["flags"]=>
                array(4) {
                  ["creation"]=>
                  int(1715005726)
                  ["last-access"]=>
                  int(1715005726)
                  ["persistent"]=>
                  bool(false)
                  ["host-only"]=>
                  bool(false)
                }
                ["reference_time"]=>
                int(1715005726)
              }
            }
          }
        }
        ["filename":protected]=>
        NULL
      }
    }
    
    

    Sorry, that was my mistake.

    Plugin Support qtwrk

    (@qtwrk)

        ["code"]=>
        int(403)
        ["message"]=>
        string(9) "Forbidden"
      }

    something blocked the admin-ajax.php , which is needed for crawler

Viewing 15 replies - 1 through 15 (of 19 total)

The topic ‘Crawler and http auth’ is closed to new replies.