Crawler won’t start

neuropassenger
(@neuropassenger)

5 months, 2 weeks ago
Hello,

I’m having a problem with the crawler. It just won’t start, even though the entire site map has been scanned correctly.

It simply displays the following information and cannot start:
- Start watching…
- 01 Jan 1970 04:00:00 Size: 1661 Crawler: #1 Position: 1 Threads: 0 Status: crawling, prepare running
I have sent you a report with more detailed information about the website and server with id PEHKWNUT.

What do you think could be the reason for this? And how can it be fixed?
Thank you!

Viewing 8 replies - 1 through 8 (of 8 total)

Thread Starter

(@neuropassenger)

In addition, I am experiencing a problem with receiving images (see screenshot for details). ВThe process starts, but no progress is visible – the “Pull Cron is running” notification remains constantly, and the number of pulled files still does not increase. The “Pull images” button does not help either.

I’m not sure, but perhaps these two problems are somehow related.

Plugin Support

litetim

(@litetim)

5 months, 2 weeks ago

@neuropassenger is the server managed by you? is the server time set correctly?

Plugin Support

qtwrk

(@qtwrk)

5 months, 2 weeks ago

<?php

require( './wp-load.php' );
$args = array(
  'timeout'   => 100,
  'blocking'  => true,
  'sslverify' => false,
  // 'cookies'   => $_COOKIE,
);

$url = 'https://' . $_SERVER['HTTP_HOST'] . '/wp-admin/admin-ajax.php?action=async_litespeed&nonce=123123123&litespeed_type=crawler';
$result = wp_remote_post(esc_url_raw($url), $args);

echo '<pre>';
var_dump($result);
echo '</pre>';

please create a php file, name it like test.php , with above code, put it at same dir as your wp-config.php

then open it in a private window as https://domain.com/test.php , and share what it returns.

Thread Starter

neuropassenger

(@neuropassenger)

5 months, 2 weeks ago

@litetim, yes, the server is managed by me and time is set correctly.

               Local time: Fri 2025-12-12 19:39:08 UTC
           Universal time: Fri 2025-12-12 19:39:08 UTC
                 RTC time: Fri 2025-12-12 19:39:08
                Time zone: Etc/UTC (UTC, +0000)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no

@qtwrk,

array(6) {
  ["headers"]=>
  object(WpOrg\Requests\Utility\CaseInsensitiveDictionary)#24642 (1) {
    ["data":protected]=>
    array(15) {
      ["content-type"]=>
      string(24) "text/html; charset=UTF-8"
      ["x-robots-tag"]=>
      string(7) "noindex"
      ["x-content-type-options"]=>
      string(7) "nosniff"
      ["referrer-policy"]=>
      string(31) "strict-origin-when-cross-origin"
      ["x-frame-options"]=>
      string(10) "SAMEORIGIN"
      ["content-security-policy"]=>
      string(23) "frame-ancestors 'self';"
      ["expires"]=>
      string(29) "Wed, 11 Jan 1984 05:00:00 GMT"
      ["cache-control"]=>
      string(55) "no-cache, must-revalidate, max-age=0, no-store, private"
      ["x-litespeed-cache-control"]=>
      string(8) "no-cache"
      ["content-encoding"]=>
      string(4) "gzip"
      ["vary"]=>
      string(15) "Accept-Encoding"
      ["content-length"]=>
      string(2) "21"
      ["date"]=>
      string(29) "Fri, 12 Dec 2025 19:46:31 GMT"
      ["server"]=>
      string(9) "LiteSpeed"
      ["alt-svc"]=>
      string(167) "h3=":443"; ma=2592000, h3-29=":443"; ma=2592000, h3-Q050=":443"; ma=2592000, h3-Q046=":443"; ma=2592000, h3-Q043=":443"; ma=2592000, quic=":443"; ma=2592000; v="43,46""
    }
  }
  ["body"]=>
  string(0) ""
  ["response"]=>
  array(2) {
    ["code"]=>
    int(200)
    ["message"]=>
    string(2) "OK"
  }
  ["cookies"]=>
  array(0) {
  }
  ["filename"]=>
  NULL
  ["http_response"]=>
  object(WP_HTTP_Requests_Response)#24641 (5) {
    ["data"]=>
    NULL
    ["headers"]=>
    NULL
    ["status"]=>
    NULL
    ["response":protected]=>
    object(WpOrg\Requests\Response)#24643 (10) {
      ["body"]=>
      string(0) ""
      ["raw"]=>
      string(714) "HTTP/1.1 200 OK
content-type: text/html; charset=UTF-8
x-robots-tag: noindex
x-content-type-options: nosniff
referrer-policy: strict-origin-when-cross-origin
x-frame-options: SAMEORIGIN
content-security-policy: frame-ancestors 'self';
expires: Wed, 11 Jan 1984 05:00:00 GMT
cache-control: no-cache, must-revalidate, max-age=0, no-store, private
x-litespeed-cache-control: no-cache
content-encoding: gzip
vary: Accept-Encoding
content-length: 21
date: Fri, 12 Dec 2025 19:46:31 GMT
server: LiteSpeed
alt-svc: h3=":443"; ma=2592000, h3-29=":443"; ma=2592000, h3-Q050=":443"; ma=2592000, h3-Q046=":443"; ma=2592000, h3-Q043=":443"; ma=2592000, quic=":443"; ma=2592000; v="43,46"
connection: close

0"
      ["headers"]=>
      object(WpOrg\Requests\Response\Headers)#24644 (1) {
        ["data":protected]=>
        array(15) {
          ["content-type"]=>
          array(1) {
            [0]=>
            string(24) "text/html; charset=UTF-8"
          }
          ["x-robots-tag"]=>
          array(1) {
            [0]=>
            string(7) "noindex"
          }
          ["x-content-type-options"]=>
          array(1) {
            [0]=>
            string(7) "nosniff"
          }
          ["referrer-policy"]=>
          array(1) {
            [0]=>
            string(31) "strict-origin-when-cross-origin"
          }
          ["x-frame-options"]=>
          array(1) {
            [0]=>
            string(10) "SAMEORIGIN"
          }
          ["content-security-policy"]=>
          array(1) {
            [0]=>
            string(23) "frame-ancestors 'self';"
          }
          ["expires"]=>
          array(1) {
            [0]=>
            string(29) "Wed, 11 Jan 1984 05:00:00 GMT"
          }
          ["cache-control"]=>
          array(1) {
            [0]=>
            string(55) "no-cache, must-revalidate, max-age=0, no-store, private"
          }
          ["x-litespeed-cache-control"]=>
          array(1) {
            [0]=>
            string(8) "no-cache"
          }
          ["content-encoding"]=>
          array(1) {
            [0]=>
            string(4) "gzip"
          }
          ["vary"]=>
          array(1) {
            [0]=>
            string(15) "Accept-Encoding"
          }
          ["content-length"]=>
          array(1) {
            [0]=>
            string(2) "21"
          }
          ["date"]=>
          array(1) {
            [0]=>
            string(29) "Fri, 12 Dec 2025 19:46:31 GMT"
          }
          ["server"]=>
          array(1) {
            [0]=>
            string(9) "LiteSpeed"
          }
          ["alt-svc"]=>
          array(1) {
            [0]=>
            string(167) "h3=":443"; ma=2592000, h3-29=":443"; ma=2592000, h3-Q050=":443"; ma=2592000, h3-Q046=":443"; ma=2592000, h3-Q043=":443"; ma=2592000, quic=":443"; ma=2592000; v="43,46""
          }
        }
      }
      ["status_code"]=>
      int(200)
      ["protocol_version"]=>
      float(1.1)
      ["success"]=>
      bool(true)
      ["redirects"]=>
      int(0)
      ["url"]=>
      string(121) "https://domain.com/wp-admin/admin-ajax.php?action=async_litespeed&nonce=123123123&litespeed_type=crawler"
      ["history"]=>
      array(0) {
      }
      ["cookies"]=>
      object(WpOrg\Requests\Cookie\Jar)#24640 (1) {
        ["cookies":protected]=>
        array(0) {
        }
      }
    }
    ["filename":protected]=>
    NULL
  }
}

Plugin Support

qtwrk

(@qtwrk)

5 months, 2 weeks ago

please go to toolbox -> debug setting, set debug log to ON, debug level to advance , debug include uri add async_litespeed

then go to toolbox -> log view -> clean up existing log if any

then go to crawler summary page again , click “manually run” , you may need to click this twice or thrice

wait for 5 minutes or so , go to log view again , copy paste and share log via https://pastebin.ubuntu.com/

Thread Starter

neuropassenger

(@neuropassenger)

5 months, 2 weeks ago

@qtwrk,

First, I pressed the Manual Run button once every 5 minutes (I did this 3 times). At the very end, I pressed this button 3 times in a row. Then I waited 5 minutes.

Logs

Regarding server load (we have 8 CPU cores):

07:33:43 up 5 days, 17:15, 2 users, load average: 1.48, 1.67, 1.72