Crawler works randomly

Resolved misterpo
(@misterpo)

7 months ago

Hi,

I am currently setting up LS Cache on some of our websites hosted at 02witch, a provider who supplies LiteSpeed Cache functionalities.

I can set up LS Cache with all the functionalities that I need (including Memcached), but when it turns to setting up the crawler (compulsory to have a preloaded cache ), the only thing I can say is that sometimes it works, sometimes it doesn’t.

I have exactly the same LS Cache plugin setup as well as hosting parameters (PHP versions and modules,….).

For instance, crawler works with:

https://www.conciergerie-annecy.fr/

And does not with:

https://mapetitecom.fr/

On the latest website, clicking on the Manual Execution button just leads to… nothing.

Crawler status is (in french) : Démarrage de l’observation.

Of course, my XML sitemap is correctly setup and discovered.

I really like the performance brought by LS Cache but I need to have a reliable crawler functionality to get rid of WP Rocket, which has become much to expensive.

Cheers.

The page I need help with: [log in to see the link]

Viewing 15 replies - 1 through 15 (of 27 total)

1 2 →

Plugin Support litetim
(@litetim)

7 months ago
@misterpo please create a report and share the ID generated.
You can generate the report from LSC => Toolbox => Report => click on “send to Litespeed” and share the generated ID.

Also, please enable debug in LSC(here) and look into Crawler Logs for the logs that are added.
Please share the logs(toolbox => log view) on sites like: https://pastebin.ubuntu.com/
- This reply was modified 7 months ago by litetim.
Thread Starter misterpo
(@misterpo)

7 months ago

Hi,

Thx for your feedback.

Report number is: DCCVZPKW

Logs: https://pastebin.ubuntu.com/p/ChYsfygf5p/

Plugin Support qtwrk
(@qtwrk)

7 months ago

the log is not showing complete message

please try add async_litespeed into debug include URI and try to grab again

Thread Starter misterpo
(@misterpo)

7 months ago

Hi,

Done.

Here is the new debug log : https://pastebin.ubuntu.com/p/WgNtQyQtwq/

Plugin Support qtwrk
(@qtwrk)

6 months, 4 weeks ago

the log doesn’t look alright

please confirm you have put this

and just in case you may need to clean up existing log first in order to get a clean log status

Thread Starter misterpo
(@misterpo)

6 months, 4 weeks ago

Hi,

The mentioned setting is correct.

Please find the link to the new log file : https://pastebin.ubuntu.com/p/3R6WZR4JNd/
Plugin Support qtwrk
(@qtwrk)

6 months, 4 weeks ago
yeah , it looks like the request to start the crawler never reached your site
```
<?php

require( './wp-load.php' );
$args = array(
  'timeout'   => 100,
  'blocking'  => true,
  'sslverify' => false,
  // 'cookies'   => $_COOKIE,
);

$url = 'https://' . $_SERVER['HTTP_HOST'] . '/wp-admin/admin-ajax.php?action=async_litespeed&nonce=123123123&litespeed_type=crawler';
$result = wp_remote_post(esc_url_raw($url), $args);

echo '<pre>';
var_dump($result);
echo '</pre>';
```
please create a php file, name it like test.php with above code , put it at your worpdress root directory

then open it as https://domain.com/test.php , in private window , see what it returns
Thread Starter misterpo
(@misterpo)

6 months, 4 weeks ago

Hi,

I have create a phpinfo.php file which returns the expected output:

https://mapetitecom.fr/phpinfo.php

Thread Starter misterpo
(@misterpo)

6 months, 4 weeks ago

Sorry, I have created the test.php file which leads to a never ending loop

https://mapetitecom.fr/test.php

I have temporarily disabled preloader, please find below the output:

array(6) { [“headers”]=> object(WpOrg\Requests\Utility\CaseInsensitiveDictionary)#2563 (1) { [“data”:protected]=> array(9) { [“date”]=> string(29) “Fri, 31 Oct 2025 07:54:12 GMT” [“content-type”]=> string(24) “text/html; charset=UTF-8” [“vary”]=> string(15) “Accept-Encoding” [“x-dns-prefetch-control”]=> string(2) “on” [“link”]=> string(208) “; rel=”https://api.w.org/”, ; rel=”alternate”; title=”JSON”; type=”application/json”, ; rel=shortlink” [“x-litespeed-cache-control”]=> string(8) “no-cache” [“cache-control”]=> string(55) “no-cache, must-revalidate, max-age=0, no-store, private” [“server”]=> string(22) “o2switch-PowerBoost-v3” [“content-encoding”]=> string(4) “gzip” } } [“body”]=> string(188321) “
Plugin Support qtwrk
(@qtwrk)

6 months, 4 weeks ago
yeah , that request is supposedly to return “0” , but in your case
```
  ["body"]=>
  string(188771) "<!DOCTYPE html>
<html lang="fr-FR">
<head itemscope="itemscope" itemtype="http://schema.org/WebSite">

	<meta charset="UTF-8">
	<meta name="viewport" content="width=device-width, initial-scale=1.0" />
```
it’s hitting a html page

which means something blocked or redirected this request, you need to undo that first
Thread Starter misterpo
(@misterpo)

6 months, 4 weeks ago

I have the following code in .htaccess file. Is it expected ?

marker ASYNC start

RewriteCond %{REQUEST_URI} /wp-admin/admin-ajax.php
RewriteCond %{QUERY_STRING} action=async_litespeed
RewriteRule .* – [E=noabort:1] marker ASYNC end

Plugin Support qtwrk
(@qtwrk)

6 months, 4 weeks ago

yes , this is expected, something else blocked or redirected
Thread Starter misterpo
(@misterpo)

6 months, 3 weeks ago
Hi,

I finally worked it out: crawler does not work on websites with Imagify plugin installed in rewrite rule mode.

Here is the set of .htaccess rules added by Imagify:

BEGIN Imagify: webp file type

AddType image/webp .webp END Imagify: webp file type BEGIN Imagify: avif file type

AddType image/avif .avif END Imagify: avif file type BEGIN Imagify: rewrite rules for avif

# Vary: Accept for all the requests to jpeg, png, and gif. SetEnvIf Request_URI “.(jpg|jpeg|jpe|png|gif|webp)$” REQUEST_image

RewriteEngine On
RewriteBase /
```
# Check if browser supports AVIF images.
# Update the MIME type accordingly.
RewriteCond %{HTTP_ACCEPT} image/avif

# Check if AVIF replacement image exists.
RewriteCond %{REQUEST_FILENAME}.avif -f

# Serve AVIF image instead.
RewriteRule (.+)\.(jpg|jpeg|jpe|png|gif|webp)$ $1.$2.avif [T=image/avif,NC]
```
# Update the MIME type accordingly. Header append Vary Accept env=REQUEST_image END Imagify: rewrite rules for avif BEGIN Imagify: rewrite rules for webp

# Vary: Accept for all the requests to jpeg, png, and gif. SetEnvIf Request_URI “.(jpg|jpeg|jpe|png|gif)$” REQUEST_image

RewriteEngine On
RewriteBase /
```
# Check if browser supports WebP images.
RewriteCond %{HTTP_ACCEPT} image/webp

# Check if WebP replacement image exists.
RewriteCond %{REQUEST_FILENAME}.webp -f

# Serve WebP image instead.
RewriteRule (.+)\.(jpg|jpeg|jpe|png|gif)$ $1.$2.webp [T=image/webp,NC]
```
Header append Vary Accept env=REQUEST_image END Imagify: rewrite rules for webp

Any idea of what affects /wp-admin/admin-ajax.php?action=async_litespeed&nonce=123123123&litespeed_type=crawler request?
Thread Starter misterpo
(@misterpo)

6 months, 3 weeks ago

I confirm your say regarding O2switch.

However, LS crawler doesn’t work only when Imagify is installed, I have made a benchmark of nearly 15 websites.

Plugin Support litetim
(@litetim)

6 months, 3 weeks ago

@misterpo can you make a test? disable Imagify and test the crawler a few days(at least until next run). See if this changes anything.

Viewing 15 replies - 1 through 15 (of 27 total)

1 2 →

The topic ‘Crawler works randomly’ is closed to new replies.