LiteSpeed Crawler Adding All MOBILE Posts to Blocklist

@pzsniper please attach a report ID
Report can be created from LSC => Toolbox => Report => click on “Send to Litespeed” and copy/paste the ID created
Most of the cases we had about crawler blocklist was from code or plugin that flags the pages not cacheable.

Thank you for report and we will help debugging from here

x-litespeed-cache-control: no-cache

the log shows all returning this no-cache

please go to toolbox -> debug setting -> set debug log to ON , debug level to advance

debug include URI add first 3 URLs on your sitemap list

add

Media
Util
CSS
Optm
LQIP
Missing
missing
Avatar

to debug exclude string

then go to toolbox -> log view -> clean all existing log

now purge all , clean blocklist , and run crawler again , after it finished , assuming all mobile are blocklisted again

then go to log view , share the log that generated by crawler

it should be like this

the link seems not working, please verify

please try this:

edit your .htaccess

move the content between BEGIN LSCACHE and END NON_LSCACHE to the top of htaccess , make it first block in your htaccess

then purge all and try again

on your previous log , it has a line X Cache_control -> no Cache ( is mobile ) , and then you have set option cache mobile enabled

and from htaccess , there is rule RewriteRule .* - [E=Cache-Control:vary=%{ENV:LSCACHE_VARY_VALUE}+ismobile]

which detects and applies mobile vary , and here kicks in the tricky part

your old htaccess , has a line RewriteRule ^index\.php$ - [L] at beginning , like first block of rewrite rule , where the L stands for Last , by that saying , when rewrite engine encounters this L flag , it will be marked as last rewrite rule , and any further rules after this line will NOT be processed anymore , so the rule that applies mobile vary , did not get applied

in result , the plugin thinks this is a mobile page, but server doesn’t , which makes plugin goes haywire , and as fail-safe fallback , it rather marks page no-cache ,instead of caching it incorrectly

in short term , the position and order of rewrite rule , also matters.

for your last question , add

define('LITESPEED_CRAWLER_RUN_INTERVAL', 300);

into your wp-config.php , next line after <?php , where 300 is the interval time