Server Cron job for Crawler CLI Issues

Perfect it’s working now. Also i thought that the crawl interval setting only applies for how long you want to wait before a fresh entire sitemap crawl? since i’m using the server cron i thought running wp litespeed-crawler run was enough.

So when I increase the crawl interval, it doesn’t seem to work and just does the default Success: Start crawling. Current crawler #1 [position] 0 [total] 1884 like earlier.

This is what’s confusing me because i thought “Crawl Interval” means “how long to wait before the job crawls the entire sitemap again”. Not “how long to wait before the job runs normally”.?

Hi, it’s me again. i must be doing something wrong because out of 8 crawlers it seems to never erach the 3rd one but will always reset after the end of the 1st or 2nd and just loop those two crawlers throughout the entire night. I have also tried setting separate turn on/turn off cli server cron jobs to atleast try and get the other crawlers to start but to no avail.

any help would begreatly appreciated.

latest report number: MUISQFOB

Is there exists better documentation around the CLI crawler commands and the front-end settings?

For example if i have Crawler set to ON on the frontend but I am using CLI cron to enable, run and disable it, is having it set to ON here still necessary?

And what happens if i have this setting ON but am also using the CLI cron job?.

My debug log doesn’t show anything but it seems like setting the crawler interval to 61 is what is causing the position reset. This is my latest cronjob lines i do not have any reset added but i feel like the frontend settings are causing a clash?

#Enable crawlers at 7:30 PM NZDT (6:30 AM UTC) with logging

30 6 * * * wp litespeed-crawler list --path=/var/www/html | grep -oE '^[0-9]+' | xargs -I {} wp litespeed-crawler enable {} --path=/var/www/html && sleep 60 && wp litespeed-crawler run --path=/var/www/html >> /var/www/html/wp-content/lscronlog.txt 2>&1

# Disable crawlers at 6:00 AM NZDT (5:00 PM UTC) with logging

0 17 * * * wp litespeed-crawler list --path=/var/www/html | grep -oE '^[0-9]+' | xargs -I {} wp litespeed-crawler disable {} --path=/var/www/html >> /var/www/html/wp-content/lscronlog.txt 2>&1

So this is my current flow:

1. This setting inside wp-admin -> LiteSpeed Cache -> crawler -> general setting -> crawler ON
2. In my cron job I have a line to enable the crawlers, then sleep for 61, then run wp cli wp litespeed-crawler run

#Enable crawlers at 7:30 PM NZDT (6:30 AM UTC) with logging
30 6 * * * wp litespeed-crawler list --path=/var/www/html | grep -oE '^[0-9]+' | xargs -I {} wp litespeed-crawler enable {} --path=/var/www/html && sleep 61 && wp litespeed-crawler run --path=/var/www/html >> /var/www/html/wp-content/lscronlog.txt 2>&1
#
#
# Disable crawlers at 6:00 AM NZDT (5:00 PM UTC) with logging
0 17 * * * wp litespeed-crawler list --path=/var/www/html | grep -oE '^[0-9]+' | xargs -I {} wp litespeed-crawler disable {} --path=/var/www/html >> /var/www/html/wp-content/lscronlog.txt 2>&1

Does that mean it won’t work because I actually need this setting inside wp-admin -> LiteSpeed Cache -> crawler -> general setting -> crawler OFF ?

I am testing it now and let you know how it goes.