Ended reason: stopped_maxtime. The crawler hangs at status: crawling, updated position. I have tried to start it manually twice in a row, nothing happens.
I have set Crawl Interval to 61 seconds. Nothing happens.
The WP-Cron calls the litespeed_task_crawler correctly.
I’m almost at the end of my ideas. Especially because the test page ran yesterday, even if only once. Sometimes the crawler starts again after an hour. With 6.1 it took 6 hours and the crawling was completely finished.
Here is an update. The crawler has run through and it took a total of 17 hours (6.2). This is because the crawler kept stopping for an hour or two.
Under 6.0 the crawler started, ran for 6 hours and the page was crawled completely.
I monitored this yesterday. The crawler started for 600 seconds, as it is set. It then waited correctly for 100 seconds and then started again. Then after a few seconds it stopped and the crawler was terminated with maxtime. Then it started again an hour later.
As I wrote, the value for interval settings is 100 seconds! It starts again after 100 seconds, but then simply stops and runs again after an hour. This should not happen according to the settings. Run Duration is 600 seconds, I have also tested 1800. The set values are ignored.
from your report , this means , let’s say at a day 00:00 , crawler starts , assuming no other factors like purge or overload , then
00:00 -> run 600 seconds -> 00:10 -> stop for 61 seconds -> 00:11 -> run another 600 seconds -> 00:21 -> stop for 61 seconds -> 00:22 -> run another 600 seconds , and so on , until , let’s say 02:22 it finished current run
then at 02:22 -> wait for 3600 seconds -> 03:22 -> it starts to run again and repeat above process as 00:00
I understood the theory and that’s how it is under 6.0. But under 6.2 it is the case that in this example:
00:00 -> run 600 seconds -> 00:10 -> stop for 61 seconds -> 00:11 -> run another 600 seconds -> 00:21 -> stop for 61 seconds -> 00:22 -> run another 600 seconds , and so on
So far 70,000 pages have been crawled in 6 hours (6.0). Now it takes 17 hours! (6.2)
But the crawler stops after 30 minutes, or after an hour and then continues somewhere else. But it should continue until the end and not wait for hours after maxtime until it continues crawling.
the crawling will also dependent on your server performance
the crawler itself is same, doing request to pages it grabs from sitemap , once the request is sent, it’s server’s job to process the request
as far as the crawler’s functionaility concern , it does same job on all versions , I can’t really think any possible reason crawler takes 3 times the difference , only possible case is the crawler did not send the request , but in that case, you will need to enable debug log and see how crawler acts
