Crawler not working

  • Resolved christophercrypto

    (@christophercrypto)


    6 months, 3 weeks ago

    Subject: LSCWP Crawler always stops after 3 URLs — reason stopped_reset (no manual reset/purge)

    Hosting: Oderland (LiteSpeed Enterprise)
    Plugin: LiteSpeed Cache for WordPress 7.6.2
    WordPress: 6.8.3 Summary

    The LiteSpeed crawler never completes a run. It consistently crawls exactly 3 URLs and then stops with reason stopped_reset. This repeats whether started by cron or manually, and even if we leave it untouched for hours (no admin actions while it runs).

    I understand from the docs that stopped_reset means the crawler’s position was reset while running (e.g., map refresh or purge). However, we are not triggering any manual resets or purges during the run, and the behavior is 100% reproducible.

    We need help identifying why it is being reset and how to make the crawler complete. Steps to reproduce

    1. LSCache → Crawler → Cron → click Run Manually (or wait for cron).
    2. Status shows the run starting; it crawls 3 items (we have 3 threads enabled).
    3. The run stops and the top of the page shows:
      “End Reason: stopped_reset”

    If we leave it for 8+ hours, the same pattern repeats (3 pages → stop). What we’ve tried

    • Verified sitemap works: (506 URLs listed).
    • Map tab:
      • Clicked Cache Map and then Use Cached Map; same result.
      • Blocklist is empty.
    • Cron: We replaced WP-Cron with a real system cron that runs
      /usr/local/bin/php -q /home/greenbal/public_html/wp-cron.php
      (Oderland’s panel). The crawler still stops at 3 URLs with stopped_reset.
    • Load limit: Increased crawler Load Limit from 1 to 16 (as suggested by LSCWP UI/host variable); no change.
    • Let it run unattended: Kept the admin logged out and did not purge/reset/update anything for hours; the issue persists.
    • General: No visible server load spikes (shown in the Crawler screen). Crawler blocklist empty.

    Environment highlights (from LS Report)

    • SERVER_SOFTWARE = LiteSpeed (Enterprise)
    • X-LSCACHE = on,crawler
    • LSCWP version: 7.6.2
    • WP version: 6.8.3
    • Crawler: enabled
      • crawler-crawl_interval = 86400
      • crawler-load_limit = 16
      • crawler-sitemap = https://www.greenbalance.se/sitemap_index.xml
    • Object Cache: on (memcached path /tmp/memcached)
    • Expected vs actual
    • Expected: Crawler proceeds through the map and warms the cache.
    • Actual: Every run stops after 3 URLs with stopped_reset.

    Questions / Help Requested

    1. Besides manual reset/purge, what specific events will trigger stopped_reset?
    2. Is there any condition where automatic or background purges (e.g., from other plugins) silently reset the crawler even when using a cached map?
    3. Which logs should we capture to pinpoint the reset source?
    4. Are there any known issues in 7.6.2 that can cause stopped_reset repeatedly?

    We’re happy to provide the full Environment Report, the litespeed_purge.log and litespeed_crawler.log right after a failing run, or temporary admin access if required.

    Thanks in advance!

    The page I need help with: [log in to see the link]

Viewing 5 replies - 1 through 5 (of 5 total)
  • joy0114

    (@joy0114)

    6 months, 2 weeks ago

    Hi,

    Have you tried clicking the “run manually” button again ?

    Have you set a “LITESPEED_CRAWLER_RUN_INTERVAL” like this (for example, for 5 minutes):

    define(‘LITESPEED_CRAWLER_RUN_INTERVAL’, 300);

    This is not the same as crawler-crawl_interval = 86400.

    This setting must be made in wp-config.php.

    With that, without clicking “Run Manually” again, the crawler will resume in 5 minutes.
    For testing purposes, you can, of course, reduce this time.

    Best regards

    Thread Starter christophercrypto

    (@christophercrypto)

    6 months, 2 weeks ago

    Unfortunately I have mashed the “run manually” button until my fingers bled. Does nothing.

    I attempted to change the interval as per your suggestion to 1 minute now, but the crawler still never progresses beyond position #1.

    However, while troubleshooting I found a rather unorthodox “solution”, of course this isn’t sustainable in the long run, but I can crawl just fine on my staging site, and if I input my production site’s sitemap on my staging site, then I can fully warm all of the URLS.

    Which, of course makes even less sense because the sites are functionally identical, so there’s no reason it should work there to begin with. But either way, doesn’t actually solve my issue, but does provide a bandaid until I can find an actual fix.

    Plugin Support qtwrk

    (@qtwrk)

    6 months, 2 weeks ago

    auto purge will trigger reset as well

    if it always happens after 3 URLs, you should be able to log the whole event during crawler with debug log , let’s see what log says

    Thread Starter christophercrypto

    (@christophercrypto)

    6 months, 2 weeks ago

    Couldn’t for the life of me tell you why, but for whatever reason after a little while the reduced interval seems to have kickstarted the crawler. It has now crawled successfully for a couple of days.

    Thanks @joy0114 !

    joy0114

    (@joy0114)

    6 months, 2 weeks ago

    Good news !
    Glad to have been of assistance, and thank you for your feedback.

    Best regards

Viewing 5 replies - 1 through 5 (of 5 total)

The topic ‘Crawler not working’ is closed to new replies.