Crawler not functioning properly

  • Resolved thomasgoshaka

    (@thomasgoshaka)


    11 months, 1 week ago

    The crawler never finishes an entire run. It always gets stuck after a while and it is unclear what is exactly happening. It always lists the reason “stopped_ reset” .

    The output looks like it stopped but both the table and the crawler status indicate it is running.
    I know that it is not actually running as no new page has been crawled in the last 45 minutes.

    Things I have already tried:
    – Setting the server load limit to max.
    – Increasing crawler interval to a day
    – Increasing the TTL of the actual cache to a week

    I have not been able to get a full crawler run after weeks of trying and it always stops with the reason “stopped_reset”. Also apart from another post on this forum I could not find any information about this issue online.

    Report Nr: VKJZJNEE

    EDIT: The screenshot did not work so it but it used to say running position: 48 running and the live crawler status shows it was running as well

    And this would be the dashboard info

    Last complete run time for all crawlers: 1749827851 seconds

    Run time for previous crawler: 55751 seconds

    Current crawler started at: 33m 29s ago

    Current server load: 4.99

    Last interval: 33m 29s ago

    Ended reason: stopped_reset

    Last crawled: 47 item(s)

    • This topic was modified 11 months, 1 week ago by thomasgoshaka.

    The page I need help with: [log in to see the link]

Viewing 3 replies - 1 through 3 (of 3 total)
  • Plugin Support qtwrk

    (@qtwrk)

    11 months, 1 week ago

    stop reset means something called purge all , so it stops crawler, and by your setting, the crawler will not run again in 86400 seconds later , you can try manually click the “manual run” button few times , it should then launch the crawler again before its due time.

    Thread Starter thomasgoshaka

    (@thomasgoshaka)

    11 months, 1 week ago

    Hi,

    Thanks for the quick reply!
    It does happen quite often when I think nobody changed anything on the site in the meantime.
    What are the most likely culprits of resetting the cache/calling purge all and can I find what or who called it in the logs somewhere?

    Regards,

    Thomas

    Plugin Support qtwrk

    (@qtwrk)

    11 months, 1 week ago

    please go to toolbox -> debug setting -> set debug log to on , debug level to advance

    then go to toolbox -> log view , clean up existing log

    then give it a day or two , then go to toolbox -> log view -> purge log , there should have something

Viewing 3 replies - 1 through 3 (of 3 total)

The topic ‘Crawler not functioning properly’ is closed to new replies.