It seems to be fixed now.
It appears that the crawler picked up a URL that was excluded from the sitemap (because it is a redirected URL), and that was causing the problem.
Apparently, even though it didn’t appear in the sitemap, the crawler had decided to crawl that URL.
This has never happened before… but now I have explicitly excluded this URL from the crawler map, and so it seems that it no longer stops at position 220.
Now the crawler no longer crashes.
Sorry for the false alarm…
@joy0114 hmmm
What was the link added to sitemap?
What plugin you use for sitemap?
-
This reply was modified 6 months, 1 week ago by
litetim.
@joy0114 can you help me with this info?
Thank you!
Hi Litetim,
Sorry for this late reply, Notifications for new replies rarely work for me: it’s quite random…
The URL causing the issue is https://lesbijouxdelhermine.fr/nos-creations/ which is redirected towards https://lesbijouxdelhermine.fr/toutes-nos-creations/
Sitemap is generated by SEOpress (pro release), and does NOT contain /nos-creations/, but contains /toutes-nos-creations/ (I verified).
To ensure that the crawler continues and does not get stuck on this first URL, I added it to the block list.
What is strange is that the problem does not occur with the same page in English, which is also redirected to another page in English:
/our-crafts/ does not block the crawler, even though it is also redirected (to /all-our-creations/ ).
/our-crafts/ is not in the crawler map, as expected, since this URL is not in the sitemap.
It’s not a big deal since I figured out how to get around this block, but I’m more bothered by a purge issue for ONE url.
I’ll make another post about that.
Thanks anyway for paying attention to my concern, even if it’s trivial and easily bypassed.
Best regards
@joy0114 how is the redirect added?
Default crawler behaviour is to add it to blocklist and mention 301 response to the source link.
Can you look into logs and see what error is showing?
