Crawlers stuck in loop
-
Crawlers have been looping over page numbers in the Photo Galleries
I have fixed this issue but I noticed on my site I was getting hundreds of crawlers requests on the gallery page. I had multiple galleries on one page and i was getting 100’s if not 1000’s of these requests from Amazon and Open AI bots. This was pushing my CPU usage from around 3% to 60%.
I have since blocked crawlers from crawling the pages in robots with this
START YOAST BLOCK —————————
User-agent: *
Disallow: /our-work/?page_number*YOAST BLOCK
172.70.43.108 – – [09/Jun/2025:08:51:04 +1000] “GET /our-work/?page_number_1=4&page_number_2=5&page_number_5=1&page_number_6=3 HTTP/2.0” 200 23294 “-” “Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot) Chrome/119.0.6045.214 Safari/537.36” “107.20.255.194”
172.71.223.130 – – [09/Jun/2025:08:51:00 +1000] “GET /our-work/?page_number_0=3&page_number_1=10&page_number_3=6&page_number_4=2 HTTP/2.0” 200 23631 “-” “Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot) Chrome/119.0.6045.214 Safari/537.36” “18.208.11.93”
The topic ‘Crawlers stuck in loop’ is closed to new replies.