• whodata

    (@whodata)


    Crawlers have been looping over page numbers in the Photo Galleries

    I have fixed this issue but I noticed on my site I was getting hundreds of crawlers requests on the gallery page. I had multiple galleries on one page and i was getting 100’s if not 1000’s of these requests from Amazon and Open AI bots. This was pushing my CPU usage from around 3% to 60%.

    I have since blocked crawlers from crawling the pages in robots with this

    START YOAST BLOCK —————————

    User-agent: *
    Disallow: /our-work/?page_number*

    YOAST BLOCK



    172.70.43.108 – – [09/Jun/2025:08:51:04 +1000] “GET /our-work/?page_number_1=4&page_number_2=5&page_number_5=1&page_number_6=3 HTTP/2.0” 200 23294 “-” “Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot) Chrome/119.0.6045.214 Safari/537.36” “107.20.255.194”
    172.71.223.130 – – [09/Jun/2025:08:51:00 +1000] “GET /our-work/?page_number_0=3&page_number_1=10&page_number_3=6&page_number_4=2 HTTP/2.0” 200 23631 “-” “Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot) Chrome/119.0.6045.214 Safari/537.36” “18.208.11.93”

Viewing 2 replies - 1 through 2 (of 2 total)
  • Plugin Support Yasin Abedi

    (@yasinabedi)

    Hello @whodata ,
    Thank you for your post,
    Could you please navigate to Photo Gallery -> Global Settings , and let us know the value for these 2 options
    Enable dynamic URLs for galleries and gallery groups
    Discourage Search Engine Visibility
    and if they are set as in this screenshot ?

    Looking forward to hearing from you!
    Best Regards,
    Yasin

    Thread Starter whodata

    (@whodata)

    Yes my settings are the same as that screenshot

Viewing 2 replies - 1 through 2 (of 2 total)

The topic ‘Crawlers stuck in loop’ is closed to new replies.