Blocking archive.org/wayback machine | ww.wp.xz.cn

Resolved mcheeks
(@mcheeks)

5 years, 9 months ago

Does WordFence block archive.org/wayback machine from crawling sites? I noticed that my site hasn’t been crawled since June 2019 by archive.org. Is there a way to whitelist their crawlers or IPs?

The page I need help with: [log in to see the link]

Viewing 1 replies (of 1 total)

Plugin Support wfphil
(@wfphil)

5 years, 9 months ago

Hi @mcheeks

Wordfence does not block the Internet Archive crawler, unless you have blocked it of course.

You can see their crawling guide here:

https://help.archive.org/hc/en-us/articles/360004651732-Using-The-Wayback-Machine

You can see the user agent that they use here:

https://archive.org/details/archive.org_bot

You can use the advanced Tools >> Live Traffic filter below to see if you have blocked the Internet Archive crawler:

User Agent contains archive.org_bot

If you have blocked it you will see the reason why and you can change or remove your blocking rule.

Viewing 1 replies (of 1 total)

The topic ‘Blocking archive.org/wayback machine’ is closed to new replies.

1 reply
2 participants
Last reply from: wfphil
Last activity: 5 years, 9 months ago
Status: resolved