Block this user agent

  • Resolved Earl_D

    (@earl_d)


    2 months, 3 weeks ago

    How do I block the user agent meta-webindexer/1.1 I do not want FB AI indexing my sites but it isn’t in the drop down list for user agents and uses multiple IP addresses so I am constantly blocking IP addresses. It is already in my robots.txt. There should be an option to add user agents that are not in the drop down list especially since new AI indexers are emerging all the time and overloading web sites as they scrape sites

Viewing 3 replies - 1 through 3 (of 3 total)
  • Plugin Support eugenecleantalk

    (@eugenecleantalk)

    2 months, 3 weeks ago

    Hello earl_d,

    As we mentioned in another topic, it is possible to block user agents using robots.txt. The official Meta article also mentions this: https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/. Please double-check that you’ve set up the rule correctly.

    I have forwarded your request to our developers regarding the possibility of blocking the meta-webindexer using our personal lists. We will respond to you within 3 business days.

    Plugin Support amagsumov

    (@amagsumov)

    2 months, 2 weeks ago

    Hello @earl_d,

    We’ve added the possibility to block the mentioned crawler by its User Agent via the Anti-Crawler feature. Please follow this guide: https://cleantalk.org/help/filter-ua

    Below is the list of User Agents available for blocking:

    facebookexternalhit/1.1
    meta-webindexer/1.1
    meta-externalads/1.1
    meta-externalagent/1.1
    meta-externalfetcher/1.1

    Does it help?

    Plugin Support katereji

    (@katereji)

    2 months, 1 week ago

    Hello.
    We haven’t heard back from you in a few days, so I’m going to mark this thread as “resolved”.
    If you have any further questions, you can start a new thread or contact us via our private Ticket System: https://cleantalk.org/my/support/open.

Viewing 3 replies - 1 through 3 (of 3 total)

You must be logged in to reply to this topic.