Adding robots.txt rules for certain bots

  • Resolved mmox

    (@mmox)


    2 years, 5 months ago

    Hi all,

    I’m trying to add certain rules to my robots.txt file with the filter references on the TSF website.

    This filter:

    // This is a WordPress Core filter. add_filter( 'robots_txt', function( $robots ) { $robots .= "\r\nDisallow: /my-custom-folder/\r\n"; return $robots; }, 11 );

    I need to add a specific allow/disallow for a different user agent for a SEO tool that I use to improve my website. Is this possible with this filter?

    I’m thinking along these lines:

    User-agent: SEO Bot
Allow: /shop/
Allow: /blog/
Allow: /help/
Disallow: /

    Would something like this be possible? I have certain URL’s I don’t want the bot to crawl, because I don’t need them in my SEO tool.

Viewing 1 replies (of 1 total)
  • Plugin Author Sybre Waaijer

    (@cybr)

    2 years, 5 months ago

    Hello!

    The filter you described should already work, and the method you described should also work.

    Here is it in full effect; it’s not the prettiest code, but it works:

    add_filter(
	'robots_txt',
	function( $robots ) {
		$robots .= '
User-agent: SEO Bot
Allow: /shop/
Allow: /blog/
Allow: /help/
Disallow: /
';

		return $robots;
	},
	11,
);

    Note that I do not encourage adjusting the robots.txt file to block pages from search engines; instead, one should rely on the “noindex” feature in TSF. To learn why, see https://developers.google.com/search/docs/crawling-indexing/robots/intro.

    • This reply was modified 2 years, 5 months ago by Sybre Waaijer. Reason: clarity
Viewing 1 replies (of 1 total)

The topic ‘Adding robots.txt rules for certain bots’ is closed to new replies.