• Resolved wazifalab

    (@wazifalab)


    I have double user agents in my website robots.txt file,
    The second one was generated by wp optimize plugin automatically, how may i keep the rule and delete the second agent?

    The page I need help with: [log in to see the link]

Viewing 1 replies (of 1 total)
  • Plugin Contributor Venkat Raj

    (@webulous)

    @wazifalab The no. of user agents doesn’t affect your site. Each user agent directive only applies to the rules that follows them. See documentation page where two user agents are present
    https://developers.google.com/search/docs/crawling-indexing/robots/create-robots-txt

    By default, there is no robots.txt file in WP setup. Mostly SEO plugins create them. What if the plugin that created it, also removes it in later stage? The Disallow rule will be there without any user agent.

    If you want to remove the rules added by WP Optimize, add the following code in your theme’s functions.php



    add_action('after_setup_theme', 'ab123_remove_robots_txt_entry');
    function ab123_remove_robots_txt_entry() {
      if (function_exists('WP_Optimize')) {
        remove_filter('robots_txt', array(WP_Optimize(), 'robots_txt'), 99, 1);
      }
    }
Viewing 1 replies (of 1 total)

The topic ‘double user agents’ is closed to new replies.