Robots.txt file global /disallow

  • Resolved Ianc8647

    (@ianc8647)


    10 years, 2 months ago

    We setup this plugin on a number of new domains and after a few days noticed that none of them were being indexed. On closer checks we found that the generated /robots.txt files had populated as

    User-agent: *
    Disallow: /

    The Robots.txt file via is not available to edit via FTP nor was there any settings in the plugin that direct it to block the entire site.

    Perhaps a permissions issue where the plugin can’t write to the sever? We’ve manually added a robots.txt file and it appears to be okay now but adding Robots Meta Settings options are not being applied.

    https://ww.wp.xz.cn/plugins/autodescription/

Viewing 3 replies - 1 through 3 (of 3 total)
  • Plugin Author Sybre Waaijer

    (@cybr)

    10 years, 2 months ago

    Hi lanc8647,

    I hope you’re doing well today :), it’s good to see you’ve found a resolution.

    Thanks for all the detailed information, I took a look at the code (once again) and The SEO Framework will only return anything like that if the blog isn’t set to public.

    If you look closely at the letters I within your page/post SEO bars, are they blue and tell you that the blog isn’t set to public?

    If this is the case, please take a look at the WordPress Reading settings. You should Allow search engines to index your site right there :).

    /wp-admin/options-reading.php -> Site Visibility

    I hope this helps! Let me know if you require any more assistance 🙂

    P.S. The SEO Framework interacts with the virtual Robots.txt file set by WordPress Core, and applies filters on it.

    Thread Starter Ianc8647

    (@ianc8647)

    10 years, 2 months ago

    Thanks for the reply Sybre. We checked the I and the blue bars etc, but we didn’t think to check the always elusive “/wp-admin/options-reading.php -> Site Visibility” option. All fixed now. Thank you.

    Plugin Author Sybre Waaijer

    (@cybr)

    10 years, 2 months ago

    Anytime!

    Be sure to let the Search Engines know you’re back online:

    https://www.google.com/webmasters/tools/submit-url
    http://www.bing.com/toolbox/submit-site-url (Yahoo and DuckDuckGo listens to this one too)

    Good luck 🙂

Viewing 3 replies - 1 through 3 (of 3 total)

The topic ‘Robots.txt file global /disallow’ is closed to new replies.