Issue with wordpress and robots.txt

  • millerb7

    (@millerb7)


    15 years, 1 month ago

    So my google webmaster account is telling me that my robots.txt is disallowing everything… yet it’s not. I can either set it to allow (which basically is the same as having no robots.txt) or I can even have NO robots.txt….. and for some reason google thinks my wordpress site has one…. any thoughts on this?

    I am allowing things through privacy setting… and my SEO plugin is allowing bots as well… and like I said, I manually removed my robots.txt.

Viewing 3 replies - 1 through 3 (of 3 total)
  • Moderator Kathryn Presner

    (@zoonini)

    15 years, 1 month ago

    Can we see your robots.txt? Either paste a link (ideally, so we can also check the path) or use a Pastebin.

    Thread Starter millerb7

    (@millerb7)

    15 years, 1 month ago

    Well right now there is no robots.txt.

    It used to say:
    User-agent: *
    Disallow: /

    Then after I was done building I swapped it to:
    User-agent: *
    Allow: /

    Only because I used googles webmaster tools to generate the robots.txt to allow the robots. That’s how it does it.

    Right now I have completely removed it, as having it say “allow” is just the same.

    It says it was read last (via webmaster tools crawler into) 17 hours ago. It updates robots.txt approximately every 24. I’m assuming I have to wait for the next crawl, although I’ve never run into this issue before.

    Moderator Kathryn Presner

    (@zoonini)

    15 years, 1 month ago

    It used to say:
User-agent: *
Disallow: /

    Well that explains why you were getting warnings in your Google Webmaster Tools – that code was telling all search engines not to spider your site at all.

    You need to wait until Google has re-indexed your site and then see what it says in Webmaster Tools – it should be happier now, though it will be happiest seeing a robots.txt file of some kind. I recommend you put back a default robots.txt – search engines like to see them, even if they don’t say much.

    Robots.txt syntax

Viewing 3 replies - 1 through 3 (of 3 total)

The topic ‘Issue with wordpress and robots.txt’ is closed to new replies.