unwanted robots.txt

  • hm2k

    (@hm2k)


    17 years, 11 months ago

    For whatever reason, when I visit http://www.example.com/robots.txt it displays the following:

    User-agent: *
Disallow:

    Yet, the file robots.txt does not exist, and in privacy settings I have…

    “I would like my blog to be visible to everyone, including search engines (like Google, Sphere, Technorati) and archivers” selected, and “I would like to block search engines, but allow normal visitors” deselected.

    The latter was previously selected, but I changed this, and upgraded from 2.3.3 to 2.6.

    I can’t work out how to get rid of this robots.txt file…

    Please help, this site is no longer listed in google because of this issue.

    Thanks.

Viewing 2 replies - 1 through 2 (of 2 total)
  • rawalex

    (@rawalex)

    17 years, 11 months ago

    You may want to check with your host… they may have put a global block in to limit bot traffic.

    Moderator Samuel Wood (Otto)

    (@otto42)

    ww.wp.xz.cn Admin

    17 years, 11 months ago

    1. That robots.txt is generated by WordPress.
    2. That particular robots.txt is not blocking anything, including Google. A blank Disallow: line means that everything is allowed. Look at the documentation on http://www.robotstxt.org/
    3. If you really want to get rid of it, make a blank robots.txt file and put it in your web root.

Viewing 2 replies - 1 through 2 (of 2 total)

The topic ‘unwanted robots.txt’ is closed to new replies.