Title: robot.txt question
Last modified: August 18, 2016

---

# robot.txt question

 *  [startribe](https://wordpress.org/support/users/startribe/)
 * (@startribe)
 * [19 years, 1 month ago](https://wordpress.org/support/topic/robottxt-question/)
 * Hey Community,
 * I am putting together a robot.txt file, and I wanted to confirm something in 
   regards to some advice from this link:
    < a href=”[http://www.askapache.com/seo/wordpress-robotstxt-optimized-for-seo.html”&gt](http://www.askapache.com/seo/wordpress-robotstxt-optimized-for-seo.html”&gt);
   wordpress-robotstxt-optimized-for-seo
 * I noticed the disallow command for these locations:
    Disallow: /cgi-bin/ Disallow:/
   z/j/ Disallow: /z/c/ Disallow: /stats/ Disallow: /dh_
 * These are not located in my html folder, i.e., my server goes something like 
   this:
    var/www/html
 * Now would I need to locate and change the paths to these if they are not in my
   html public folder?
 * Also, for the disallow location of the wp directories:
    Disallow: /wp-admin/ 
   Disallow: /wp-includes/ & etc…
 * Should I update my path to stem from my html/web folder to the location of these
   directories, e.g., if my wp files are located in “content” then change:
    /wp-
   admin/ to /content/wp-admin/
 * Any advice is greatly appreciated.
 * Sincerely,
    Orion

Viewing 8 replies - 1 through 8 (of 8 total)

 *  [whooami](https://wordpress.org/support/users/whooami/)
 * (@whooami)
 * [19 years, 1 month ago](https://wordpress.org/support/topic/robottxt-question/#post-562320)
 * is this your box?
 * /var/www is usually a symlink, and shouldnt for these purposes used that way.
 * In other words, your ‘path’ isnt www, it’s public_html or something equivalent.
   I mention this because your robots.txt should NOT be placed inside the symlink
   www/ it needs to go inside public_html or its equivalent
 * as far as your questions:
 * _Should I update my path to stem from my html/web folder to the location of these
   directories, e.g., if my wp files are located in “content” then change:
    /wp-
   admin/ to /content/wp-admin/
 * yes.
 * really to simplify this, there is a plugin:
 * [http://wordpress.org/extend/plugins/kb-robotstxt/](http://wordpress.org/extend/plugins/kb-robotstxt/)
 *  Thread Starter [startribe](https://wordpress.org/support/users/startribe/)
 * (@startribe)
 * [19 years, 1 month ago](https://wordpress.org/support/topic/robottxt-question/#post-562328)
 * Thanks for the reply, whooami. Ok, I checked out the plugin and I uploaded it.
   I then entered my robots.txt with out the added directory info and clicked the
   update button, but when I viewed the robots.txt file, the plugin did not add “/
   Content” to “/wp-admin/”. Should I expect the plugin to add the associated directory?
 * My WordPress Address is “example.com/content” and my Blog Address URL is “example.
   com”… Does this mean that “/content” isn’t needed for my type of install?
 * Thank you again.
 *  [whooami](https://wordpress.org/support/users/whooami/)
 * (@whooami)
 * [19 years, 1 month ago](https://wordpress.org/support/topic/robottxt-question/#post-562330)
 * thats corrent, HOWEVER, your robots.txt needs to go inside the upper most direcotry
   at your web-root.
 * In other words, if you have a blog @ example.com/content you still need that ..
 * Follow?
 * If you are using the “wordpress in its own folder” option and redirecting so 
   that example.com/ leads to your blog, you still need to have the robotx.txt available
   here:
 * example.com/robots.txt
 * And since you are redirecting the /content is unecessary.
 * Remember, though, robots.txt files go in your web_root, which isnt always your
   blog_root.
 *  Thread Starter [startribe](https://wordpress.org/support/users/startribe/)
 * (@startribe)
 * [19 years, 1 month ago](https://wordpress.org/support/topic/robottxt-question/#post-562333)
 * oh man… ok, I think I got it…
    1. robots.txt always goes in the root regardless of the install
    2. the associated paths to the directories in the robots.txt folder do not need
       the actual path of the wp-install if I am redirecting from the root, i.e., if
       my wp url is victorybelt.com and my location of the install is “/content”.
 * Thank you so much for your help. Your knowledge is appreciated. I am also interested
   in your thoughts on these two methods of robots.txt SEO:
    [Simple Robots.txt](http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/)
   [Elaborate Robots.txt](http://www.askapache.com/seo/wordpress-robotstxt-optimized-for-seo.html)
 * Which would you prefer, the simple or more elaborate?
 *  [whooami](https://wordpress.org/support/users/whooami/)
 * (@whooami)
 * [19 years, 1 month ago](https://wordpress.org/support/topic/robottxt-question/#post-562336)
 * I got askimeted, heres my reply:
 * yes, you got it exactly! Think about what you see when you go to your site, and
   you are all set 🙂
 * As far as seo and a robots.txt goes.. There are going to be those that disagree
   with me, BUT **generally speaking**, using a robots.txt will not help SEO, if
   you define SEO by an increase in traffic.
 * Think about googlebot for a second. It follows links. Nothing else nothing more.
   In the case of the second post you linked above, they suggest something similar
   to this:
 *     ```
       Disallow: /wp-content/themes/askapache/4
       Disallow: /wp-content/themes/askapache/c
       Disallow: /wp-content/themes/askapache/d
       Disallow: /wp-content/themes/askapache/f
       Disallow: /wp-content/themes/askapache/h
       Disallow: /wp-content/themes/askapache/in
       Disallow: /wp-content/themes/askapache/p
       Disallow: /wp-content/themes/askapache/s
       ```
   
 * Where on any wordpress install is there a LINK to themes/ that Googlebot has 
   standard access to ???
 * There isnt.
 * And none of what that poster writes provides anything that might aid SEO.
 * How **might** one use robots.txt to help SEO?
 * By preventing your feed from showing up in Google.
 * There’s an example of that on the first link.
 * By potentially blocking multipage views, in other words, directing searchers 
   to the permalink post instead of the pages views we so often see with categorys
   and archives.
 * I’ll tell you .. I can even show you what I do.
 * [http://www.village-idiot.org/robots.txt](http://www.village-idiot.org/robots.txt)
 * I dont have and never have had an issue with duplicate posts in Google. Case 
   in point:
 * [http://www.google.com/search?hl=en&q=wp-chunk&btnG=Search](http://www.google.com/search?hl=en&q=wp-chunk&btnG=Search)
 * Yes, my feed is there, thats by choice 🙂 Googlebot is smarter than we all think.
 *  Thread Starter [startribe](https://wordpress.org/support/users/startribe/)
 * (@startribe)
 * [19 years, 1 month ago](https://wordpress.org/support/topic/robottxt-question/#post-562337)
 * edit: your complete reply just showed up… reading it now.
 *  [whooami](https://wordpress.org/support/users/whooami/)
 * (@whooami)
 * [19 years, 1 month ago](https://wordpress.org/support/topic/robottxt-question/#post-562338)
 * no, it means my first reply was caught by askimet on here, so I reposted it.
 * I should add that when speaking about duplicate posts, one can ALWAYS do the 
   following:
 * [http://www.google.com/search?as_q=&hl=en&num=10&btnG=Google+Search&as_epq=wp+chunk&as_oq=&as_eq=&lr=&as_ft=i&as_filetype=&as_qdr=all&as_nlo=&as_nhi=&as_occt=any&as_dt=i&as_sitesearch=village-idiot.org&as_rights=&safe=images](http://www.google.com/search?as_q=&hl=en&num=10&btnG=Google+Search&as_epq=wp+chunk&as_oq=&as_eq=&lr=&as_ft=i&as_filetype=&as_qdr=all&as_nlo=&as_nhi=&as_occt=any&as_dt=i&as_sitesearch=village-idiot.org&as_rights=&safe=images)
 * which will, of course, return duplication.
 * My point in the google link in the previous post is to show that Google and its
   bot have a way of taking care of things if your site is set up properly and ‘
   standard’ search queries are used..
 *  Thread Starter [startribe](https://wordpress.org/support/users/startribe/)
 * (@startribe)
 * [19 years, 1 month ago](https://wordpress.org/support/topic/robottxt-question/#post-562339)
 * Awesome, well said. Thanks!

Viewing 8 replies - 1 through 8 (of 8 total)

The topic ‘robot.txt question’ is closed to new replies.

## Tags

 * [robot.txt](https://wordpress.org/support/topic-tag/robot-txt/)

 * In: [Developing with WordPress](https://wordpress.org/support/forum/wp-advanced/)
 * 8 replies
 * 2 participants
 * Last reply from: [startribe](https://wordpress.org/support/users/startribe/)
 * Last activity: [19 years, 1 month ago](https://wordpress.org/support/topic/robottxt-question/#post-562339)
 * Status: not resolved

## Topics

### Topics with no replies

### Non-support topics

### Resolved topics

### Unresolved topics

### All topics