Title: Default wordpress robots.txt?
Last modified: August 21, 2016

---

# Default wordpress robots.txt?

 *  Resolved [Peter_L](https://wordpress.org/support/users/peter_l/)
 * (@peter_l)
 * [12 years, 9 months ago](https://wordpress.org/support/topic/default-wordpress-robotstxt/)
 * I have a site running with a custom theme. It has a robots.txt file (in root 
   of theme folder) saying:
 *     ```
       User-Agent: *
       Disallow: /wp-content/plugins/
       ```
   
 * However, on calling the robots.txt from the url (www.mysite.com/robots.txt) it
   shows a robots.txt file containing following:
 *     ```
       User-agent: *
       Disallow: /wp-admin/
       Disallow: /wp-includes/
       ```
   
 * So, it’s clearly not the one I added in the theme. Does anybody have an idea 
   where it’s coming from (virtual or actual file).
    There’s no actual other robots.
   txt in any other folder, or at least I didn’t find one. Plugins: Better WP Security
   and WordPress SEO.

Viewing 5 replies - 1 through 5 (of 5 total)

 *  Thread Starter [Peter_L](https://wordpress.org/support/users/peter_l/)
 * (@peter_l)
 * [12 years, 9 months ago](https://wordpress.org/support/topic/default-wordpress-robotstxt/#post-4034246)
 * Well, things have gotten worse. Google webmaster tools downloaded a version of
   robots.txt:
 *     ```
       User-agent: *
       Disallow: /
       ```
   
 * I’m not too concerned about this though. I reckon it’s just a previous version
   showing up and google is a bit slow to use the new version.
 * What does concern me however is the above problem. Anybody ideas? I’m clueless.
 *  Thread Starter [Peter_L](https://wordpress.org/support/users/peter_l/)
 * (@peter_l)
 * [12 years, 9 months ago](https://wordpress.org/support/topic/default-wordpress-robotstxt/#post-4034247)
 * Ah, nevermind. Proper place for robots.txt is root of site, not theme folder.
 * Closing the topic.
 * As a final sidenote, WP does create it’s own default robots.txt. For more info
   see this [http://wordpress.org/support/topic/robotstxt-set-to-disallow-cant-change?replies=35](http://wordpress.org/support/topic/robotstxt-set-to-disallow-cant-change?replies=35)
 *  [Krishna](https://wordpress.org/support/users/1nexus/)
 * (@1nexus)
 * [12 years, 9 months ago](https://wordpress.org/support/topic/default-wordpress-robotstxt/#post-4034249)
 * In your robots.txt file,
    If you do NOT want to block anything use only: `User-
   Agent: *`
 * If you want to block access to admin and includes files, use:
 *     ```
       User-agent: *
       Disallow: /wp-admin/
       Disallow: /wp-includes/
       ```
   
 * As some plugin authors may be including URLs to other sites, some site owners
   exclude plugins also in which case, you may also add:
    `Disallow: /wp-content/
   plugins/`
 * More info on can be had here:
    [http://codex.wordpress.org/Search_Engine_Optimization_for_WordPress](http://codex.wordpress.org/Search_Engine_Optimization_for_WordPress)
   [https://support.google.com/webmasters/answer/156449?hl=en](https://support.google.com/webmasters/answer/156449?hl=en)
   [http://stackoverflow.com/questions/17029811/how-to-set-up-robots-txt-file-for-wordpress](http://stackoverflow.com/questions/17029811/how-to-set-up-robots-txt-file-for-wordpress)
 *  [sidhusharma](https://wordpress.org/support/users/sidhusharma/)
 * (@sidhusharma)
 * [12 years, 2 months ago](https://wordpress.org/support/topic/default-wordpress-robotstxt/#post-4034478)
 * I experienced a problem with my wordpress website. It took 20 days for me to 
   build a website that suits my profession and looks classy but when I search it
   in google, it appears nowhere in SERP.
    I think my robots.txt file is not appropriate,
   it is showing ‘noindex, follow’ When I tried changing the coding for my site,
   wordpress is not allowing me to change the coding .What shall I do and how can
   I make my website visible in the SERP?
 *  [iwd1](https://wordpress.org/support/users/iwd1/)
 * (@iwd1)
 * [11 years, 9 months ago](https://wordpress.org/support/topic/default-wordpress-robotstxt/#post-4034499)
 * In the WordPress admin section check Settings > Reading. At the bottom of the
   page there is Search Engine Visibility. Make sure the box is unchecked so search
   engines can index your site. A lot of times this is checked during development
   and not unchecked when the site goes live thus causing the entire site to not
   get indexed by search engines.

Viewing 5 replies - 1 through 5 (of 5 total)

The topic ‘Default wordpress robots.txt?’ is closed to new replies.

## Tags

 * [robots.txt](https://wordpress.org/support/topic-tag/robots-txt/)

 * In: [Fixing WordPress](https://wordpress.org/support/forum/how-to-and-troubleshooting/)
 * 5 replies
 * 4 participants
 * Last reply from: [iwd1](https://wordpress.org/support/users/iwd1/)
 * Last activity: [11 years, 9 months ago](https://wordpress.org/support/topic/default-wordpress-robotstxt/#post-4034499)
 * Status: resolved

## Topics

### Topics with no replies

### Non-support topics

### Resolved topics

### Unresolved topics

### All topics