Title: Site Indexing errors
Last modified: August 21, 2016

---

# Site Indexing errors

 *  Resolved [mobeustech](https://wordpress.org/support/users/mobeustech/)
 * (@mobeustech)
 * [13 years, 1 month ago](https://wordpress.org/support/topic/site-indexing-errors/)
 * I am getting the following errors when I submit a sitemap to google or bing. 
   The error that I get is “Sitemap contains urls which are blocked by robots.txt”.
   It shows that I have submitted 145 items and that 44 items are blocked by the
   robots.txt file.
 * I have added the following to my robots.txt
 *     ```
       User-agent: *
       Allow: /
       Disallow:
       Sitemap: http://www.mobeustech.com/sitemap_index.xml
       ```
   
 * I have checked the reading page in my admin panel and made sure that Discourage
   Search Engine Visibility was unchecked.
 * My .htaccess file is the wordpress default for %postname%. I really have no ideas
   on what to look at.
 * My server is centOS 6.2, php 5.3, mysql 5.2, apache 2.2

Viewing 2 replies - 1 through 2 (of 2 total)

 *  [Krishna](https://wordpress.org/support/users/1nexus/)
 * (@1nexus)
 * [13 years, 1 month ago](https://wordpress.org/support/topic/site-indexing-errors/#post-3675864)
 * The purpose of a robots text file is basically to **block** bots. The first line
   of your file allows all bots. So, just keep that unless you have some files to
   block.
    Review: [http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449](http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449)
 * Also, see in you have noindex,nofollow settings in your SEO plugins, theme settings
   or else where?
 *  Thread Starter [mobeustech](https://wordpress.org/support/users/mobeustech/)
 * (@mobeustech)
 * [13 years, 1 month ago](https://wordpress.org/support/topic/site-indexing-errors/#post-3676094)
 * I cant find any meta robots that have the noindex,nofollow. The only plugins 
   that I am currently using are Yaost’s SEO and Google Analytics.
 * Looks like it is a 24 hour refresh from google on when they refresh/update a 
   robots file. It is working now.

Viewing 2 replies - 1 through 2 (of 2 total)

The topic ‘Site Indexing errors’ is closed to new replies.

## Tags

 * [errors](https://wordpress.org/support/topic-tag/errors/)
 * [index](https://wordpress.org/support/topic-tag/index/)
 * [indexing](https://wordpress.org/support/topic-tag/indexing/)
 * [robots](https://wordpress.org/support/topic-tag/robots/)
 * [sitemap](https://wordpress.org/support/topic-tag/sitemap/)

 * In: [Fixing WordPress](https://wordpress.org/support/forum/how-to-and-troubleshooting/)
 * 2 replies
 * 2 participants
 * Last reply from: [mobeustech](https://wordpress.org/support/users/mobeustech/)
 * Last activity: [13 years, 1 month ago](https://wordpress.org/support/topic/site-indexing-errors/#post-3676094)
 * Status: resolved

## Topics

### Topics with no replies

### Non-support topics

### Resolved topics

### Unresolved topics

### All topics