Title: Blocking Crawling
Last modified: August 22, 2016

---

# Blocking Crawling

 *  [ritch47golf](https://wordpress.org/support/users/ritch47golf/)
 * (@ritch47golf)
 * [11 years, 7 months ago](https://wordpress.org/support/topic/blocking-crawling/)
 * I have a site with a public section and a members only section. The members only
   section is accessed by a login/logout plugin. That works fine. However, Google
   etc. still crawls the pages and posts in the members section thus defeating the
   purpose of a ‘protected’ area.
 * Is there a way to restrict search engines from crawling the content in the members
   section?

Viewing 2 replies - 1 through 2 (of 2 total)

 *  [Michael](https://wordpress.org/support/users/mikeyhoward1977/)
 * (@mikeyhoward1977)
 * [11 years, 7 months ago](https://wordpress.org/support/topic/blocking-crawling/#post-5411658)
 * You can manually update your robots.txt file and add entries to discourage search
   engines from crawling – [http://www.robotstxt.org/robotstxt.html](http://www.robotstxt.org/robotstxt.html)
 * You also need to ensure that the pages you do not want crawled are not included
   in your sitemap.xml file.
 * The SEO plugins that are available (free) can handle all of this for you. There
   are a few good ones out there. i’d suggest you take a look. They will enable 
   you to exclude pages being crawled from the Admin UI and also dictate which pages
   should not be included in the sitemap. (As well as lots of other great stuff)
 * Two good SEO plugins are;
    [Yoast WordPress SEO](https://wordpress.org/plugins/wordpress-seo/)
   and the [All in One SEO Pack](https://wordpress.org/plugins/all-in-one-seo-pack/)
 *  Thread Starter [ritch47golf](https://wordpress.org/support/users/ritch47golf/)
 * (@ritch47golf)
 * [11 years, 7 months ago](https://wordpress.org/support/topic/blocking-crawling/#post-5411671)
 * Many thanks Mike.
 * I’ll have a got at that this evening.
 * Ritch

Viewing 2 replies - 1 through 2 (of 2 total)

The topic ‘Blocking Crawling’ is closed to new replies.

## Tags

 * [crawling](https://wordpress.org/support/topic-tag/crawling/)

 * In: [Fixing WordPress](https://wordpress.org/support/forum/how-to-and-troubleshooting/)
 * 2 replies
 * 2 participants
 * Last reply from: [ritch47golf](https://wordpress.org/support/users/ritch47golf/)
 * Last activity: [11 years, 7 months ago](https://wordpress.org/support/topic/blocking-crawling/#post-5411671)
 * Status: not resolved

## Topics

### Topics with no replies

### Non-support topics

### Resolved topics

### Unresolved topics

### All topics