css,js access control

WordPress recommends that you use the WordPress virtual robots.txt function and so do I. http://forum.ait-pro.com/forums/topic/wordpress-robots-txt-wordpress-virtual-robots-txt/ So I do not recommend handling anything related to robots via htaccess code.

The Files and FilesMatch htaccess directives only allow using filenames or file extensions. You cannot use a URI/folder path.

http://httpd.apache.org/docs/current/mod/core.html#files

The <FilesMatch> directive limits the scope of the enclosed directives by filename, just as the <Files> directive does. However, it accepts a regular expression. For example:

Or of course you can use a robots.txt file to do whatever you want to do since that is what a robots.txt file is for. It is kind of silly to do this with .htaccess code when there is already a Internet standard for this: a robots.txt file.

So how do you keep Google from indexing stuff without the no index meta tag.

If you do not want the googlebot to crawl and index something you would add that in your robots.txt file or in a WordPress Virtual Robots function.

Example: I do not want the googlebot to crawl or index these folders and pages: /members/, /groups/, /activity/p/ and the login page /wp-login.php

// WordPress Virtual robots.txt additions
add_filter( 'robots_txt', 'v_robots', 10, 2 );

function v_robots( $output, $public ) {
$output .= "Disallow: /members/" . "\n";
$output .= "Disallow: /groups/" . "\n";
$output .= "Disallow: /wp-login.php" . "\n";
$output .= "Disallow: /activity/p/" . "\n";
return $output;
}

has Google finally learned not to index the items in the wp-content and wp-includes folder of a wordpress site.

The googlebot has never crawled or indexed any files in these folders on any of my websites so if that is happening on your site then something is fubar on your site.

\n means newline and not no index. Disallow means do not crawl or index.

I have never done anything with my /wp-content/ folder or /plugins/ folder that has anything to do with Robots ever – meaning the last 10 years. I have never had a single problem with any of the files in the wp-content folder or plugins folder being indexed on around 10 different websites and in the last 10 years. So the problem you are describing has never happened to me before and I have never heard anyone else ever mention this except for you. So logically the only conclusion I can come to if you are experiencing a problem with this is – it is an isolated unusual issue that is only occurring on your sites because of some problem on the sites that is causing that issue.

If you do not use Disallow then the default is Allow. In other words, you do not need to add/use Allow.

The answer that I use on all my sites is that I do not do anything for/with or to either the wp-content or plugins folder that has anything to do with Robots. In other words, I do not have any other suggestions because you should not have to do anything at all with either of these folders that has anything to do with Robots.

I literally do not do anything with any of the WordPress Core folders that is related to robots.txt in any way. BPS adds security protection for all WordPress Core folders and all your website files. BPS does security protection and does not do SEO stuff with a robots.txt file.

Plus it boosts my SEO up quite nicely since it acts like a sitemap and without showing all that crap in my actual site menus. 😉