I think the virtual robots.txt file may come into play in the absence of a robots.txt in your root directory. It would seem that this is the overall effect of selecting the Privacy Settings in the dashboard.
There is some interesting reading on Robots.txt Optimization for WordPress located here, though.
Thanks Clayton. I guess I may be a little confused about the ‘virtual’ piece of this…
In using Platinum SEO and XML Sitemap plug-ins, together they dynamically create my robots file entries (at least I believe that’s the way they work). And, XML Sitemap creates a virtual robots file.
Granted, I could turn off the virtual part and create one, but then I would have to manually edit it going forward with any changes made to new posts/pages, correct?
-Scott
I’m not really sure how the virtual file works with the plugin. My thought was that anything in an actual robots.txt file would take precedence over a virtual file. Now I’m not all that sure. I’ve been no help at all! Sorry about that..
Actually, I’m curious about the opposite.
My images are somehow not indexed by the Google crawler. The xml sitemap has been created longtime ago and I have given enough time to get indexed, but no images are shown.
I tried adding a wp-content/uploads directory. Though it’s not clear to me if the sub-directories are also indexed. So far no success on this.
Anyway to add the images or change settings for the image directory?
