Plugin Author
AITpro
(@aitpro)
1. Uh yeah and who isn’t these days 🙂 I just recently got all my sites mobile friendly and yeah probably a year or 2 behind every other website that already did that a year or many years ago. 🙂 Better late than never right.
2. Key thing to remember about a robots.txt file – ONLY good/legitimate bots follow the rules in a robots.txt file. Bad bots do not follow the rules in a robots.txt file. A robots.txt file is not a security measure and is instead for SEO purposes.
3. I think that disallowing the entire wp-content folder is probably not a good idea. If you do not want your plugins folder crawled then you would add a robots.txt rule like this one: Disallow: /wp-content/plugins/
You can even create Disallow rules for specific files: Disallow: /some-folder/some-file.xxx
4. ???
5. Remember a robots.txt file is not a security measure. It is for SEO purposes. See answer 3.
Help References:
https://support.google.com/webmasters/answer/6062608?hl=en
http://www.google.com/robots.txt?cx=010413690784631863417:cfcdruiwfxi
Thanks for the info. The webmaster tools people also gave me some robots.txt code that was unfamiliar. Of course in 7 years things do change. That was the last time I made a big effort to learn what you can do with the robots.txt file. So time to learn some new stuff about it.
I do understand there is no security in a robots.txt file. Frankly I would rather run the whole thing from htaccess if I knew how.But then the Big GBot would be unhappy.
Plugin Author
AITpro
(@aitpro)
Personally I use a WordPress virtual robots function in my theme’s functions.php file on my forum site since there are a couple of URL’s/URI’s that I do not want indexed. The clue – 10,000 indexed search results when I only had around 1,000 actual forum topic posts. Every single member profile was getting indexed and some other whacky stuff before I caught that. On my other sites I have not done anything with a robots file or function and just use the default WP robot settings.
The problem now with disallow the wp-includes is some themes use the jquery files in this folder. And of course Googlebot has to have access to EVERY file called or it can’t render the page. Hence nasty messages in webmaster tools. Emails from no-reply at Big G about your site being mobile unfriendly and Yada Yada.
So I was told on the webmaster forum to allow the specific files each theme and plugin needs so Googlebot can do its thing. It is only the jigglybot that is having a problem. The pages render fine in browsers.
I don’t want to have to manage a robots.txt file every time I make a theme change.
This mobile friendly thing is going to affect more than my little sites. I am using weaver ii on http://www.pixlerproductions.com/ I have enabled one of the mobile settings suggested and it look great on my S5 phone and on testing with small browser windows as suggested by the developer. Yet GBot is unhappy. I can of course use a different theme but Googlebot should not be able to dictate what I use on my site.
Plugin Author
AITpro
(@aitpro)
Actually the answer is very simple. Just don’t disallow anything except for things that only have to do with SEO/pages and posts being indexed that you don’t want to be indexed in search results.
Here are some logical reasons for keeping it simple and not disallowing any WordPress folders:
A robots.txt file is not going to do anything to protect anything since it is only for SEO purposes and to control what you do and do not want indexed from ONLY an SEO standpoint and NOT a security standpoint.
Worst case scenario something shows up indexed in the search results that you did not want indexed. You can correct that by creating a robots.txt rule or just ignore it if it is not a big issue.
A hacker can probe your /plugins folder or other WordPress folders, at any time, that do not explicitly have security measures implemented on them. A robots.txt file is not a security measure.
Long story short you are creating unnecessary additional work and problems for something that is not important at all.
I really do understand that robots.txt is not for security. I just don’t want the contents of my plugins, themes, includes and admin folders showing in the Google and Bing index. I trust BPS to take care of the rest of the hacking issues.
Disallow was not a problem with html or with wp until this mobile thing started. So how about this. Leave the WP Folders open. Then add x-robots rules to htacces to not index anything in the admin, includes or content folders except uploads.
This would be a lot simpler than what I am doing and still would offer some indexing protection from the regular bots.
Is this doable?
Plugin Author
AITpro
(@aitpro)
Good or bad bots cannot access or index the /wp-admin folder so no need to do anything with that folder.
Anything in wp-includes should already not be getting indexed, but it does hurt to add a wp-includes robots rule.
So I guess you could do an allow rule for /wp-content/uploads and /wp-content/themes
Plugin Author
AITpro
(@aitpro)
Typo|Correction: Anything in wp-includes should already not be getting indexed, but it does “NOT” hurt to add a wp-includes robots rule.
Plugin Author
AITpro
(@aitpro)
Assuming all questions have been answered – thread has been resolved. If you have additional questions about this specific thread topic then you can post them at any time. We still receive email notifications when threads have been resolved.
Thread Start Date: 3-15-2015 to 3-16-2015
Thread Resolved/Current Date: 3-18-2015
