A robots file doesn’t block anything. For legitimate bots, they’ll respect the requests listed there, but there are many illegitimate bots who will plow through your website regardless of any robots.txt directives.
You can safely remove wp-admin from your robots.txt file (since it’s not doing you any good being there anyways).
The only way to block access at a file level like that is with .htaccess
(Note: I’m not affiliated with WordFence in any way – just a long-time user!)
Thread Starter
Gnomag
(@gnomag)
Appreciated for your comments Blue Bear. I have always automatically blocked robots from admin areas thinking it would help security wise and also so that Google does not index the files and put it all over the internet.
Thank you 🙂
Just out of habit I always place admin folders in robots.txt as disallowed, just in case I somehow set things up by mistake, even briefly, in a way that would cause Google to crawl and index. But just as Bluebear says, doing so is NOT blocking, it’s just an optional advisory for search engines that would actually obey the advisory, which is very few. Even MSN (bingbot) in my case seems to ignore robots.txt. Andt know that criminals make a sport out of faking like the are coming from Google, and one way to figure out if that’s the case is if something you think is Googlebot is ignoring your robots.txt.
As for search console, it is lame and will take days out of your life if you try to do everything it asks, or implies. For example, 3 years after I changed directory structure on one of my sites and entirely deleted thousands of files, Googlebot is still looking for those files and showing me a huge list of 404 errors in the Search Console. I’ve pretty much given up on getting any meaningful information from the Search Console.
MTN
Thread Starter
Gnomag
(@gnomag)
Thanks mountainguy2
3 years after I changed directory structure on one of my sites and entirely deleted thousands of files, Googlebot is still looking for those files and showing me a huge list of 404 errors in the Search Console
I had the same problem when changing one of my directories from html to WordPress. The problem derives from external links coming to your site that are pointed to the wrong page.
If you find where those links are coming from, (it should show you in Search Console), it is important to remember those sites for future linking benefit. At least you know any SEO juice is going to get through.
My suggestion would have been to redirect all the pages to the equivalent new pages. A lot of work but, well worth it. Turning your back on long time links can really hurt your performance as it can take years to build up a good authoritative link flow.
Keep in mind, an SEO debate is like a Theology debate. Very opinionated with a dash of faith.
Thanks once again.
