Hello elmalak, I don’t believe the plugin is blocking Googlebot. Do you have other security plugins installed?
Kind regards
No, this is the only security plugin I am using.
Are there any other type of plugins that can cause this?
What other plugins do you have installed?
What is your URL?
I have lots of plugins( around 25) but things have always worked nicely and I am not sure what caused this lately.
URL: http://www.elmalakrx.com/blog
Have you upgraded any of the plugins lately before this issue?
Are you checking Googlebot under webmaster tools?
Where is your robots.txt file located?
As for updates, I can’t be so sure if a particular upgrade has caused this.
I am checking with Webmaster tools, yes, and it keeps giving me the same error.
As for the file, I don’t a manually created one, but I used a plugin WP robotos.txt for creating it.
However, when I try to visit it’s URL it’s not there and I get redirected to the 404 not found page!
Does that make any sense to you?
Your robot text file is on the following path.
http://www.elmalakrx.com/Arabic/robots.txt
What URL did you enter in Google Webmaster Tools to manage?
I am certain that is the wrong path. But I don’t know how you have set up your Google Webmaster Tools.
This actually a different site and separate WP installation found in that directory http://www.elmalakrx.com/Arabic/
And I have them set up separately in Webmaster tools.
Ok so how many wordpress installations do you have running under the following domain http://www.elmalakrx.com?
Each domain should have it’s own robots.txt file.
Also in Google Webmaster Tools you should have added each different wordpress installation with their own robots.txt if you want to use one and if you want to manage them through Google Webmaster Tools which I am sure you do.
i.e.
http://www.elmalakrx.com/site1/robots.txt
http://www.elmalakrx.com/site2/robots.txt
http://www.elmalakrx.com/sete3/robots.txt
Do you understand my example above.
Yes, that’s exactly how I have them setup, but I just didn’t create the robotos.txt file manually, rather dynamically generated using plugins as I told you earlier.
Yes you did tell me, however your plugin is not generating the robots.txt and that is why you are receiving that error.
FTP or use cpanel to log into your server and check to make sure the robots.txt file exists. If they don’t then you have to create them yourself manually or using other means.
Kind regards
Thank you for all your help.
I will try that and see how it goes.
You can also get that error from google if your server was down when googlebot tried to access your site. As long as your site is up and running and you can access it using a browser, you don’t have to worry about it. Googlebot will come back to your site and see that everything is fine.
Hi mra13,
Thank you for your comment.
I am aware of this but it’s been repeating over the last couple of days that’s why I am concerned about it.
What I find strange is that I can’t access the robots.txt file in the browser, not sure if the plugin is preventing that or what?
It keeps giving me a 404 not found error when I Try to view it in the browser.
Regards,
The plugin doesn’t block access to the robots.txt file. Do you have a file called “robots.txt” in your public_html folder?
