Your robots txt file looks OK
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
You may check if you have settings elsewhere to override the above.
thanks Krishna, any other ideas then because something is definately stopping it being crawled and indexed, I’ve not had this problem with any of my other sites so I am really lost as to why the bots are saying “unreachable page” and why it wont be indexed, can the cache plugins cause any sort of problem? or the actual theme itself?
Can you check dashboard > settings > reading and see if you have checked/unchecked the box for Search Engine Visibility?
Also see if you have any SEO plugins and how you have configured it?
okay I have checked deshboard>settings>reading and I have “Discourage search engines from indexing this site” unchecked.
Here is all the plguins I have installed:
Add Meta Tags (I have this on my other sites though and it hasn’t affected them)
Hide Title (have this installed on one other site which was indexed no problem)
WP customer reviews (first time using)
Yeah I just don’t know whats causing it I’ve never had trouble getting indexed before.
Do you have a setting for robots in Add Meta Tags?
There is a setting for SEO in the above plugin. Look for a tag like noindex,follow in that. If you have checked it, please uncheck it.
oh no! I think it is this plugin, which is strange because I have it installed on four of my websites and the others don’t have any trouble, but yes in the settings it says:
“Add NOINDEX,FOLLOW to the robots meta tag on following types of archives. This is an advanced setting that aims at reducing the amount of duplicate content that gets indexed by search engines:”
and under “search results” it is ticked causing a no follow, I unchecked it and clicked save but it just checks itself again, it isn’t saving my options
I just realised the other sites I have this plugin installed on I installed them AFTER the sites were indexed, this is the only site I have put this plugin on before being indexed, I think I might have to pull the plug on this plugin,
thank you for all your help, I think you’ve found the problem, thank you 🙂
