Title: Virtual robots.txt missing Last modified: August 30, 2016 --- # Virtual robots.txt missing * Resolved [HaraldC](https://wordpress.org/support/users/haraldc/) * (@haraldc) * [10 years, 5 months ago](https://wordpress.org/support/topic/virtual-robotstxt-missing/) * Hello all, * WordPress is supposed to generate a robots.txt * On latest 4.4. pristine install I cannot find it. * How is is generated ? * Is there something special I have to do ? * I’ve searched half a day, still couldn’t find anything meaningful. * Any hint appreciated. * Thanks. Viewing 14 replies - 1 through 14 (of 14 total) * [Clayton James](https://wordpress.org/support/users/claytonjames/) * (@claytonjames) * [10 years, 5 months ago](https://wordpress.org/support/topic/virtual-robotstxt-missing/#post-6899168) * > WordPress is supposed to generate a robots.txt… > ….On latest 4.4. pristine > install I cannot find it. * That’s because it’s a virtual file. Create a robots.txt file and upload it to your WordPress root directory. That file will then automatically override the virtual file. * Thread Starter [HaraldC](https://wordpress.org/support/users/haraldc/) * (@haraldc) * [10 years, 5 months ago](https://wordpress.org/support/topic/virtual-robotstxt-missing/#post-6899254) * Hello, * Thanks for your reply. * I understand that it’s not an actual file. * When I say I cannot find it I mean “[http:///robots.txt”](http:///robots.txt”); returns “Not Found” * What I don’t understand is how is it generated e.g. is it some sort of web server rewrite rule ? * I don’t mind writing my own robots.txt , no big deal, except e.g. the “Googla XML Sitemaps” plugin have an option to add sitemap URL to virtual robots, etc. * Regards. * [Clayton James](https://wordpress.org/support/users/claytonjames/) * (@claytonjames) * [10 years, 5 months ago](https://wordpress.org/support/topic/virtual-robotstxt-missing/#post-6899293) * Do you have your WordPress files installed in a sub-directory? * Thread Starter [HaraldC](https://wordpress.org/support/users/haraldc/) * (@haraldc) * [10 years, 5 months ago](https://wordpress.org/support/topic/virtual-robotstxt-missing/#post-6899307) * Linux + Apache + VirtualHost on localhost * It is installed under the same dir as defined by the DocumentRoot directive. * WordPress 4.4 plain vanilla install, out of the box. * Thanks, Cheers. * [Clayton James](https://wordpress.org/support/users/claytonjames/) * (@claytonjames) * [10 years, 5 months ago](https://wordpress.org/support/topic/virtual-robotstxt-missing/#post-6899308) * > It is installed under the same dir as defined by the DocumentRoot directive. * Installed **under** the same directory as the document root? Or **in** the document root? * When you open the site in your browser, is the address `http://localhost` or ` http://localhost/some_folder_name` ? * Here’s what I believe to be true: * When WordPress is installed in the root directory of a domain, and the “Discourage search engines from indexing this site” option is **not** checked in your Reading Settings, `http://localhost/robots.txt` should look like this – * ``` User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php ``` * When the “Discourage search engines from indexing this site” **is** checked, you should see this – * ``` User-agent: * Disallow: / ``` * If WordPress has been installed in a sub-directory inside of root – example: ` http://localhost/some_folder_name/` you’re probably going to get a 404 on `http:// localhost/some_folder_name/robots.txt` * I don’t think WordPress will create a virtual robots.txt file if it’s in a sub- directory install. * Some references to take a look at: [class-wp-rewrite.php line 1250](https://core.trac.wordpress.org/browser/trunk/src/wp-includes/class-wp-rewrite.php#L1250) * [wp-includes/functions.php line 1301](https://core.trac.wordpress.org/browser/trunk/src/wp-includes/functions.php#L1301) * That should shine some light on just how _**little**_ I actually know about the WordPress virtual robots.txt file (LOL!) but I _think_ that’s all good information. I can’t think of any other reason off the top of my head why you wouldn’t be seeing a robots.text file unless the WordPress files are in a sub-directory. * Hopefully someone who actually knows what else could be going on might chime in and point us in the right direction if I’m totally off the mark. * Moderator [Samuel Wood (Otto)](https://wordpress.org/support/users/otto42/) * (@otto42) * WordPress.org Admin * [10 years, 5 months ago](https://wordpress.org/support/topic/virtual-robotstxt-missing/#post-6899328) * Make sure you have pretty permalinks turned on, and have an .htaccess file created for the site. * [Clayton James](https://wordpress.org/support/users/claytonjames/) * (@claytonjames) * [10 years, 5 months ago](https://wordpress.org/support/topic/virtual-robotstxt-missing/#post-6899345) * > Make sure you have pretty permalinks turned on, and have an .htaccess file > created for the site. * Oh, yeah… then there’s that, too _(kicks toe in the dirt and hides face)_ Thanks Otto! 🙂 * Thread Starter [HaraldC](https://wordpress.org/support/users/haraldc/) * (@haraldc) * [10 years, 5 months ago](https://wordpress.org/support/topic/virtual-robotstxt-missing/#post-6899353) * James: It is installed _in_ the DocumentRoot but as a VirtualHost * Samuel: I see that Pretty Permalinks is always on, my only choice is to select what format I want. I tried a couple, made no difference. * The .htaccess is the default one coming with WordPress. * This is a pristine WordPress 4.4 install. * Many thanks to all that replied. * Moderator [Samuel Wood (Otto)](https://wordpress.org/support/users/otto42/) * (@otto42) * WordPress.org Admin * [10 years, 5 months ago](https://wordpress.org/support/topic/virtual-robotstxt-missing/#post-6899403) * > I see that Pretty Permalinks is always on, my only choice is to select what > format I want. I tried a couple, made no difference. * The “default” setting is for pretty permalinks to be Off, actually. If your URLs look like ?p=1234, then they are not enabled. * > The .htaccess is the default one coming with WordPress. * WordPress does not come with an .htaccess file, it creates one when you enable non-default permalinks, or it tells you what to put in the file itself if it cannot create the file. Furthermore, the .htaccess will only work if you are using Apache and have Apache configured to allow that sort of thing correctly. * These are things you need to check. If permalinks are not working properly, then the “default” robots.txt piece, which relies on them working, won’t work. * You need to perform checks and tests on the site in order to solve the problem. We don’t care how “pristine” the site is, it is a unique site, with a unique configuration. We need to know that configuration. * You say that you tried some other permalinks settings. What settings? How did you test after those settings? What was the result of your tests? What is the actual content of your .htaccess file? What permalink settings do you currently have? Do the pretty URLs work for your site in other places, like Single Post pages or Archives? * These are the kinds of things we need to know to diagnose the problem. * Thread Starter [HaraldC](https://wordpress.org/support/users/haraldc/) * (@haraldc) * [10 years, 5 months ago](https://wordpress.org/support/topic/virtual-robotstxt-missing/#post-6899413) * Samuel and all — thanks again for your help. * Permalinks: there is no such thing as ‘off’ but there is ‘plain’ so I guess this is what you refer to. * If it doesn’t work under a couple of permalinks settings, it is pointless to try them all, I’m not willing to dance naked under full moon and sacrifice a rubber chicken just to to make it work 😀 😀 😀 * I was expecting it to work under most circumstances, especially without any potentially interfering plugins and changed settings. Clearly that’s not the case and I am not into poking blindly in the dark. * It’s not in ‘WordPress for Dummies, 7th ed’ It’s not in ‘WordPress 4.4 Complete’( Packt) It’s not in ‘Professional WordPress, 3rd ed’ (Wrox) The first thing that comes up in a Codex search is a 6 years old post I’ve searched on the web for almost a full day, nothing. Everybody is using a real robots.txt. * So clearly something is going on: it’s not being used and/or it’s not working anymore, nobody’s paying attention to it … or something equivalent. * I’m giving up. * But again thank you all for your help, time, and effort, much appreciated 🙂 ( even if it did not lead to a solution) * All the best. Regards. * [Clayton James](https://wordpress.org/support/users/claytonjames/) * (@claytonjames) * [10 years, 5 months ago](https://wordpress.org/support/topic/virtual-robotstxt-missing/#post-6899415) * Oh, boy.. don’t give up yet! You’ve come so close! * I’m not so sure you’re actually suffering from a _WordPress_ condition at this point. I think “pretty permalinks” probably aren’t really working. * > Permalinks: there is no such thing as ‘off’ but there is ‘plain’ so I guess > this is what you refer to. > If it doesn’t work under a couple of permalinks settings, it is pointless to > try them all, * While you will always see the permalink choices in your dashboard settings, they only work if you’ve enabled mod_rewrite on your server, and then added the proper directive to your virtual host configuration. * So whether or not “pretty permalinks” work is completely dependent on your apache and virtual host configuration. What Linux distribution did you choose for installing WordPress? Was it Ubuntu by any chance? * It might actually be a pretty simple fix. Feel like giving it one more shot? Maybe we can confirm whether or not it’s actually an apache configuration oversight that’s causing the problem. * Thread Starter [HaraldC](https://wordpress.org/support/users/haraldc/) * (@haraldc) * [10 years, 5 months ago](https://wordpress.org/support/topic/virtual-robotstxt-missing/#post-6899416) * Bummer! OK fixed. James, thanks for the mod_rewrite reminder * mod_rewrite: was enabled in the main config but not in the VirtualHost *facepalm* 🙂 * ``` ... ... AllowOverride All <- needed for .htaccess not sure 'All' is best (security ?) RewriteEngine On <- this was it ... ... ``` * Thanks 🙂 * [Clayton James](https://wordpress.org/support/users/claytonjames/) * (@claytonjames) * [10 years, 5 months ago](https://wordpress.org/support/topic/virtual-robotstxt-missing/#post-6899417) * Glad you got it worked out! * [joshibhargav20](https://wordpress.org/support/users/joshibhargav20/) * (@joshibhargav20) * [9 years, 8 months ago](https://wordpress.org/support/topic/virtual-robotstxt-missing/#post-8233284) * I was getting this error and really it took toll of my nerves * Finally I found a solution and I suggest either of these works * 1. If you use plugin called “Google XML Sitemaps” then uncheck “Add sitemap URL to the virtual robots.txt file.” under Basic Options. I have a line for the XML Sitemap in the Robots.txt file so I am not using this option. * 2. Try installing some virtual robots.txt editor plugins. I have used WP Robots Txt and then edited the file. It worked! * Please note: I still have the robots.txt on my root. I just copied the text on WP Robots Txt plugin and it worked! Gosshh * [I will post the same answer to some of the similar questions.. so those who are in same situation may get instant relief] Viewing 14 replies - 1 through 14 (of 14 total) The topic ‘Virtual robots.txt missing’ is closed to new replies. ## Tags * [robots.txt](https://wordpress.org/support/topic-tag/robots-txt/) * [virtual](https://wordpress.org/support/topic-tag/virtual/) * In: [Localhost Installs](https://wordpress.org/support/forum/localhost-installs/) * 14 replies * 4 participants * Last reply from: [joshibhargav20](https://wordpress.org/support/users/joshibhargav20/) * Last activity: [9 years, 8 months ago](https://wordpress.org/support/topic/virtual-robotstxt-missing/#post-8233284) * Status: resolved ## Topics ### Topics with no replies ### Non-support topics ### Resolved topics ### Unresolved topics ### All topics