Just a little something I ran into myself once which may be of service. I had initially set my WP install to private during the build and toggled it off when I was ready to launch and submit to Search Engines but, like many of you, had warnings/errors indicating a robots.txt file was blocking access. Of course, there was no robots.txt file in my root folder – and no, WP was dynamically “generating” one. It turned out the culprit was a security plugin I was using (Bulletproof – although I imagine WP Security Scan and others might produce similar issues) which, when set in full security mode was adding some stringent deny code to my .htaccess file. I simply toggled off root .htaccess security (kept admin areas locked down) and, within minutes, bots were happily chomping away!
Thought I’d offer this up as one more thing you could check! Cheers!
