Robots.txt Ghost File

  • karanvij

    (@karanvij)


    1 year, 3 months ago

    I have migrated my website hosting from Godaddy Cpanel hosting (WordPress Website) to Hostinger’s WordPress hosting 2 days back.

    The transition was not easy. I had to disable WP Rocket and start using LiteSpeed Cache. There were a few errors in the beginning but I figured my way out, somehow. The problem began 2 hours back when I tried to check PageSpeed Insights. It showed no error in Desktop report but in Mobile reports it shows that Robots.txt file has errors. It showed complete HTML document of my website instead of the robots.txt file. Then I tried to reach my robots.txt file by going to mydomain/robots.txt but it kept redirecting me to my homepage.

    I checked file manager and the robots.txt file was present. I deleted it and tried to create with Yoast Seo. As soon as I clicked “create robots.txt file”, it takes me to my source code. I hot back and it created a file. I saved.

    But still the file was inaccessible.

    Finally, I deleted the robots.txt file. But now, after going to mydomain/robots.txt it’s showing my file but with xml code in the beginning.

    It’s some sort of Ghost file which Google Search Console can read and produce error to.

    I have tried disabling LiteSpeed Cache, Quic.cloud CDN, Yoast SEO. I’ve tried clearing cache from CDN dashboard as well as from LiteSpeed Cache plugin.

    Still, it’s showing a ghost file with wrong code. I’ve searched file manager for other robots.txt files but couldn’t find any.

    Mydomain/robots.txt reads:

    <?xml encoding="UTF-8"><?xml 
encoding="UTF-8"><?xml 
encoding="UTF-8"><p># START 
YOAST BLOCK
# ---------------------------
User-agent: *
Disallow:

Sitemap: https://mydomain/sitemap_index.xml
# ---------------------------
# END YOAST BLOCK</p>

    I’m not a developer but I can implement code changes if suggested. Plz help.

    • This topic was modified 1 year, 3 months ago by Steven Stern (sterndata). Reason: removed useless paste from stack overflow
Viewing 1 replies (of 1 total)
  • Moderator bcworkz

    (@bcworkz)

    1 year, 3 months ago

    As long as you don’t have an actual robots.txt file on your server, WP will generate one upon request. The one generated by plain vanilla WP is very basic and is 100% valid syntax. The faulty syntax is likely due to your theme or one of your plugins. Yoast has influenced the content, which it normally does correctly. Something else is also getting involved.

    I know you’ve tried deactivating things and flushing caches, etc. Please try again, this time going whole hog. Deactivate ALL plugins. And switch to one of the default Twenty* themes. Deactivate or flush any caching. robots.txt ought to now be correct. If not, you WP core files have become corrupted. You should perform a manual update. Do not use the re-install button on the updates admin page. Manually replace the files yourself.

    robots.txt ought to be correct with plain vanilla WP. Determine what’s corrupting the content by restoring your theme and plugins, one at a time, checking robots.txt after each. When the content is again incorrect, the last activated module is the cause.

Viewing 1 replies (of 1 total)

The topic ‘Robots.txt Ghost File’ is closed to new replies.