How to include robots.txt

  • Resolved gorahul

    (@gorahul)


    2 years, 10 months ago

    I want robots.txt to be available on Publishing. But I am getting an Error as below

    Crawl 'http://localhost/agi/robots.txt' rejected (Client error: GET http://localhost/agi/robots.txt resulted in a 404 Not Found response: <!DOCTYPE html> <html lang="en-US" prefix="og: https://ogp.me/ns#"> <head> <meta charset="UTF-8"> <meta name="viewpo (truncated...) ) (found on )

    Kindly let me know how to Include files

    Thanks

Viewing 3 replies - 1 through 3 (of 3 total)
  • Plugin Author Team Staatic

    (@staatic)

    2 years, 10 months ago

    Hi @gorahul, first of all thanks for trying Staatic to improve the performance and security of your WordPress site.

    Normally, in order to include robots.txt to the build, you should go to Staatic > Settings > Build and add /robots.txt to Additional URLs.

    Since the error indicates that robots.txt is being crawled, but with a prefix of /agi, resulting in a 404 page not found error, are you running WordPress from the /agi sub directory? What happens if, in the Additional URLs setting, you change /robots.txt to the full URL, e.g. http://localhost/robots.txt?

    Looking forward to your response.

    Plugin Author Team Staatic

    (@staatic)

    2 years, 10 months ago

    @gorahul, were you able to get things working? If not, can you please give us an update. We’re here to help!

    Plugin Author Team Staatic

    (@staatic)

    2 years, 9 months ago

    Hello @gorahul ,

    We haven’t heard from you for over a week now, so it looks like you don’t have any more questions for us.

    Feel free to re-open this ticket if needed.

Viewing 3 replies - 1 through 3 (of 3 total)

The topic ‘How to include robots.txt’ is closed to new replies.