Sitemap & Robots.txt Issues

  • Resolved Henry

    (@henrybaum)


    7 years, 9 months ago

    Since upgrading to the new robots.txt method, the sitemap is no longer working. I get this message: XML Parsing Error: no element found.

    Additionally, the sitemap that was listed in my robots.txt file now has .gz appended to sitemap.xml – a sitemap link that doesn’t work.

    Finally, with the upgrade I get two User-agent sections and no way to edit the first section with Allow admin-ajax. Regardless if that’s necessary, how does one clean up the file?

    • This topic was modified 7 years, 9 months ago by Henry.
Viewing 11 replies - 1 through 11 (of 11 total)
  • Thread Starter Henry

    (@henrybaum)

    7 years, 9 months ago

    Deselecting – Create a compressed sitemap file in .xml.gz format – doesn’t help. I still get the parsing error.

    • This reply was modified 7 years, 9 months ago by Henry.
    Michael Torbert

    (@hallsofmontezuma)

    WordPress Virtuoso

    7 years, 9 months ago

    1) The parsing error is likely this: https://semperplugins.com/documentation/this-page-contains-the-following-errors/

    2) Once you’ve fixed #1, see if ti fixes #2 (broken compressed file link) as well.

    3) These are the default rules added by WordPress, to remove them, use the WordPress API filter hook robots_txt

    Thread Starter Henry

    (@henrybaum)

    7 years, 9 months ago

    I’ll look into it – possible conflict with Autoptimize?

    The sitemap is working fine on a dev site with the same plugins/theme, but I’ve had other problems with the main site with thousands of posts.

    For instance, when I enable sitemap indexes now (1000), the sitemap loads but it just has urls like /sitemap_addl.xml, sitemap_post_1.xml, sitemap_attachment_7.xml

    • This reply was modified 7 years, 9 months ago by Henry.
    Michael Torbert

    (@hallsofmontezuma)

    WordPress Virtuoso

    7 years, 9 months ago

    possible conflict with Autoptimize?

    You can use these troubleshooting steps to narrow down a plugin/theme conflict. https://semperplugins.com/documentation/how-to-troubleshoot-issues-with-all-in-one-seo-pack/

    when I enable sitemap indexes

    urls like /sitemap_addl.xml, sitemap_post_1.xml, sitemap_attachment_7.xml

    Those are the sitemap index URLs.

    Thread Starter Henry

    (@henrybaum)

    7 years, 9 months ago

    Ah! OK, working now: sitemap_post_1.xml has all the posts displaying. I had a problem before with indexes showing gz links that weren’t displaying properly and confused the issues.

    Thanks for the help!

    Thread Starter Henry

    (@henrybaum)

    7 years, 9 months ago

    One last question – is there any problem with having two separate sections of User-agent:* in robots.txt?

    cattivik3

    (@cattivik3)

    7 years, 9 months ago

    The new system overwrites the old robots.txt file and I lose all the commands I had inserted for months.
    The current way does not allow to edit it as a simple text.
    This implies a limitation in its use over very long times when you need to insert disallow.
    Other times it inhibits the robots.txt file hand-crafted via ftp generating a not found.
    It also generates a compressed sitemap that I do not want.
    Re-enable the fuction to edit by hand, currently I have disistallato on all the sites of my customers the current relase because it creates problems this new management of the robots.txt file and so I am forced to do so via ftp.
    Thanks for the help!
    E.

    Michael Torbert

    (@hallsofmontezuma)

    WordPress Virtuoso

    7 years, 9 months ago

    The new system overwrites the old robots.txt file

    Only if you click delete.

    The current way does not allow to edit it as a simple text

    That’s correct.

    This implies a limitation in its use over very long times when you need to insert disallow

    You should be able to still do that.

    It also generates a compressed sitemap

    This is a setting.

    Any feature requests, bug reports, etc can be submitted as a Github issue. One item per issue. https://github.com/semperfiwebdesign/all-in-one-seo-pack/issues/new/choose Please note, you must be logged into Github with a free account to view the link.

    cattivik3

    (@cattivik3)

    7 years, 9 months ago

    If I have to insert 50/100 disallow and I have not got ftp to the site I have to be able to do it quickly from within WP being able to edit the robots file, as could be done before, it is unthinkable to do it with the current system.
    How can it be done?
    Thank you
    E.

    cattivik3

    (@cattivik3)

    7 years, 9 months ago

    The robots.txt file from the server has since deleted all the sitemap.xml I had created.
    I had 4 sitemap.xml, lost everything.
    It should remain the original file, so you lose the job and set various settings.
    How can you handle this if you have several sitemap.xml?
    Everything was simpler and faster before now not.
    Thank you
    E.

    cattivik3

    (@cattivik3)

    7 years, 9 months ago

    Last thing, the robots file is vortuale and no more physical in the server and then you lose all the work if you skip the plug in, absurd.

Viewing 11 replies - 1 through 11 (of 11 total)

The topic ‘Sitemap & Robots.txt Issues’ is closed to new replies.