Some issues in Sitemap

  • Resolved ariyannew005

    (@ariyannew005)


    1 year, 4 months ago

    Hello

    My pages were not crawling and I am really tensed after spending 3 days I got to know about this issue.

    I found an article in google search that we need to append “all” in “X-Robots-Tag” in sitemap.xml from 11 Jan 2024.

    As you can see I was gettint this result
    https://prnt.sc/QRsPUpQzBirx
    https://prnt.sc/7kGEDO_YB7jx

    Then I added this header from htaccess and now the pages are indexing and crawling properly. Updating things instantly.

    So is it actually happening or not, if yes then I think you should work on it? Some also said that x-robots-tag doesn’t matter in Sitemaps but I am still in confusion, should i add this header or not?

Viewing 4 replies - 1 through 4 (of 4 total)
  • Mushrit Shabnam

    (@611shabnam)

    1 year, 4 months ago

    Hi @ariyannew005

    XML sitemaps do not need to be indexed by search engines in order to be read or parsed. Therefore, we add a x-robots-tag HTTP header (with a value of noindex, follow) to all XML sitemap responses.

    If you have a sitemap that is located on example.com/sitemap.xml, your sitemap is not being generated by our Yoast SEO plugin. It is probably generated by another plugin or WordPress core itself. Please disable other sitemap plugins and remove any physical sitemap files via FTP before enabling the sitemaps in Yoast SEO plugin.

    Thread Starter ariyannew005

    (@ariyannew005)

    1 year, 4 months ago

    Hello

    Yes, you are righ that sitemaps are not required to indexed. But my pages were not crawling, I have around 200 blog posts only 50 indexed in console. And after adding “all” value now I can see 166 Posts indexed in 2 or 3 days.

    Even these are not like images or malicious links, these are the post links I have verified multiple times.

    and The sitemap is sitemap_index.xml. You can see in the Image also, I have not covered them.

    I don’t know what’s the issue or is there any new rule but yes after adding “all” it is now crawling and indexing, but the actual question is, is it Okay to use or is there any other issues in my site ( Probably not )

    Mushrit Shabnam

    (@611shabnam)

    1 year, 4 months ago

    Hi @ariyannew005

    It can take some time before all the pages are indexed in Google Search Engine Result Page. You can check the following to confirm there is no technical error or blockage for your site:

    1. Kindly visit the robots.txt file of your site and check if you have the correct directives. I can see you have multiple directives added. You can follow the help article to check and confirm that you have the correct directives in the robots.txt file.
    2. I have checked and the sitemap loads correctly.
    3. No error for Rich Results Test and for schema validator.

      Finally, please know that crawling can take anywhere from a few days to a few weeks. Be patient and monitor progress using either the Index Status report or the URL Inspection tool.
    Thread Starter ariyannew005

    (@ariyannew005)

    1 year, 4 months ago

    Hello

    I have checked the robots.txt it is good and also valid as per google and bing both.

    My issue is there in many pages
    https://prnt.sc/Vm8S65kMcbYG
    https://prnt.sc/WNc7EREB-SW6
    https://prnt.sc/ZRaSSaW_N8wa
    And like 70% of blog articles are still only crawled on december and 30% is not indexed yet written in Nov 2024

    It was last crawled on Dec 2024 then I have requested many times, even changed so many things in the page still not crawling and indexing the latest updates.

    • This reply was modified 1 year, 4 months ago by ariyannew005.
Viewing 4 replies - 1 through 4 (of 4 total)

The topic ‘Some issues in Sitemap’ is closed to new replies.