bbole

Absolutely. While discoverability refers to a file being found by a crawler (e.g., via links, sitemaps, or direct access), indexation is indeed a prerequisite for a page to appear in search engine results, which is a key component of discoverability by traditional search engines. The llms.txt specification is designed specifically for large language models (LLMs), not for traditional search engine indexing. Including these files in the sitemap index increases the likelihood that Googlebot will crawl and index them, treating them as regular web pages.

Since these files (especially .md versions of blog posts) often have content identical or near-identical to their HTML versions, Google may flag them as duplicate content. This could dilute our SEO rankings, confuse search algorithms, or even confuse people if they ever click on such results or lead to penalties in Google Search Console, as it may interpret the .md files as alternate versions of the same page without proper canonicalization.

By excluding llms.txt, llms-full.txt, and related .md files from the sitemap index, we reduce the risk of traditional search engines indexing them, thereby minimizing their discoverability in search results.

This aligns with the llms.txt proposal, which intends these files to be accessed directly by LLMs or AI agents (e.g., via https://example.com/llms.txt) rather than surfaced in Google’s search results. LLMs don’t rely on sitemaps for discovery.

So:
– Excluding from sitemap indexes
– Testing using robots.txt rules to allow specific AI crawlers while disallowing traditional crawlers.
– Testing canonicals

Reduce the likelihood of anything described above happening. To me, the first step is very straightforward: the plugin shouldn’t modify the sitemap index at all. Risk mitigation.

Also @ryhowa asking because https://beebole.com/blog/sitemap_index.xml is added to GSC, Bing Webmasters, etc and the new llms sitemap is there.