Restricting Sitemap

  • Resolved metalsoft

    (@metalsoft)


    3 years, 8 months ago

    Hello there,

    I’ve recently migrated from Yoast Seo to Rank Math because Rank Math offered a method to hide my Sitemap address.
    My problem was someone was using the sitemap to find my new content and copy them.
    I successfully changed my sitemap address and I removed the address from the robots.txt file.

    But checking the access logs I can still see they are accessing the sitemap and going through all my recent updates right from there.

    I don’t know how they found the address again but anyways, this didn’t work as I intended.
    So I decided to deny access to the sitemap to users via the .htaccess file.

    I used this bit of code to deny access to post-sitemap1.xml (which is the most important for me) and only allow access to search engines :

    
RewriteEngine on
# Check so we have a user-agent equal to 'googlebot' or other bots
RewriteCond %{HTTP_USER_AGENT} !^(googlebot|bingbot|yandexbot)$ [NC]
# If the user-agent is not correct we return a 403: Forbidden
RewriteRule ^post-sitemap1\.xml - [F,L]

    but unfortunately, this didn’t work and the sitemap was still being served to users.

    So I wanted to know if there is anything I can do to prevent users from accessing the sitemap and only serve it to the search engines.

Viewing 1 replies (of 1 total)
  • Plugin Support Rank Math Support

    (@rankmathteam)

    3 years, 8 months ago

    Hello @metalsoft,

    Thank you for contacting the support and bringing your concern to our attention.

    We don’t have an option to hide the sitemap in our plugin, but the only way to “hide” it from your users is to rename it. However, we also suggest not removing the sitemap URL from the robots.txt file, as this is important so the search engine can discover the sitemap path.

    Please refer to this thread on how you would effectively hide your sitemap from your users:
    https://webmasters.stackexchange.com/questions/59849/how-to-hide-my-xml-sitemap-from-competitors-but-not-from-search-engines

    Please stay advised that if someone wants to, they can still find the path to the Sitemap file using external tools. Rather than focusing on hiding it, we would recommend focusing more on working on the authority of your website, so stolen content does not get ranked higher.

    Hope this helps you with your situation.

Viewing 1 replies (of 1 total)

The topic ‘Restricting Sitemap’ is closed to new replies.