robots.txt, sitemap.xml question

  • cartpauj

    (@cartpauj)


    18 years ago

    I was wondering if anybody knows how to allow bots and spiders to see private posts or content. Is there something I can add in the sitemap.xml or robots.txt files. Also would this work for pay only content using easypaypal or arepaypal plugins?

    Thanks for the help

Viewing 4 replies - 1 through 4 (of 4 total)
  • Michael Torbert

    (@hallsofmontezuma)

    WordPress Virtuoso

    18 years ago

    Bots/spiders see exactly what a person can see. If the content is password protected by WordPress, a bot can’t see it.

    krembo99

    (@krembo99)

    18 years ago

    well.. hallsofmontezuma is right about the bots.

    But the sitemap.xml is different, you can add whatever you want there , (even manually) and the tools (like google’s) will be able to see THE LINK ITSELF, even if not what’s on it , but they will know it exist.

    Michael Torbert

    (@hallsofmontezuma)

    WordPress Virtuoso

    18 years ago

    Search engines don’t like it when you have a link to something they can’t see.

    Thread Starter cartpauj

    (@cartpauj)

    18 years ago

    hallsofmontezuma and krembo99 – Thanks for the info, so how do large, commercial websites with pay-only content still manage to have high google rankings, if the crawlers can’t see it?

Viewing 4 replies - 1 through 4 (of 4 total)

The topic ‘robots.txt, sitemap.xml question’ is closed to new replies.

  • 4 replies
  • 3 participants
  • Last reply from: cartpauj
  • Last activity: 18 years ago
  • Status: not resolved