palamedes

Are you sure they don’t already have a unique class?

Link the site.

Ah cool.. Shows you what I know (I’ve always used route.. heh )

Its also important to understand that most feed readers are by request only, not push.

Just because you are updating stuff doesn’t mean it will suddenly be on their computer. More over they are date sensitive, so if you add a new bit of content but set its date to June of 2000, it will already be off the page.

Usually adding old content back into a new wordpress is perfectly safe =) (Unless you’re using some third party plugin to alert people as whooami mentioned..)

I used to use it.. It works and is likely the problem you’re seeing on_edge.

Ah sorry..

http://www.robotstxt.org/robotstxt.html

The robots.txt file is a file you can put on your system that will instruct the various web search bots out there what they can and can’t search. More over you can put in a time limit on there that says “only search this often”. (google doesn’t listen to it, but you can log into their site and control it from there)

The robots.txt file on my site looks like this;

User-agent: *
Crawl-delay: 240
Disallow: /mint/
Disallow: /uploads/
Disallow: /trap/

The lines that are important for you are the Crawl-delay and the Disallow. Basically it tells robots to not crawl anything in the mint, uploads or trap directory and only to crawl my site every 240 seconds.

The crawl-delay will do a lot to help keep the bots that follow robots.txt at bay.

ANY robot that doesn’t follow that file, or falls into the trap of going to a directly specifically disallowed will show up in my logs. My log scrapper will then route them to local host:

route add -host {incoming.annoying.bots.ip} gw 127.0.0.1

What this does is it basically tells their bot to search themselves. Usually a bot will hang here .. opening up their tcp port until it times out.. (5 minutes or so).. and costs you nothing ~ its a way to slow’em down or at the very least to give them a “go away” message.

Search your logs for any IP that is pounding on your site and route it..

You might want to check your apache (or server) to make sure .php is first and .html is second in order of load.

=)

I would suggest following robots.txt as well ..

Then parse your logs, any one that isn’t following it route them to local host.