kmccall

robots.txt only works for “good” bots. It serves as a road map for bad bots – telling exactly where they need to go (where you Don’t want them to go). The only way I have found to deal with these bots is through bot stopping scripts and htaccess.