ApacheToday.com has a new article on how to configure your Apache server to get web crawlers and robots to get the most out of your site. It explains a bit about the "robots.txt" file and what spiders really do. So, if this interests you (it does me), pop on over to this link and take a look.