# $Id: robots.txt,v 1.1 1999/07/18 04:32:56 dag Exp $ # # This file is used by automatic WWW Robots, Web Crawlers, and # other automated web mapper programs. It defines which areas # are not to be explored. # See http://info.webcrawler.com/mak/projects/robots/norobots-rfc.html # and http://searchenginewatch.com/webmasters/spiderchart.html # for more information. # # Robots.txt User-agent: * Disallow: /lists/ Disallow: *.inc Allow: *.php