#robot.txt for http://www.msc.cornell.edu/ #disallows all robots into /msccf/ and beyond #written 9/16/96 mrg #information found at #http://info.webcrawler.com/mak/projects/robots/norobots.html User-agent: htdig Disallow: User-agent: Inktomi Search Disallow: User-agent: gsa-crawler Disallow: User-agent: Googlebot Disallow: User-agent: * Disallow: /