Also known as a spider, robot or bot. A software program that visits websites and indexes the html pages and their meta tags, collecting word frequency data for a search engine. The crawler literally crawls the first page and then recursively checks every page linked until all pages have been analysed. Some search engines eg Fast Search crawl through every page (or so they claim) on the web, most others like Google or Altavista crawl pages linked from ones in their database or in response to URL submission by people trying to promote the pages. Crawlers have to follow rules of politeness and these are mostly laid down in the Robots Exclusion Standard. Crawlers will look for a file called robots.txt which tells them about the site and which pages not to index.