A Web crawler is an Internet bot that systematically browses the World Wide Web, typically for the purpose of Web indexing.

A Web crawler may also be called a Web spider,an ant, an automatic indexer.

Web search engines and some other sites use Web crawling or spidering software to update their web content or indexes of others sites' web content. Web crawlers can copy all the pages they visit for later processing by a search engine that indexes the downloaded pages so that users can search them much more quickly.Crawlers can validate hyperlinks and HTML code.

Uses:

1. Webcrawlers are used for
2. indexing pages for search engines
3. archiving the web (e.g. The Internet Archive www.archive.org)
4. analysing the web

What are the requirements for web crawlers?

1. distributed
2. scalable
3. portable
4. high performance
5. (e.g. 50 million documents / 17 days with 4 compaq server)
6. continuous
7. extensible
8. polite

Types of Crwler:

1. Yahoo! Slurp is the name of the Yahoo! Search crawler.
2. Bingbot is the name of Microsoft's Bing webcrawler.
3. FAST Crawler is a distributed crawler, used by Fast Search & Transfer, and a general description of its architecture is available.
4. Googlebot is described in some detail, but the reference is only about an early version of its architecture.The crawler was integrated with the indexing process, because text parsing was done for full-text indexing and also for URL extraction.