Web crawling is the automated fetching of data from a web page via an Internet bot. Web crawlers can take information from HTML, XML and other sources and adjust it for display. The process of a web crawler collecting data via a website is known as web indexing or web harvesting; the data collected generally comes in the form of an XML file or equivalent.