Spider

A spider, also known as a crawler, bot or ant, is a program that methodically browses the World Wide Web in order to create a copy of all visited pages for processing by a search engine. This data is later used by the search engine to generate faster listings. The crawler generally starts with a list of URLs to visit, then identifies links in these pages and adds them to the list, essentially "crawling" down through the site and throughout the world wide web.