A spider, also called a robot or a crawler, is generally a program that crawls all links throughout the Internet, collect fresh content from websites and helps to adding it to search engine indexes.

Spiders follow all the links from one page to other and from one website to another website. Spiders or crawler are the software that the various search engines sends out to index websites. Spider searches these Web pages by following links from other Websites.

One can also submit Web pages directly to a search engines and visit request by spiders. It is a great method to manually submit the website to human-edited directories and so that spiders will easily find it and add it to their database. They send back information to the search engine databases to keep them up-to-date and more relevant.