Indexing websites on the Internet is actually a two step process. The first involves having your website and its pages crawled and downloaded by a web crawler. This downloaded content is then handed over to be indexed over which the search engine's algorithm can be applied.

Like every other search engine, Google uses a web crawler, a computer program, which browses through websites in a systematic automated fashion. Web crawlers provide search engines with the latest information regarding changes to web pages, as well as new web pages and entire websites that appear across the Internet. The crawlers make copies of every visited page and pass these on to the search engine that places them in an index for fast, efficient retrieval. The problems with the web that crawlers have to deal with are its vast size, constantly changing content and the dynamic generation of web pages.

Google generally refers to its web crawler as Googlebot but this term is applied to two different bots it employs to crawl the web. Google uses two bots to reduce the impact of the problems caused by the Internet. DeepBot is a deep crawler that follows every link and tries to download every page on the Net for the search engine. FreshBot on the other hand keeps Google as fresh as possible by adding changes to the Net to the search index.

The second step behind the search process should be the more tedious one. After all, a simple search query must link to millions of pages and sifting through this ton of information should take much more than the fraction of a second that it does. Google achieves this with the comprehensive index that it has built which points in reverse. To put it simply, the index looks up the words from the search query which then points to all the pages containing that set of words. The list of pages generated is ranked based on Google's proprietary Page Rank algorithm that search engine optimisation is constantly trying to best.

Every SEO company fights to get ranked higher at each step of the search process. Enhancing their pages for the web crawlers, they make sure their pages are crawled regularly and comprehensively. The keywords selected place them on the index for different search queries relevant to their website. Finally, using both relevance and reputation building tools and tactics, SEO experts make sure Page Rank places them at the top of the search engine rankings.