The Crawling Power of the Googlebots

In a recent interview Google's Dan Crow offered some insight into Google and how their search engine Crawls and indexes your webpages.

While Google's indexing is hyper efficient for HTML, especially where web standards are maintained properly the search engine has hit some stumbling blocks when it comes to rich media and some new web technologies. AJAX was mentioned as a current difficulty for indexing, The format that allows asynchronous page updates can also create a block to indexing and unles the code is stored off page it can create a lot of 'noise' for the indexer to read through and which waters down your keyword density.

Flash was mentioned as anoher possible stumbling block, though when asked if Google and Adobe were working together to provide a means of indexing rich media pages the answer Dan Crow gave was "I can't talk about that" which, lets be honest, makes it a pretty fair bet that they are talking about it. However it was revealed that the indexers have the ability to extract and index text from within a flash file and that work is progressing on a better understanding of how to index flash files, if only because Google owns YouTube, which is pretty much totally flash driven.

It does appear that Google's crawling ability does have its limitations in finding completely unlinked sites. This is possibly the most interesting point for SEO as it truly makes it clear that without links to and from your site it is incredibly difficult for even Google to find and index your site. Thus in any SEO strategy it is really important to construct pages which link well and to generate and create links to your site to make sure that Google has a clear path to follow when crawling your site.

SEO Junkies live, breathe and dream Search Engine Optimisation. For more advice please give us a call!