The Madness of King Google

When Google arrived on the scene in the late 1990s, they came in with a new idea of how to rank pages. Until then, search engines had ranked each page according to what was in the page – it’s content – but it was easy for people to manipulate a page’s content and move it up the rankings. Google’s new idea was to rank pages largely by what was in the links that pointed to them – the clickable link text – which made it a little more difficult for page owners to manipulate the page’s rankings.

Changing the focus from what is in a page to what other websites and pages say about a page (the link text), produced much more relevant search results than the other engines were able to produce at the time.

The idea worked very well, but it could only work well as long as it was never actually used in the real world. As soon as people realised that Google were largely basing their rankings on link text, webmasters and search engine optimizers started to find ways of manipulating the links and link text, and therefore the rankings. From that point on, Google’s results deteriorated, and their fight against link manipulations has continued. We’ve had link exchange schemes for a long time now, and they are all about improving the rankings in Google – and in the other engines that copied Google’s idea.

In the first few months of this year (2006), Google rolled out a new infrastructure for their servers. The infrastructure update was called “Big Daddy”. As the update was completed, people started to notice that Google was dropping their sites’ pages from the index – their pages were being dumped. Many sites that had been fully indexed for a long time were having their pages removed from Google’s index, which caused traffic to deteriorate, and business to be lost. It caused a great deal of frustration, because Google kept quiet about what was happening. Speculation about what was causing it was rife, but nobody outside Google knew exactly why the pages were being dropped.

Then on the 16th May 2006, Matt Cutts, a senior Google software engineer, finally explained something about what was going on. He said that the dropping of pages is caused by the improved crawling and indexing functions in the new Big Daddy infrastructure, and he gave some examples of sites that had had their pages dropped.