Google: The search engine that lost its way

The hegemony of Google on the internet is under attack from 'content farms' and other rivals

It looks as clean and simple as it always has: the little white letterbox into which we type our search queries and begin our light-speed journeys across the world wide web. Behind the unchanging facade, however, Google's engineers are engaged in a frantic, never-ending battle to make sure that the search results we get are the ones we expect.

The battle is not new. But over the past few months, for perhaps the first time in the 13 years since Google revolutionised the way we surf the web, it has felt like Google is not always winning.

Perhaps you have noticed. Useless web pages are appearing high up its search results, pages from never-before-seen websites that promise the information we are looking for, yet seem to contain only ads. At its worst, it is as if the spam filter has been flung open and all the monstrous emptiness of the web has been revealed. Google admitted: "We're not perfect."

What is going on? The bottom line is that Google is a victim of its own success. It dominates the web. In the UK, it is the most visited website, and nine out of 10 search queries go through Google, according to the web analyst Hitwise. To get traffic to your webpage, you have to appear high in Google's search results. Meanwhile, spending on online advertising passed spending on newspaper ads last year and is set to grow by 10 per cent this year. Since advertisers pay per page view, or per click on an ad, the spoils from clawing your way up the Google rankings are high indeed.

There are huge incentives to try to game the system – which is why there are ever more parties trying to game the system.

Google says its search quality is higher than it has ever been, in terms of relevance and comprehensiveness. At the same time, it admits it is going to have to tackle new phenomena, such as blogs that automatically plagiarise other people's content just to steal their traffic, or so-called "content farms" which produce vast quantities of low-quality webpages.

"As pure web spam has decreased over time, attention has shifted instead to 'content farms', which are sites with shallow or low-quality content," Matt Cutts, Google's principal engineer, wrote in a mea culpa blog post last week.

"In 2010, we launched two major algorithmic changes focused on low-quality sites. Nonetheless, we hear the feedback from the web loud and clear: people are asking for even stronger action on content farms and sites that consist primarily of spammy or low-quality content ... The fact is that we're not perfect and, combined with users' skyrocketing expectations of Google, these imperfections get magnified in perception. However, we can and should do better."

Google already tweaks its algorithm two or three times a day. It's as if the scientists at Coca-Cola had to come up with a new secret formula every few hours. The worry is that the internet is changing around it, and that Google itself is the problem.

The system that Google's founders, Sergey Brin and Larry Page, invented blew away the less sophisticated search engines, such as Yahoo and AltaVista, that existed before. By judging the importance of a page by counting the number of pages that linked to it (and judging the importance of those pages), they changed the way searching worked. Ever since, web publishers – from porn merchants to multinational media giants, and from individual bloggers to corporate brand managers – have spent their time working on "search engine optimisation".

This has marketing departments creating "landing pages" buried inside corporate sites, to hit all the different possible permutations of a search query relating to their industry. Bloggers linking to each other. A whole army of advisers to tell publishers where to put what keywords on a page.

And now, content farms. Intriguingly, one is on a roadshow of investors right now, ahead of a New York stock market flotation this week – a flotation that got a last- minute spanner put in the works by Mr Cutts's blog post.

Demand Media, founded by Richard Rosenblatt, the boss of MySpace who sold the social networking pioneer to News Corporation, has created a mirror-image version of the Google algorithm, which suggests a page that would do well in search results – and then builds it. The company, whose own-brand sites include eHow.com and thousands of others, has been criticised for producing low-quality content often barely better than spam. Yahoo and AOL are experimenting with similar content farms.

Demand Media is trying to raise $58m, with Goldman Sachs leading the offering. Analysts think Google's hint of a clampdown could hurt demand for Demand Media shares. "If Google tweaks its algorithm in a way that causes Demand's content to fall in the search rankings this could decrease traffic significantly and lead to material revenue losses," Rory Maher, of Hudson Square Research told clients. "Even small changes, can have an impact.

Such is the power of the Google algorithm. Our favourite search engine company doesn't just order the internet for us. It shapes it.