Charles Arthur: 'Microsoft thinks search engines are the key to internet use: it's building Google-style functionality into its next operating system'

The search engine market just got a lot more vicious. The tales have begun to leak out that earlier this year Microsoft simply went along to Google and offered to buy it, for upwards of $10bn. The reason: Microsoft thinks that search engines are the key to future internet use. It's building a Google-style functionality into Longhorn, its next operating system version. And it would have been nice, really nice, to have had that technology under its roof, just as it bought Hotmail when it saw how popular free e-mail was in the mid-Nineties.

Sergey Brin and the other founders of Google, however, have their eyes on a stock market flotation; and, treasuring the company's independence, said no.

Last week Google even took the battle to its new enemy, by launching a desktop toolbar for Windows users that lets them search Google without opening a browser. The Google Deskbar is free, and works with versions of Internet Explorer 5.5 and up, with Windows 98 onwards; it's available from http://labs.google.com/.

That's quite a thumbing of the nose at Microsoft, which in its battle with Netscape was able to dominate because in the four consumer operating systems it released between 1995 and 2001 (Windows 95, 98, ME and XP) its browser was always on the desktop first; people had to seek out the rival. One could say that this time Google has got its retaliation in first. It has "mindshare": its name is known to the public to a far greater extent than Netscape's ever was. "Google" - meaning to "search for the name of (someone) on the Internet to find out information about them" - can be found in the latest edition of the Oxford English Dictionary.

So now, Microsoft, the "800-pound gorilla" of computing is spending some of its $46bn cash reserves on developing new search technology. Yet the search landscape is, at present, not so simple as it might appear. It's not just Google and a sometime-in-the-future Microsoft; there are a huge number of search engines scrapping for a slice of your attention.

What's not so well known is that many search engines (though not Google) use one of the dirtiest little secrets of the web - a method called "paid inclusion". It's an arrangement between companies touting products and the search engines: the companies pay to have their results pushed up the search results, even if their product isn't germane or if other pages could reasonably be judged more relevant. If a surfer clicks through that paid-for link, the search engine is paid between 10p and 50p.

"Paid inclusion" is one of the three tiers of searching, which start with "pure search" - which is what you see with Google's standard results - and continue with "paid placement", which is the term for the adverts you see above and to the side of any Google page. What's not clear is how many companies offer "paid inclusion". AOL uses Google, which doesn't. But Yahoo! does, and claims that "it can provide users with better information".

One company that has used paid inclusion is LookSmart: once you started paying to be included in its results, your pages would shoot up its rankings, where they would be more visible to LookSmart's customers - which included MSN, with its millions of users. (An investigation by the US magazine BusinessWeek established a link.)

However, it may be that paid inclusion is on the wane. Last month, MSN ended its contract with LookSmart - a move that will actually cost it money (because of the cash that LookSmart used to pay whenever a surfer clicked through). But it's decided to do it because Microsoft has realised that paid inclusion just isn't popular - and people can see through it.

(And they can always make a comparison with another search engine's results for the same search string; it's only a browser window away.)

Indeed, it was MSN's own testing that revealed the weaknesses of the paid inclusion method against purer searching. "The testing was conclusive that the more relevant results were outside the LookSmart listings," Judy Redetzki, MSN's head of search, told the industry website Search Engine Watch (www.searchenginewatch.com).

With that business gone, LookSmart looks a goner to analysts: MSN generated 65 per cent of its listings revenues and all of its licensing income in the second quarter of this year.

So will Web searching turn clean? Adrian Cox, the chief executive of Ask Jeeves (www.ask.co.uk), seems evasive on the point. Ask Jeeves has recently undergone a complete facelift aimed at helping people to shop, and doesn't use paid inclusion, but he doesn't seem to be ruling it out. "For Ask Jeeves it is not about one or the other, but creating an excellent search experience utilising a combination of different technologies to offer the best possible results for users and customers," he says. And what marks out the winners from the losers? "Relevance and quality." But he too thinks that LookSmart "have some tough challenges".

That still leaves Yahoo!, owner of the Inktomi search technology, as the biggest user of paid inclusion. One of the problems that observers have found is that if ads aren't labelled as such, it simply looks as though the search engine isn't very good at answering your requests. Yahoo! insists that paid inclusion is what people want: "If you deliver a poor user experience, people leave, and they leave rapidly," says Tom Wilde, the general manager of search services. Perhaps - or maybe they just find it convenient to use the Yahoo! search after they check their e-mail.

We're witnessing the rise of a dramatic new industry, says Danny Sullivan, the editor of Search Engine Watch. "The search engine business is a new advertising medium," he notes in an analysis on his site. "Some people search for information, but many are looking to purchase products and services."

He thinks that Microsoft's approach to Google - and the rejection - was no surprise: "Overall, Microsoft needs Google more than Google needs Microsoft."

The reason: "Google's web site is incredibly popular, a giant billboard that it completely owns. It's not a media agency for its own site - it is the media owner... As long as Google's own site stays popular, and there's every reason to expect that it will, Google is in the enviable position of being a major media owner." Indeed, Brin told Sullivan that he would classify his own company as "a tech company, but one where that technology is linked to media". The struggle is still to come, though. MSN will be using a search engine developed by Microsoft once its "spider" has finished crawling and indexing the Web - probably some time in 2005.

In the meantime, it may rely on another partner, perhaps even Google. There's a saying in politics that it's a good idea to keep your friends close, and your enemies even closer. After all, knowing what MSN's customers want would only be helpful in the fight lining up in the future.

network@independent.co.uk