Why Google News signals the death of the online exclusive

The biggest player is undermining those who break stories, says Charles Arthur

Quick: which newspaper and which journalists got the exclusives on Watergate? That's easy, of course: The Washington Post, and Woodward and Bernstein. Now a harder one: who had the first stories about the abuses inside Abu Ghraib prison? Thirty years on, in a world where media are constantly updated, it might be hard to recall, even though it was only a few months ago. (It was CBS TV, on 29 April; the next day The New Yorker had a long, detailed piece by Seymour Hersh, accompanied by photographs).

But for the growing number of purely online publications, the problem of trying to remember who was first has become completely irrelevant. Instead, the most popular news site on the Net - Google News - has created a system which is killing the value of the "exclusive" online.

The reason why is subtle, but simple. Google News works by automatically "aggregating" news from thousands of news and other sources around the Web every minute of the day, and then collecting those into "groups" around a topic headline such as "Sopranos, Angels hit high notes at Emmys", which detailed the recent Emmy Awards. Around that main link (to The Australian's website) were links to seven other online versions from different publications, and another link to what may be hundreds of similar reports.

And why was The Australian chosen for the top spot in the topic headline? Because it was the most recently updated of the websites carrying the story. But in the case of what starts out as an exclusive story, such as Abu Ghraib, the most recently updated site won't be the one which got the exclusive. It'll be the me-too site, probably reporting what's on the newswires, and adding it (perhaps automatically via a computer) to its site. The organisation that got the exclusive languishes, perversely, at the bottom of Google's huge list, in a place where nobody is ever likely to "click through" to it to see the adverts there.

That means no one sees the publication or the journalist(s) who got the story. And in the world of online publications, "clickthrough" is all. Web servers allow editors and publishers to examine in minute detail what sort of stories work and which don't, because they can see exactly how many people read it, and whether they clicked on adverts on the page, and whether they looked at other stories on the site, and whether they continued to the next page in a long story. They also know if people have come there from a news site, or by searching from a search engine such as Google. The "referrer" logs, as they're known, are voluminous - and analysed as minutely as any newspaper circulation returns.

And what they've discovered is that with Google News, it's not enough to be first. Quite the opposite; it's best to be last, because then your site will top the list. If you're acute enough to spot someone else's exclusive and reword it, Google News will reward you by giving you more clickthroughs.

In the absence of that, though, some sites have discovered an equally good way to push themselves up the rankings: rejig a story. Thus something which might be languishing in the Google News rankings at 11am can be back on the top at prime clickthrough time (lunchtimes and the evenings) by some clever tweaking of content. In short, Google News, part of the most powerful site on the Net, actively works against exclusives.

The company itself denies that this is so - or at least that it's intentional. "The headlines on Google News homepage are selected entirely by a computer algorithm, based on many factors including how often and on which sites a story appears elsewhere on the Web," explained Debbie Frost, a spokeswoman for the company. News sites decide which stories are worth putting up; Google News then detects trends in coverage. Ms Frost said: "The story we run at the top [of a list] is not just the 'most recent' - it's determined by a number of factors, including 'first to break' and 'duplicate stories', but also factors like PageRank [Google's system for determining which of any number of pages is the most authoritative]."

Ms Frost insists: "Google News does take into account who broke the content first and does not just compile stories based on time."

Those at the sharp end don't see it that way. "Since I arrived on MacWorld five years ago I think we have achieved a fairly good reputation for occasional news breaks," said Jonny Evans, news editor on the magazine and site. "But I have definitely seen enough to confirm that trend, which is frankly a little frustrating."

Peter Kirwan, a former editor of Computing magazine and now head of Fullrun, a British information service for IT marketers and PRs, said that what we are seeing online is only the beginning: "This is a trend that started long ago. Readers using the Web to gather news was the first stage in what some publishers refer to as the 'commodisation' of news.

"The issue at this point wasn't exclusivity, but the fact that the media market's barriers to entry suddenly collapsed, resulting in a proliferation of news and what you might call pseudo-news ('let someone else do the hard work, and hire crap hacks')."

For online journalists, the worry is that their job will become completely deskilled, as bigger sites simply take any good content they produce and re-write it - and then get the benefits. Indeed, the tendency of Google News to rely on a site's "authoritativeness" also works against smaller sites, according to a study carried out in August by the American media consultancy Digital Deliverance. This found that although Google News checks roughly 4,500 news sources around the Web every day, the top 10 sources of news provide two-thirds of the stories that appear on the front of Google News - with dominant names like Reuters, The New York Times, Voice of America, Xinhua and Bloomberg providing 48 per cent.

"Is it possible that 48 per cent of stories should be coming from only five sources?" asked Vin Crosbie, who led the research. "Or that Xinhua and the Voice of America, official news sources respectively of the Peoples' Republic of China and the United States of America, are the third and fourth most prevalent sources of Google News? All that doesn't seem plausible, but the data shows that is how Google News is operating."

For its part, Google insists the news site is still in "beta" - that is, under development, although it has been publicly accessible since the end of September 2002. "We think your comments are very useful and have sent them to our product team," said Ema Linaker for Google UK, responding to this article. "It is exactly for this reason, to get more feedback, that we keep a product in beta."

The question that remains unanswered - but nags away - is this: if Woodward and Bernstein were working for an online publication now, would we know what it was? And would they be able to afford to keep doing it - or would their publishers point them to stories that generated better clickthroughs?