Rupert Cornwell: Does America need so many spooks?

I left town earlier this month for a holiday with the headlines full of one spy scandal. I returned this week to be greeted by another. The first of course was the uncovering of a network of Russian "illegals", operating in the US under deep cover as innocuous suburbanites, tasked to scoop up secrets for Moscow Centre. In reality they seem to have scooped up next to nothing, but no matter. It was a splendid tale, redolent of a simpler Cold War age.

The second is less amusing, but in its way no less scandalous: the documentation by The Washington Post of the vast national security empire, part-public and part-private, that has developed here in the wake of the 9/11 attacks. The paper's investigation took two years and its conclusions are astounding. By its reckoning, no less than 854,000 people – almost one and a half times the entire population of Washington, DC, and roughly a third of them private contractors – have top-secret government security clearances. Over 3,000 federal agencies and private companies are working on intelligence, counter-terrorism and homeland security, at 10,000 sites across the country.

The empire is largely out of control, to the point that even Defence Secretary Robert Gates confesses he hasn't been able to establish how many private contractors work for his own office at the Pentagon. The myriad entities frequently overlap; there are apparently 51 separate bodies tracking the financing of terrorist groups.

Co-ordination between them is sketchy at best. Analysts across this empire spew out an estimated 50,000 reports – 150 a day – many of them unnoticed and unread. Officially, the US spends $75 billion annually on intelligence. The Post's findings suggest that this figure an underestimate. Even so, it would be fine if the US got value for these colossal outlays. But does it?

To be sure, the world is far more complicated than it was even a quarter century ago, when almost everything could be seen through the prism of US/Soviet relations. Now, new rivals like China have emerged, alongside non-state terrorist organisations, constantly mutating and so different from the Cold War's structured threats. And ever-present is the ultimate nightmare, that one of these groups will get its hands on a nuclear device and use it, against America or one of its allies. If the US accounts for two thirds of the world's intelligence spending (as it does by one authoritative estimate), that too is not entirely unreasonable, given the country's global reach and responsibilities.

And you could argue that the secret empire has succeeded in its primary mission of keeping the country relatively safe. Since September 11, 2001 no major terrorist attack has taken place on US soil, and if the likes of former vice-president Dick Cheney are to be believed, many have been thwarted (details, naturally, are not available, for reasons of national security). And of course, the FBI cracked that Russian spy ring.

But looked at another way, the empire's record is not good. It failed for instance to prevent the rampage of the army psychiatrist Nidal Malik Hasan at a US military base last November, in which 13 people were killed – even though his erratic behaviour and links with radical Islam were known. Nor was it top-notch security sleuthing that foiled the attempt on Christmas Day 2009 to blow up a US commercial airliner near Detroit. Various US agencies had a host of clues. But it took an alert fellow passenger to stop the young Nigerian Umar Farouk Abdulmutallab from detonating plastic explosives hidden in his underwear as the plane prepared to land. And while US forces have captured or killed many militants (and not a few innocent civilians as well) in Afghanistan, Yemen and elsewhere, they have failed to catch their number one target, Osama bin Laden.

And, one might add, we've seen this movie before. The American security establishment wasn't exactly small a decade ago. But one big reason 9/11 happened was a lack of co-operation, in that case between the CIA and the FBI. Nothing suggests that weakness has been cured across today's even more sprawling and unwieldy secret empire.

Up to a point, what's happened since has been typically American. The country's usual response to a crisis is to throw money and expertise at it. Often this works; witness re-armament in the Second World War, or the race to the moon 40 years ago. Since 2001, vast resources have similarly been channelled into intelligence and counter-terrorism.

Congressional gridlock is legendary – but not when national security is involved. Funding requests are invariably granted, often with little scrutiny of how those funds will be spent. As a result, the system has grown like Topsy. But even when deficit-cutting is at the top of the political agenda, no President, Senator or Representative facing re-election dares face the accusation he his not doing enough to keep the country safe.

But here a paradox arises. Americans, famously, are leery of bureaucracy and big government, and never more so than now, as shown by the rise of the Tea Party movement. Yet this huge, ill-coordinated and wasteful security bureaucracy has not been thrust upon the people by a wicked government; it has been created by popular demand. Bureaucracy by its nature is difficult to prune. This one however will be even tougher to bring to heel.

And it's not as if no one has tried. Six years ago, George W Bush created a new post of director of national intelligence to bring the country's fragmented and competing intelligence agencies under central control. But by all accounts the re-organisation has failed. Since 2005, the job has seen off three directors and, as the Post's investigation suggests, the baronies have only multiplied.

"OK, we've built tremendous capacity," asks Gates, who wants to reduce the number of private contractors in his office to pre-9/11 levels, "but do we have more than we need?" The answer, undoubtedly, is yes – but how to make cuts when you don't know how many contractors are working for you in the first place?

This week, the Senate was holding confirmation hearings for a fourth director. As did his predecessors, retired General James Clapper vowed he would provide the strong management required. He would not, he insisted, be a "titular figurehead or a hood ornament". Anyone want to bet?

r.cornwell@independent.co.uk

For further reading

The Washington Post – 'The Secrets Next Door': tiny.cc/4mym7