John Kampfner: Liberal values have never been more important – or less popular

People everywhere have been too willing to trade freedom for prosperity or security

Why has the Left become so suspicious of civil liberties? Why has the Right hijacked the issue and turned it to its advantage? A dozen years of New Labour rule have left this sorry legacy. A party that should have intervened for social justice and greater equality instead allowed the markets to let rip. Having raised the white flag to the bankers, ministers instead sought to exert their power elsewhere, at the level of the citizen, seeking ever more ingenious ways of watching us, listening to us and telling us how to lead our lives. I am no Freudian psychoanalyst, but I can find no better example of displacement theory in modern politics.

In 1997, New Labour contained several prominent figures who cared about these issues. They were soon replaced by machine politicians who saw the delivery of populist outcomes as the most important marker of success. Civil liberties were reduced to a lobby rather than a core part of the political project.

From ID cards to CCTV, to a national DNA database, to long periods of detention without charge, to public-order restrictions on protest and curbs on free expression through draconian libel laws, this government rewrote the relationship between state and individual.

By the time Blair left office, he had bequeathed to his successor a surveillance state unrivalled anywhere in the democratic world. Parliament passed 45 criminal justice laws – more than the total for the whole of the previous century – creating more than 3,000 new criminal offences. That corresponded to two new offences for each day parliament was sitting. The scope was broad; police and security forces were given greater powers of arrest and detention; all institutions of state were granted increased rights to snoop, and individuals were required to hand over unprecedented data. Abroad, the government colluded with the transport of terrorist suspects by the US to secret prisons, giving landing rights at British airports for these so-called "rendition" flights, while serious questions were raised about the UK's role in torture.

The crisis of liberalism in the British Left went beyond Tony Blair's willingness to please George W Bush. It has produced a situation in which those who complain about individual rights are regarded with something between disdain and outright hostility. My new book tackles the issue head on. I look at eight countries, four notionally authoritarian – Singapore, China, Russia and the UAE – and four notionally democratic, India, Italy, the UK and the United States. My central thesis is that people around the world, whatever their different cultures or circumstances, have been willing over the past 20 years of globalised glut to trade certain freedoms in return for the promise of either prosperity or security. We have elevated private freedoms, especially the freedom to earn and spend money, over public freedoms, such as democratic participation and accountability and free expression. I call the thirst for material comfort the ultimate anaesthetic for the brain.

My British chapter is the most critical, not from the point of comparing the performance of each state (one could not sensibly equate anything here with Putin's Russia, for example) but from the point of view of expectation. In Britain, we pride ourselves on our liberal traditions. We have moved on to new terrain. My critique led a supposedly Left-wing commentator to challenge me only the other day: "What kind of political journey have you been on?"

My answer is: none at all. I believe as passionately as before in the politics of economic interventionism, constitutional reform, an ethical foreign policy and an enlightened but also sensible approach to criminal justice and human rights. This was the politics of people such as Robin Cook and Mo Mowlam. Their passing, and the sidelining of others like them, has led to the rise of what I term "Thuggish Labour", ministers who rarely show the courage to challenge the lowest common denominator. "Do whatever it takes" has become the mantra, particularly on security.

Two arguments are put to me by people who still label themselves as liberal or of the Left, but who see little problem with the surveillance state. One says: poor people are more likely to be victims of street crime, so if you believe in social justice, you should have no objection to tougher measures. Well, yes, but is not the most pressing issue that has led to crime and despair the link between inequality and the lack of social mobility which this government has been so scared to tackle?

The other admonishment says: surely you want your daughter to come home safely from school, don't you? Well, yes, but does that mean we need cameras on every corner? If you believe that the task of government is to be the protector against all risk then, logically, there is no balance to be struck, there is no place where rights triumph over security. Arguments such as those above are intellectually no more sophisticated than the tabloid, "Lock 'em up and throw away the key".

I had never disavowed the role of the State in helping to provide more equitable social outcomes. Several spells of living in continental Europe in the 1980s and early 1990s made me appreciate the more communitarian spirit in a number of countries, where good neighbourliness and social responsibility were derided as "nanny-ism" in a Britain where Margaret Thatcher had famously said there "was no such thing as society". I carried an identity card in Germany and in Spain, and neither I nor my friends thought of questioning it. I took that view back to the UK, but a few years of life under a succession of Home Secretaries with a thirst for authoritarianism made me change my mind.

So now we have the bizarre situation of civil-liberties advocates being denounced as Right-wing. This is a sad reflection not on human-rights campaigners but on the tribal Left who make these claims. And those of a genuinely Conservative disposition have been allowed to seize the mantle of battlers for individual rights. Invoking John Stuart Mill as one of their own, they have reframed his thinking in patriotic, libertarian tones. Their subtext is: regain the rights for freeborn Englishmen and roll back the State.

This is not a zero-sum game. This Labour government could have been more courageous not just on civil liberties, but also on social justice, concepts that should be mutually reinforcing. This is the opportunity lost and lamented, but, perhaps some way down the line, it can be regained.

"Do we really want the police, security services and other organs of the State to have access to more and more aspects of our private lives?" The question was posed last year by Richard Thomas, the government-appointed Information Commissioner, in his annual report. "Sometimes the best-intentioned plans bring the most insidious threats, where freedoms are not appreciated until it is too late to turn the clock back." Is he a libertarian? I very much doubt it.

John Kampfner is chief executive of Index on Censorship. 'Freedom For Sale' is published by Simon and Schuster