Andrew Anthony: My book upset a few people - and now I'm a racist

In a free society debate is open, but that doesn't mean it's necessarily challenging or productive. It often happens that a kind of ideological tribalism restricts ideas to discrete sections of the political spectrum, where orthodoxies are established less through fervent agreement than tacit acceptance. What starts out as radical can decline, through a mixture of intellectual allegiance and timidity, into the complacency of received wisdom.

When I set out to write my book The Fallout: How a Guilty Liberal Lost his Innocence, I wanted to look at questions from which I thought the liberal left shied away. What responsibilities do we have to each other as fellow citizens, say, in protecting one another against violent crime? Have we over-emphasised cultural identity at the cost of individual liberty? And is there a tendency to appease or admire totalitarian ideologies in the cause of "anti-imperialism"?

This was territory that for many years I had skirted around, deterred by the unpleasant consequences of trespass. To venture into this no-go zone was to risk being denounced as right-wing or, worse, racist. And that is precisely what has happened. I have lost count of the number of times I've had the right-wing racist charge levelled at me recently, most often by people who have not read my book. But while there's been no shortage of insults, there has been precious little analysis or counter-argument from the left.

Rather than discuss the issues my book raises, The Guardian simply diagnosed a midlife crisis. I had turned right, the reviewer decided, bizarrely claiming that I was angry at the Third World poor "for resenting my lifestyle". Time Out accused me of lying. The New Statesman suggested I had produced "expositions on race that come close to some of the arguments made by the BNP".

Really? What expositions? Oh, no need to go into all that, just make the slur and move on. Well, for the record, I believe race is close to a biological fallacy and such races that exist are all equal. I'm in favour of a multiracial society. And I want to see far more members of ethnic groups in positions of power. Having investigated and exposed the BNP, I think it's an unforgivable slander to suggest that I share its views.

Yet by far the most hysterical reaction has come in this newspaper. First Iqbal Ahmed wrote a review declaring. "Anthony believes that to be liberal is to be middle class?" That's news to me. "He despised the fact that in the Seventies it was cool to be black." Ditto. Among the litany of my crimes, he cites the fact that I criticise Tony Benn. What for, drinking tea? Being a loveable old cove? Ahmed didn't say. Perhaps because I criticise Benn for his opinion, recorded in his Diaries, that Mao Zedong was "the greatest man of the 20th century". Ranking the greatest murderer of the 20th century as its greatest man, is that not worthy of a liberal's criticism? And if it isn't, what is?

Ahmed characterised my parents as racists, and claimed that I think Britain did not benefit from its empire. In fact I pointed out that historians are divided over whether there was a net economic gain, but my contention was that it didn't do a lot for my grandparents, two of whom were veterans of the workhouse, while the other two died prematurely in poverty.

I mention in my book that my mother was fatherless at seven, evacuated at nine, and had left school, practically illiterate, at 14. She came out of an undereducated, white working-class background, and unsurprisingly absorbed some of the attitudes of her era. But she died in a south London cancer ward, filled with respect and affection for the black women patients she befriended. Hers was a journey, a positive tale of our times, but Ahmed omitted all of that.

Nevertheless, he was a model of intellectual engagement compared with Yasmin Alibhai-Brown. According to her column in The Independent, I am a "wretched" author of a "sorrowful confession", a "slash and burn apostate" and, for good measure, a "fanatic". Any details to substantiate this invective? Not likely. I wrote to Yasmin to try to establish what warranted this abuse. Her response was that, while she agreed with me on many issues, I was "closing down openness" and "Stalinising" liberalism. Again, no evidence, just accusation.

My book is a polemical memoir. It's not 'The Truth'. It's part of a debate. I may be wrong. It could be that if the troops are withdrawn from Iraq and we turn a blind eye to Islamic extremism at home, Jihadist terrorism will disappear. I happen to believe otherwise and base my case on historical fact and precedent.

It may be that an emphasis on cultural difference and group identities is the best means of fostering a harmonious and progressive society, but my own experience, observation and study suggests that it can and does have the contrary effect. It could be that by turning away from violent street crime and waiting for the government to make society more fair and equal, a shared sense of social responsibility will grow. But I don't think so.

These aren't fanatical or right-wing positions. They are concerns shared by millions – black and white, gay and straight, religious and atheist – in this country. Attempting to drown them out with shrill denunciations won't make them go away. But it will help leave the left estranged from reality. And that's when people really do turn to the right.

Andrew Anthony's 'The Fallout: How a Guilty Liberal Lost his Innocence' is published by Jonathan Cape at £14.99