The public has made up its mind: the quiet man of politics is just a loser

They must get rid of IDS and elect one of the new generation. Oliver Letwin, Damian Green or David Willetts, arise. It is time

Sir Walter Raleigh was at dinner with some friends and family members one night when – according to the gossipy antiquarian, John Aubrey – his son suddenly started regaling the company on the subject of a visit that he'd made that morning to a whore. "I was very eager of her," said the young man, "kissed and embraced her, and went to enjoy her, but she thrust me from her, and vowed I should not, For (she said) your father lay with me but an hower ago."

The outraged Walter senior responded by giving his son a "damned blow" about his ears. The boy (though clearly a brute) was unwilling to strike his own father, and instead punched the face of the man sitting next to him, explaining: "Box about, 'twill come to my Father anon."

In the late summer of last year, during the last stages of the campaigning for the Conservative Party leadership, Iain Duncan Smith would tell Conservative audiences of his life as a rebel, and how he voted with the Opposition on 11 separate occasions as he attempted to scupper the Maastricht treaty, negotiated by the Tory Prime Minister, John Major. But it was all right (he would reassure his Eurosceptic audience), for had not the great Churchill himself shown something less than total loyalty to his party, when it conflicted with that higher thing – duty to his country?

The comparison was precarious even then, when we hardly knew IDS. Today it seems surreal. This autumn Mr Duncan Smith has put to the test all we thought he knew about the roles of charisma and communication in politics. He wrote a perfectly good party conference speech and murdered it. Far from challenging the gladiatorial nature of Prime Minister's Questions (as he could have done), he has simply emerged from his corner of the arena with a limp trident and his subrigaria round his ankles. He has fired his party chairman, failed to explain why and then left this enemy on the front bench. And now that revolt is in the air, IDS has his minions briefing ineffectively against the rebels.

He simply isn't any good at being leader and everyone but his most dogged supporters can see that the game is up. To get a minus 7 per cent rating among his own supporters in the period after the party conference (remember the concept of "bounce"?) is an almost heroic negative achievement. It means that the public has made up its mind about him. Mentally they have daubed a huge red L for Loser on the top of Mr Duncan Smith's head. Naturally some Tory MPs, unhappy at the thought that the next defeat could be as bad as the last, are beginning to wonder whether it wouldn't be best if the thing that has to be done were done now.

Equally naturally, those whose disastrous counsel has ruled the Conservative Party too often in the past decade now resist the necessary conclusion. The anti-IDS plotters were excoriated in yesterday's press. A Telegraph leader described them as "Ex-ministers and ex-whips from the dingiest corners of the Major twilight". Maybe, but at least there is some light at twilight, when "still to us comes love's old song/Comes love's old sweet song". With the quiet man, there are no songs at all.

In the Mail, the conspirators are the "bunch of losers who knifed Margaret Thatcher, tormented John Major and undermined William Hague" – a category that, as IDS has boasted, includes himself as an unapologetic member. Indeed, when taxed on his disloyalty, Andrew Mitchell – fingered by the leader's minders as one of the rebellious cabal that is currently rolling gunpowder barrels into the Tory cellars – made reference to being a whip in the early Nineties. That was when his job was made impossible by (though he named him not) the likes of Mr Duncan Smith – to whom Mr Mitchell feels he now owes no loyalty at all.Boxed about, it has come to IDS anon.

Meanwhile the shadow leader of the House, Eric Forth, is described as being "out of control", wandering the tearooms, reminding colleagues of the mere 25 names that are needed to trigger a no-confidence vote. The grass-roots right is unhappy with IDS's new-found Portilloism (didn't they vote just a year ago for the new Tebbit?), and the left and the centre – who support the new agenda – have now realised with a horrid certainty that their leader cannot sell it.

If we wait, say the plotters, we will waste the next three years and then go down to humiliating defeat. If we act, reply the plodders, then we will be seen as a totally useless bunch that sacrifices its leader every year and goes down to humiliating defeat. The problem is that they are both right.

The first and most obvious problem is who would replace IDS. David Davis is – we are told – a connoisseur's politician whose flavour it takes time and a refined palate to appreciate. Problem: there is no time, and no chance of refinement. Next!

So how about the man who should have got it last year (but has since ruled himself out)? The IDS bunch badly wounded him last year, and I can't help thinking that the light has gone out. Next!

Let me repeat this for the hard of thinking. Kenneth Clarke cannot become the leader of the Conservative Party because just about the only current reason these days for being a Tory – and not something else – is that you hate Europe. The idea floating around Westminster of a Clarke/Michael Howard "dream ticket", in which Clarke's Europeanism is "balanced" by Howard's Europhobia, is wonderfully funny (can you imagine how it would play on the Today programme?), but ridiculous. Next!

But there is no next. Not yet. And nor is there (the 25 policies notwithstanding) a mission that will capture the popular imagination. Neil Kinnock gained the headlines by smacking his own party around and by attacking Arthur Scargill. With evident courage he took on the impossibilism of Eighties Labour and – battle by battle – won. This year Theresa May and IDS told the Tories they were "nasty", and had the vast majority of them reply: "No we aren't! Now bog off!"

But they were nasty. IDS's own conference speech contained passages that – had they been delivered by anyone but IDS himself (or, to be fair, David Davis) – would have set the conference alight with controversy and excitement. Thatcher (he said, in effect) might have been great in many ways, but if you want to know how Britain turned mining villages into unofficial needle-exchange parks, then you have to look at her legacy. We now need (he did not quite say) New Conservative.

Too late for him to say that. Thatcher's sad orphans had voted for him because he represented the opposite – the blow has been returned. And in any case he is hopeless at selling the new big idea. This, to me, leaves only one possibility, but it is a hope that the Conservatives seem bound to disappoint. They must get rid of IDS and elect one of the new generation of New Conservatives. Oliver Letwin, Damian Green or David Willetts, arise. Agree between you who it will be, because your party needs and awaits you. It's time.

David.Aaronovitch@btinternet.com