The 'big beast' is felled and young Tory bucks breathe more easily

To the average voter, he seemed an approachable guy with ordinary ways, the sort you would like to meet in a pub. But his fellow MPs did not buy the avuncular image the public saw. They accused him of being stubborn, divisive, and arrogant.

He was the "big beast" who refused to budge when the party shifted to the right. He did not believe in "extravagant" tax cuts, did not get exercised about asylum-seekers, opposed the war in Iraq, and stuck to the pro-European views of his youth.

Even in his last appeal for support from his fellow MPs, on Monday's hustings, he insouciantly reminded them of how many points of disagreement he had with them, and seemed to imply that he was always right, and they were wrong. This refusal to give way, so attractive to voters, was a provocation to those faced with the prospect of having to work with him. For some MPs on the anti-EU right, nothing was more important yesterday than removing Mr Clarke's name from the ballot paper. To others, less fired by hostility to the EU, Mr Clarke was just too old, at 65, too remote, too much a name from the 1990s, and too disengaged.

They accused him of not being interested in his role as an opposition MP - to which he would have to plead guilty. But until yesterday, he had always clung to that faint chance that he might be the next Conservative prime minister.

That was something for which he hungered, which is why - according to those who worked with him - he took this leadership campaign more seriously than the previous two, in 1997 and 2001, when he was competing for the post of leader of the opposition.

This time, he believed the party was choosing a prime minister in waiting - "and boy, have you kept me waiting", he told the cheering crowd at the annual party conference this month. He had convinced himself that if he could somehow survive this week's eliminating rounds, he could win the final ballot of party members, and then lead the party to victory at the next election. Yesterday's defeat hit him hard.

But he was competing in a party very different from the one he joined as a student at Cambridge University, 45 years ago. That was a period when the Cambridge Conservative Club suddenly changed from being a social circle for former public school boys to a practice ground for ambitious young men from lower middle class backgrounds, such as Kenneth Clarke and Michael Howard, who in the 1950s might have joined the Labour Party.

He caused a scandal at Cambridge by inviting the British Fascist leader, Oswald Mosley, to address students. But from the moment he entered national politics, he was part of the liberal, One Nation, pro-European Conservative tradition associated with Iain Macleod and Edward Heath. He was a loyal young whip under Heath, and was never a true Thatcherite.

If Margaret Thatcher had felt strong enough, she would have kept him out of her Cabinet. In her memoirs, she praised his gift for presentation, and named as the sort of politician it was best not to sack, because he would be more trouble outside the Cabinet than in.

She also recorded that when her own future hung by a thread in November 1990, Mr Clarke was the first cabinet minister to troop into her office behind the Speaker's chair and tell her "in the brutalist style he has cultivated" that it was time to go. This was an act of matricide for which many Tories have never forgiven him. The late Alan Clark, for instance, loathed his near namesake, and described him in his diary as a "pudgy puff ball".

Yet despite Thatcherite hostility, Mr Clarke was one of the very few who served as a minister right through from 1979 to 1997. He was the only one of that handful who had also served in the Heath government in the early 1970s, which makes him the longest serving minister in postwar Britain - 35 years in Parliament, 21 of them in government.

He was thrown into the centre of a political storm when he became Health Secretary in 1988. Opposition to his health reforms, and the way he handled an ambulance drivers' dispute earned him the title of Britain's most hated minister. But even as he attracted the odium of the left, he also defied the right by insisting on the principle that health care was not something that could be subject to market forces, because the sick are not in a position to exercise consumer choice.

Appointed Education Minister in Mrs Thatcher's final reshuffle, he annoyed her by telling her he was not going to introduce the voucher system so dear to her heart - again on the basis that education should not be a market commodity.

With a customary absence of modesty, Mr Clarke claims to be one of the most successful chancellors since the war, making much of the fact that Gordon Brown stuck rigidly for two years to the spending plans he inherited in 1997.

But barely half of today's 198 Tory MPs were in the Commons before 1997. To the newer MPs, he was like an absentee MP, dropping in occasionally to make a weighty speech then going off to make money.

Andy McSmith is the author of 'Kenneth Clarke - A Political Biography' (Verso, 1994)

Three leadership bids, three defeats

1997: After John Major stood down as leader following the Tories' huge electoral defeat, Clarke was among the first to enter the leadership race. He stood against William Hague, John Redwood, Peter Lilley and Michael Howard. Among the most pro-Euro Tory MPs, he lost to the more Eurosceptic William Hague.

2001: Clarke stood for leader when Hague resigned after the 2001 election defeat, making it to the final round. He lost to Iain Duncan Smith.

2005: Clarke lost for the third time yesterday. He came fourth in the poll of Tory MPs, gaining 38 of the 198 available votes.

Geneviève Roberts