John Rentoul: Brown will be ditched. But when?

Labour does not need to lose another by-election to know that it is all over for this Prime Minister

It feels like it has all fizzled out. All the end-of-term excitement at Westminster about Gordon Brown's uncertain hold on No 10 seems to have subsided. Was it all just the silliness induced by the imminence of the holidays? Just as, at school, the older pupils play practical jokes and teachers break the routine with a bit of fun. You can tell what sort of school I went to: for our maths teacher this consisted of doing a whole lesson in base six; most teachers nowadays just put on a video.

At the end of term, anything seems possible. The head prefect, Miliband Major, was allowed to hold his own assembly and pretend to be the headmaster, and the pupils all went off for the summer full of the possibilities that the new term holds. I had already left London, but I was told that the atmosphere at the Foreign Secretary's news conference with a bemused Italian foreign minister, on the day that David Miliband published his "leadership bid" article, was headily expectant.

Labour had just lost a Scottish by-election to the SNP by the sort of swing that spelt electoral meltdown. But MPs were already heading out of the school gates. Most Cabinet ministers, among whom the plots that matter will be hatched, were only too keen to get away, leaving only Harriet Harman and Alistair Darling arguing over who was in charge of the shop while the boss was away – and Miliband, advertising his availability.

Now the new political term is about to begin, and the anything-could-happen mood seems a distant memory. The headmaster is back in his study, pondering a lesson plan entitled, "Barack Obama and the vacuity of 'change' as a slogan." The political news is being driven by the vote-losing tendency of the Labour Party, agitating for a windfall tax on the energy companies instead of looking on high oil prices as an opportunity to promote green objectives.

And, for Labour MPs, the opinion polls are still dire. Some of them, possibly in a state of shock after Glasgow East, went on holiday thinking that the 20-point Conservative leads were not real and would return to figures that did not look quite so life-threatening. But no, the average Tory lead during August increased slightly.

The only thing that really matters is whether Labour MPs think that ditching Gordon Brown for a new leader is likely to make matters better or worse. At the moment, they may think that they do not have to decide, but it is not an assessment that they can put off for ever. And it is not an assessment that will be made in Brown's favour. The Labour Party does not need to wait for further evidence that something fundamental is broken about its leader's electoral appeal. The opinion poll that found that 42 per cent feel sorry for the Prime Minister; the jokes about Team GB slipping from third to fourth in the medals table the moment he arrived in Beijing; the fact that Boris Johnson – "ping pong is coming home" – thoroughly upstaged him there. Labour does not need to lose another by-election in Scotland to know that it is all over for Brown.

I remember a long time ago, when the Labour Party was hauling itself back from the wilderness of opposition with the help of groups that described themselves as "soft left". At one meeting, Paul Thompson, editor of a magazine called Renewal, was taking questions from the audience but ignoring an attractive young woman at the front who had her hand up all the time. Eventually, when someone else in the audience protested, he said: "I'm not taking questions from the Revolutionary Communist Party. We may be the soft left, but we're not that soft."

The same applies now. The Labour Party may be the nice party, but it is not that nice. David Miliband may not be the compellingly popular alternative – if he were, I do not doubt that he would be prime minister already. Indeed, the Foreign Secretary's belligerent warnings about Russian "aggression", repeated in the Ukraine yesterday, are possibly ill-judged in policy terms, but they keep him on the media stage and make him sound tough.

The revolt against Brown was never going to be a continuous build-up. These things tend not to be. With Blair, the conditions were set, first by the Iraq war and then by his refusal to condemn the Israeli invasion of the Lebanon in 2006. But it was not until a totally unrepentant interview in The Times seven weeks later – after the summer holidays – that Labour MPs went into what Nye Bevan might have called an emotional spasm. In less than a week, Blair had been forced to hand in his notice.

The crucial fact, on which Labour MPs have pondered by the pool side, is that there is no hurry to get rid of Brown yet. They know that a new leader would have to promise to go to the country quickly. A new leader really would have no personal mandate (unlike Brown: anyone who voted Labour in 2005 knew he was likely to be prime minister – some may have voted Labour for that very reason). I think that Labour MPs are wrong to fear an early election. The longer they leave changing their leader, the weaker their party becomes – down to 176,891 members, it was quietly reported over the summer – as Cameron grows stronger. But MPs have salaries and pensions to think about: no wonder they are cautious.

So, it may look as if the mutiny has fizzled out. But, as the last possible date for an election – 2 June 2010 – starts to be measured in months rather than years, they also have personal interests at stake in doing anything that might possibly give Labour a better chance than it faces under Gordon Brown. As one former Cabinet minister told me before the summer, "The one fixed point of certainty is that we are going down if GB stays. To get rid of him is a risk. It is a very big thing, but it gives us a chance." It may not do any good but, at some point, Labour MPs will try to save their skins.

The writer is chief political commentator for 'The Independent on Sunday'