Wrong again, but – like Ukip – I'm learning

It might win three seats in next year's general election, but I still don't see where it goes from there

Nigel Farage rules out Ukip pact with Tory party (Getty)

If you want to know what last week's by-elections mean I may not be the best person to tell you. Eight months ago I wrote that Ukip would come second in the European Parliament elections and that the party was "finished". It won the largest share of the vote, 27.5 per cent, two points ahead of Labour and three and a half points ahead of the Conservatives. Last week, it won its first elected MP with a record 44-point swing in Clacton and came within two points of winning a second, this time from Labour, in Heywood and Middleton.

However, I have learned from Benjamin Zander, the conductor who taught his players, when they made a mistake, not to swear but to say, "How fascinating!" The important thing is not that I was wrong but to work out why and to learn from it. The fascinating part is that I was wrong about the Scottish referendum too, for a similar reason. A month beforehand, I thought the Yes vote for independence would be 40 per cent. As we know, because every tribal nationalist on Twitter has the number as a badge of defiance across their picture, it was 45 per cent. (I shouldn't be too rude: I had a "Don't blame me, I voted Labour" badge in 1983, which came from the same "moral victory" school of respect for democracy.)

In both cases, I was wrong because I failed to realise how strong the anti-politics force was. Of course, "anti-politics" is a simplification, and to describe Nigel Farage and Alex Salmond as exponents of similar rebellion against the established order is insulting to both of them. Nor is this ruling sentiment unpolitical. All the manifestations of "anti-politics" are deeply political: Ukip, the Scottish National Party, the Green Party and all those whizzy internet things that I, in my new learning mode, must try not to dismiss as clicktivism.

They are anti-politics-as-usual. They express a yearning that has always been around for something different, better, but never specified, but which waxes and wanes in intensity. It was intense when the Social Democratic Party was formed in 1981, the liberal, progressive, middle-class Ukip of its day. In one of the richer ironies of British history, the last time there was a by-election in Clacton, the Labour candidate, who lost to the Tory in 1954, was 23-year-old Shirley Catlin. Better known today as Baroness Williams, one of the Gang of Four who founded the SDP.

Here, we come to the more familiar lessons of the past. The astonishing rise, arrested development and merger of the SDP into the Liberal Democrats after seven years shows how resilient the British party system is. Ukip is unlikely to be even as successful as the SDP was. It might win three seats in the general election next year rather than one, but I still don't see where it goes from there. The general election will be decided by the competition between Labour and Conservative. Ukip is relevant to the outcome only if it has a differential effect on Labour and Conservative votes.

So what effect will it have? First, Douglas Carswell's emphatic victory in Clacton is bad news for the Tories. It means that they have no hope – which at their party conference they thought they had – of holding on to Rochester and Strood. Mark Reckless is not as popular with his constituents as Carswell is, but we have to remember: the anti-politics force is strong.

However, the result in Heywood and Middleton was more significant. For Ukip's support to rise from 19 points behind Labour in two opinion polls just a week and a half before to two points behind on polling day suggests that the Labour vote is soft.

We know that more of Ukip's current support is drawn from those who voted Tory in 2010 than from those who voted Labour. But those Ukippable Tories have gone already, and Labour's average lead in the polls is just two points. I guess that the "go to bed with Farage and wake up with Miliband" argument would check any further Ukip advance with Tory voters, whereas if there is an anti-politics surge in the general election it would be Labour that would suffer.

To make matters worse for Miliband, he faces an anti-politics challenge on two other fronts. The Green Party is running at 5 per cent in national opinion polls. The left-wing protest vote that defected from the Lib Dems to Labour the moment the coalition was formed in 2010 now has somewhere else to go. In Scotland, the derangement of the 45s, a leftist insurgency from Labour into the Scottish National Party, could deprive Labour of seats and ensure that Nicola Sturgeon, the incoming leader of the SNP, benefits from the retreat of the Lib Dems in Scotland.

It is not even as if Miliband could use the trick of exceeding below-sea-level expectations to gain a boost from the television debates, because there won't be any – Cameron will find a way of confusing the issue. There is only one thing to do: send for Alan Johnson.

twitter.com/@JohnRentoul