The rise and rise of Ukip – the new party of the protest voter

Once dismissed by the Prime Minister as 'loonies', Nigel Farage's party benefited from anger at the coalition to exceed all expectations in last week's by-elections. Are they really about join the mainstream?

In the end, Ukip's announcement of a historic second place in the Rotherham by-election was as telling as the result itself. A colourful headline, next to a picture of a blonde, declared: "Jane rocks Rotherham as coalition flops!"

It was more suited to a red-top newspaper than the website of the traditional home of middle-aged Eurosceptic men who take The Daily Telegraph. After almost two decades of ridicule, in-fighting and shouting from the sidelines, Ukip is finally within touching distance of mainstream British politics.

The steady rise of the party originally known as the United Kingdom Independence Party has spanned a decade, taking in a second place at the 2009 European Parliament elections and extending into its remarkable performances in three parliamentary by-elections on Thursday. Ukip is now widely predicted to win the next European elections in 2014.

"Our previous best-ever by-election result, a fortnight ago, was 14.3 per cent and this one is comfortably over 20 per cent," Ukip's oddly charismatic leader Nigel Farage declared on Friday. "The political establishment is just going to have to wake up to the fact that Ukip is here and here to stay as a significant and rising mainstream part of British politics."

Ukip is still far from winning a parliamentary seat, but its most recent achievements are acknowledged with some concern by the three main parties. Mr Farage's claim that he is now leading the "third force in British politics" might be a little overexcited, but after Thursday, the Liberal Democrats have been put on notice that they are in mortal danger, as their traditional ability to vacuum up protest votes is challenged. Senior Conservatives are openly debating an electoral pact with a party David Cameron once dismissed as a bunch of "fruitcakes, loonies and closet racists", in an effort to neutralise the electoral damage Ukip could wreak on the Tory Eurosceptic vote. Mr Farage is demanding a place on the podium at the leaders' debates during the next general election.

It is a far cry from the early days, when Ukip – founded from the Anti-Federalist League by the academic Alan Sked to campaign for the UK's withdrawal from the European Union and dominated by middle-class males of a certain age – struggled to cast off its oddball reputation. In its first venture into parliamentary campaigning, at four by-elections in June 1994, its candidates – including Mr Farage – won a total of 2,324 votes. Mr Sked claimed they would win "six or seven" seats at the 1997 general election, but their 193 candidates garnered only 0.3 per cent of the national vote between them.

"We were all fiercely passionate about the cause, but it was a bloody nightmare from start to finish," one veteran of the 1997 campaign recalled. "We had no money and not much organisation to speak of; we were characterised as a quaint bunch of eccentrics and we couldn't do anything about it."

Fifteen years on, Ukip is almost unrecognisable, as a fighting force but also as a party. The outfit once derided as monocultural, single issue and single gender has made strenuous attempts in recent years to diversify.

Ukip has a "youth wing" – "Ukip's social and political home for people under the age of 35" – which party officials claim has almost 1,000 members. Although critics insist the membership of Young Independence is closer to 700 – a fraction of the size of similar groupings in the larger parties – its ability to attract young members is a signal of the fundamental shift in the party's image. Mr Farage's deputy, Paul Nuttall, is a Ukip MEP, a former Young Independence secretary and a youthful 36.

Ukip has also attempted to counter accusations of xenophobia – or even racism – by broadening its candidate base. The list of Ukip MEPs includes Marta Andreasen, an Argentinian-born Spanish accountant, who used to work for the European Commission. Mr Farage dismissed accusations of racism last month by pointing out that the Ukip candidate in the Croydon North by-election was a Jamaican-born former boxer, Winston McKenzie. The row after a couple had three foster children removed from their care by Rotherham Borough Council because they belonged to Ukip also helped the party counter any accusations that they were "extremist".

The party's website is dominated by Europe, but it does lay out its positions in a number of areas of domestic policy, including tax, health and same-sex marriage.

Joe Twyman, director of political and social research at the pollsters YouGov, detects a conscious move towards the mainstream, which has been aided by an unpopular government and concern over the direction of the European Union. Ukip is becoming a haven for protest voters who have lost their political home.

He said: "The people who want to use by-elections to register a protest vote against the Government can't go to the Lib Dems any more, as they are the Government. Ukip's preoccupation with Europe is a commonly referenced one, particularly at the moment, and there are votes in it.

"They are clearly trying to position themselves as not just in effect a pressure group that you can vote for. They are trying to put across a message about the economy linked to immigration and Europe."

Ukip has not abandoned all its right-wing convictions, however, leading to accusations that the party is simply opposed to the modern world. The policy platform remains obsessed with Europe, and Mr Nuttall's blog contains diatribes against political correctness and in favour of the death penalty. Last month, Mr McKenzie provoked uproar when he claimed that allowing gay couples to adopt was "unhealthy".

Nevertheless, the other parties are concerned. The Conservative deputy chairman Michael Fabricant talks of an electoral pact with Ukip, while Labour's former home secretary Charles Clarke claims the mainstream parties have to listen to Ukip preoccupations because they reflect the concerns of many voters. He said: "I don't think Ukip is even so much about Europe; it's about the fact that the people you are talking about want somebody else to vote for other than Labour or the Conservatives."

Yet experts expect Ukip to continue its rise while the coalition is unpopular, and they might well win the Euro elections in 2014. But parliamentary seats may be beyond them. "Even Ukip would be very surprised if they actually won a parliamentary seat at the next election," Mr Twyman said. "But what they can hope for is to be king-makers in local areas. They may be able to take votes off candidates from major parties and change the result, which would give them an influence that outweighs their share of the vote."

Ms Andreasen agrees, for now. "It may be true that we exist primarily in the European Parliament but we will really take off when we get representatives elected to the Commons," she said. "This the key to our future success. Without it, it is hard to see how we can grow further."

The latest results strengthen Mr Fabricant's conviction over an electoral pact with Ukip. He said: "If Ukip does well in the European elections, it will give them momentum. By-elections are not general elections, when Ukip can only be 'spoilers', not winners. For that reason, all parties should keep their options open."