So, David Cameron, why did you fail to seal the deal?

You started out against the most unpopular PM of recent times. The Lib Dems were running at less than 20 per cent in the opinion polls. The right-wing press was uniformly behind you and you had a double-digit lead for most of the four years you have led your party

(JUSTIN SUTCLIFFE)

Shortly before 5am on the morning after polling day, designer Anya Hindmarch and TV presenters Carol Vorderman and Kirstie Allsopp mixed with sharp-suited young men and fashionable twentysomething women in the River Room at the Conservative Party's Millbank headquarters.

A lady in a floor-length black fur coat and immaculate grey hair arrived with a giant Harrods bag. Only a statuesque young black woman in a bright orange body-con dress stood out from the sea of white faces.

The hundred-strong crowd of Tory party workers and supporters was boisterous and bleary-eyed. Goblets of red and white wine lined the bar, while a popcorn machine and candyfloss stand created a carnival atmosphere. Blue uplighters bathed the room in an otherworldly glow, while posters on the wall boasted "Bye bye Bureaucracy" and "Strong Families; Strong Society".

David Cameron, who had just arrived from his Witney constituency, delivered an upbeat message, assuring workers that if they achieve 300 or more seats they'd have the "moral authority" to kick Gordon Brown out of Downing Street.

But something was not quite right. When Mr Cameron arrived at the party he looked rather dejected. The eight vast TVs screening the BBC election coverage should, according to the script of the Cameron project, have brought news of the Conservatives' return to power after 13 years. Yet the images of Tory gains were interspersed with too many scenes of missed targets: there was no "Portillo moment" for Ed Balls, the Schools Secretary, who beat his Tory opponent by 1,101 votes.

Lucy, a young brunette in an eye-catching yellow and green frock, burst into tears when it was announced, around 5.15am, that Shaun Bailey had failed to win the seat of Hammersmith and Fulham. Sobbing, she said: "But he would have been so good for the party."

Mr Bailey, the 38-year-old black former youth worker, epitomised the new wave of candidates symbolising Mr Cameron's "changed" party and had been tipped as a star of the 2010 intake. He lost by 3,459 votes to Labour – not a narrow margin.

As the sun started to rise over the Thames and the crowd thinned out, waiters circulated with trays of mini croissants and pastries. Weary-looking guests drifted to tables lined with pots of strong coffee and tea, then wandered, blinking, out into the cool May morning. It was a sobering scene.

The Conservatives' final tally of seats – barring the delayed result in Thirsk and Malton later this month – is 306, 20 short of the winning post.

So why – with an unpopular Prime Minister at the head of a 13-year-old Labour government and a recession under his belt, a Liberal Democrat party that failed to improve on its 2005 performance, and the backing of the majority of newspapers, including The Sun – did Mr Cameron not seal the deal with the electorate?

The Tory operation during the four-week campaign was slick, but there were missed opportunities and strategic problems. The most obvious tactical error was Mr Cameron's agreement, struck weeks ago, to give Nick Clegg, the Liberal Democrat leader, an equal platform in the three televised leaders' debates. In the first, on 15 April, Mr Clegg's performance was revelatory, capturing the nation's "anti-politics" mood. It diluted Mr Cameron's ability to offer himself as the fresh-faced "change" candidate, because Mr Clegg was, to the audience watching at home, fresher.

Adrian Pepper, a former Tory government special adviser and now a polling and campaign strategist, said Conservative campaign chiefs had focused too much on differentiating Mr Cameron from Gordon Brown. They had been apparently taken by surprise by Mr Clegg in the first debate and were still wrong-footed in the second. Mr Pepper said: "It took until the third debate for David Cameron to expose Nick Clegg's amnesty on illegal immigrants and his policy on the euro, but by then the Conservative campaign had lost a lot of momentum."

Tim Montgomerie, the editor of the grass-roots website conservative home.com, warned months ago that Mr Cameron should not have agreed to the debates because as the frontrunner he risked losing his lead.

Yet, in the end, the "Cleggmania" created by the debates was neutralised by polling day. The Lib Dems' poll ratings of early to mid 30s during the campaign never materialised: they achieved 23 per cent of the popular vote, the same as in 2005. This suggests something more fundamental was wrong with the Tory campaign which led to Mr Cameron's offer to the electorate falling short.

Mr Pepper questioned the strategy of putting so much emphasis on Mr Cameron's persona at the expense of other members of the Shadow Cabinet. "The party only managed to increase share of vote marginally above Michael Howard's in 2005, so serious questions need to be asked.

"It was a risk to run a campaign based around David alone. People were asking why William Hague was not more visible in the north and why Ken Clarke wasn't more prominent in the Midlands. You have to look behind the personal popularity numbers to identify which voters your politicians are popular with."

Tory strategists deserved credit for trying to fight a positive campaign with slogans such as "I've never voted Conservative, but ...". But not enough emphasis was placed on the negatives of the Labour government, Mr Pepper said. He added: "In any campaign you have to show that the other side doesn't deserve to be in office, alongside positive messages about why you need to be in office."

What of the Tory policies? Mr Cameron's flagship idea, the brainchild of his head of strategy Steve Hilton, is the "Big Society", but it left voters on the doorstep baffled. Critical insiders say it was launched too late – just a week before the election was called on 6 April – leaving not enough time to convince the public it was workable. Instead, it was reported as a "DIY government", with question marks raised over how hard-pressed people would fit in a bit of gardening or painting for their local council estate in between taking the kids to school and putting in a day's work.

The other flagship policy, the tax break for married couples, took a backseat position, despite it being so central to the Cameron message on stable family life being the answer to mending Britain's "broken society".

But the difficult economic climate meant any promise on tax could be picked apart by opponents. The party "won" the first week of the election campaign when dozens of business leaders came out to back Mr Cameron and George Osborne in opposing the government's 1 per cent rise in National Insurance contributions from next year – Labour's "jobs tax".

Yet this letter-writing could not sustain the whole campaign. There was also an obvious contradiction between this policy and the Conservatives' core message of austerity and fiscal restraint. The party was forced to tread carefully between promising immediate spending cuts to reduce the deficit and offering voters optimism and hope for the future – and so the message became confused.

Mr Pepper said the Tories should have been more honest in telling voters what cuts were coming. "Some people were scared to vote Conservative because we had not spelled out in enough detail what we were going to do. The idea of an austerity package was not a turn-off – it was the fact that people didn't know what it would mean for them. If I had to have an operation, and if the surgeon doesn't tell me what the operation is going to involve, I am going to be far more scared. But if the surgeon tells me what is involved I will most likely accept it."

Other areas were ignored for fear of repeating the catastrophic defeats of 2001 and 2005. Immigration, on the orders of senior Tories, was off the agenda, despite it being consistently the number two issue for voters after the economy. Only when it provided an opportunity – to attack Mr Clegg's migrant amnesty – was it thrust to the forefront of the Tory agenda.

There is also the balance of power between the central figures in the Tory operation. Mr Hilton and communications director, Andy Coulson, have clashed over how hard to push anti-Brown messages. There has been constant criticism from the right, particularly about the cliquey nature of Mr Cameron's team. James Forsyth of The Spectator magazine wrote last week: "The campaign has highlighted the insular nature of the Cameron operation. Talk to Tory big beasts and their most frequent complaint is that their telephone calls have not been returned. Alarmingly few people were kept in the loop during the campaign."

But it is perhaps too simplistic to go looking in the four-week campaign for reasons why Mr Cameron could not seal the deal. The Tory poll rating went beyond 40 per cent just once since the election was called on 6 April, when they reached 41 per cent, nine points clear of Labour and 23 points ahead of the Lib Dems. That poll, for YouGov/The Sun, was taken on 14 April, the day before the first televised leaders' debate.

Yet support in the polls was already dropping before last month. Mr Cameron's leadership, which began with so much hope in December 2005, needs to be looked at in its entirety to see why the project veered off course.

The Conservative modernisation strategy, it can be argued, was based not on having any radical game-changing policies but on combining changing the perception of the party – adopting "modern" candidates like Mr Bailey – with a modern economic model. Before the summer of 2008, there was a firm policy of not clashing with Labour on the economy but instead matching their spending plans.

All Mr Cameron needed to do was offer himself as a fresh start, centre-right, progressive alternative to a tired old Labour government and the election would fall into his lap, because it was the "turn" of the Conservatives. Until the economic crash of autumn 2008, this was a sensible strategy.

It is of some significance that the highest poll rating of Cameron's leadership was on 14 September 2008, when 52 per cent for Ipsos Mori gave the Tories a thumping 28-point lead over Labour. The next day, Lehman Brothers collapsed and the world's financial markets went into meltdown.

The economic crisis demanded a response that was distinct from the Government's, Tory strategists reasoned. The Conservative Party could no longer match Labour's spending plans, because the Government had got the country into the economic mess and ratcheted up a huge deficit. Instead, they went for austerity. Mr Osborne delivered the grim message to a Birmingham conference hall in October 2008: "The cupboard is bare."

Yet when this Tory austerity message was tested with voters, there was a steady decline in the polls. Mr Cameron and Mr Osborne then qualified their signals, suggesting the party was uncertain about its response.

The party was forced to sideline its modernisation agenda, as many of the policies were too costly. The right wing that Mr Cameron had been determined to avoid had to be squared with promises of cuts and low taxes.

Then a year ago yesterday, another truly game-changing crisis hit British politics: the expenses scandal. Mr Cameron's response to this was more sure-footed. Despite the individual embarrassments for Tory MPs over their moats and duck houses, the Tory leader made an early pitch as the clean-up candidate, ahead of his rivals. But it left the party suddenly fighting an anti-politics mood. It was difficult to offer change when the party was associated with some of the most memorable expenses claims.

In January this year, the Tories had a "soft launch" of their election campaign, confirming they would include a tax break for marriage, but not sketching out further details. A poster of an apparently airbrushed Mr Cameron, alongside the words "We can't go on like this; I'll cut the deficit, not the NHS" was widely criticised. Advertising experts said it was a mistake to place the words "cut" and "NHS" so closely on a Tory poster. It was also ridiculed in thousands of spoofs.

Mr Cameron then appeared to dilute his marriage tax plans in an interview with BBC's political editor, Nick Robinson. By the end of January, the strategy appeared more confused. During an interview at the World Economic Forum in Davos that month, Mr Cameron sounded a note of caution on Tory plans for savage cuts to public spending. This conflicted with a hardline message from Mr Osborne.

The electorate seemed to pick up on the uncertainty: since the new year, the Tories have been routinely scoring in the mid 30s in the polls, their lead over Labour reduced to below 10 points after hitting the mid 40s in 2009.

But something else happened in January: Britain emerged from recession, albeit only marginally. Yet it underscored the Labour argument that they had got it right on the economy, and that only Gordon Brown was to be trusted to steer Britain into recovery.

Conservative insiders concede other mishaps that did not help the Cameron project. Having relegated the need to modernise and detoxify the party, perhaps complacently taking it for granted that this had taken place, there were new signs of toxicity and that the party had not changed. There was the sleaze row following the revelation that Lord Ashcroft, Conservative deputy chairman and a key funder of the marginal seats the Tories were targeting, had been non-domiciled for UK tax purposes. There were also unfortunate comments by leading Conservatives on gay rights, including the shadow Home Secretary, Chris Grayling, who said people who ran B&Bs from their own homes had a right to turn away homosexual couples.

By contrast, Labour's election campaign was a disaster. There were behind-the-scenes clashes between Peter Mandelson, Ed Balls and Harriet Harman, who all wanted to impose their influence on the campaign. Yet still the Conservatives did not prevail.

The Prime Minister's personal campaign was defined by two contrasting moments: when he called Rochdale pensioner Gillian Duffy a "bigot" and when, last week, he gave a passionate speech on fairness and social justice to Citizens UK. The Tories' response to the Duffy affair was to rise above it. There have since been mutterings that they should have gone in for the kill. Mr Brown's rousing speech was overshadowed by mixed messages from cabinet ministers on tactical voting. Perhaps the closing stages of Labour's campaign did sway some votes.

The Lib Dem vote appeared to collapse in seats where the Conservatives were up against Labour – suggesting some tactical voting did take place. There was evidence of former Labour voters who had toyed with the Lib Dems switching back to their old party, galvanised by the looming threat of a Tory government. Ultimately, there were too many voters who felt unable to rush into Mr Cameron's arms: proof that the detoxification strategy was never completed.

On Thursday night, before Mr Cameron went for a pre-declaration drink in his local pub, he spent the evening chopping wood in his garden. Perhaps Tory spin doctors felt this made him sound like an ordinary family man. Yet it left the impression that he was taking out his frustrations on a chunk of tree. This weekend, as he tries to strike a coalition deal with the Lib Dems instead of putting together his first all-Tory cabinet, he is clearly a frustrated man.