Andrew Grice: Cameron's terrible week shows he can no longer get benefit of the doubt

For David Cameron, the worst moment in a torrid week came when he watched a TV news bulletin showing the footage of Peter Cruddas, the Conservatives' former co-treasurer, telling undercover reporters that a £250,000 donation to the party would secure dinner at the Prime Minister's Downing Street flat.

Mr Cameron cringed. In his mind, a few boastful and totally untrue statements by a newly-appointed fundraiser could undo his painstaking efforts to put Tory finances on a more healthy footing. When he became leader in 2005, the party was dependent on huge donations from a handful of millionaires. Today, it has broadened its funding base and receives many £50,000 donations.

There was plenty of competition for Mr Cameron's worst moment. His most sticky period since becoming Prime Minister began with last week's Budget. Mr Cameron believed the Government could win the argument over George Osborne's controversial decision to cut the 50p tax rate on incomes over £150,000. But he had not reckoned on the outcry over the "granny tax"– the freezing of the personal allowances of pensioners. The two were a toxic combination, reviving suggestions that the Tories are the "party of the rich".

The image was reinforced by Mr Cruddas's ill-judged remarks to Sunday Times reporters posing as would-be Tory donors. The huge amounts of money involved – and the explosive suggestion that donors could buy influence over policy – echoed Labour's favourite attack lines on "out of touch" Tories looking after their rich friends.

There was worse to come. Amid near farcical scenes, Mr Osborne and Mr Cameron both tripped up over the Chancellor's decision to extend VAT to pies and pasties sold hot. First Mr Osborne could not recall when he last had a pasty and was dubbed "Marie Antoinette". When Mr Cameron tried to avoid the same trap, he fell into a deeper hole. He recalled his most recent pasty – at Leeds railway station, but a sceptical media soon established that the shop he had claimed to visit closed five years ago.

Francis Maude, the Cabinet Office Minister, added to the impression of an "out of touch" ruling party by playing down Mr Cameron's "kitchen suppers" with Tory donors and then suggesting motorists beat the threatened tanker drivers' strike by filling "jerry cans".

Conservative jitters were enhanced by a ComRes poll in The Independent which gave Labour a ten-point lead, its highest for seven years, and found that 66 per cent of people view the Tories as "the party of the rich".

"They think we're toffs," David Davis, the former shadow Home Secretary, told BBC Radio 4 yesterday. "They look at the front bench, see them all very well dressed, well turned out, well fed, and perhaps feel that they're in a different world to them. That's why the 'we're all in this together' phrase is very important – but at the moment it's not working."

Mr Cameron started 2012 with a purple patch, while Ed Miliband came in for growing criticism within Labour. Now, suddenly, the roles have reversed.

Mr Cameron remains cool under fire, dismissing recent events as a bout of the mid-term blues that afflict all governments and are inevitable in an age of austerity. He calms his nerves by reminding himself that he is less than two years into a five-year administration. But he will want to "get a grip" – and soon.