The Panama Papers are damaging George Osborne and Boris Johnson's leadership chances

The Prime Minister's admission that he had offshore investments reminds voters that he is much better off than they are and therefore lives in a different world – one also inhabited by Osborne and Johnson

David Cameron has admitted that he profited from his father's Panama offshore trust

The first time I heard David Cameron speak, in about 2002, I thought there was no way he could be a successful leader of the Conservative Party. I wasn’t prejudiced. Some of my best friends have plummy accents. Some of them went to Eton. I just thought that the great British public wouldn’t put up with him.

I was not saying that the average British voter is prejudiced, either. The average voter is never wrong, so he or she cannot be prejudiced, exactly, but he or she, not paying much attention to politics, might use some cues as shortcuts to judgement.

I realised, too, that a certain level of poshness can be an asset. Tony Blair, when he isn’t doing his Estuary demotic, sounds quite high-born. I remember focus groups in the early days commenting approvingly on how well he spoke, for a Labour person. But I thought Cameron’s accent was so extreme in its 1950s Received Pronunciation that he would be dismissed as hopelessly out of touch before he got to the end of his first sentence.

The whole of Cameron’s premiership has been fought out against that disadvantage. He moderated his accent slightly by 2005, when he became leader, and it has continued to become a little less distinctive since then. But he and George Osborne had to fight the 2010 election against the perception that they were two young privileged men who were playing at politics at a time of national hardship. That is why the word “austerity” has been so damaging to them. And it is why Nadine Dorries’s description of them as “two posh boys who don’t know the price of milk” stuck so fast.

The biggest setback of their first government, the cut in the top rate of income tax, damaged them because it trashed the rhetoric of being “all in it together” and reinforced the image of the Conservatives as the party of the rich. At the time, I wrote that, if Cameron lost the 2015 election, the 2012 Budget would have been when it happened.

That is what makes Cameron’s victory last year all the more remarkable: that he won the grudging votes of people on low incomes who thought he had no idea what their lives were like and yet who still trusted him more than the leader of the people’s party. It is a tribute to Cameron’s skill that he could win with the handbrake of poshness on.

The handbrake is still on, intermittently malfunctioning because, good though they are at politics, Cameron and Osborne really have never had to put something back on the supermarket shelf because they can't afford it, as Dorries also said. Hence last year’s plan to cut working tax credits, postponed (not yet abandoned) in the Autumn Statement. Hence this year’s plan to reduce the increase in disability benefits, abandoned days after the Budget, but not soon enough to keep Iain Duncan Smith on board.

And hence the further damage done by last Sunday’s Panama Papers leak. The damage is hardly Cameron’s fault. He cannot help being rich, and having had a father who was director of an investment trust registered in Panama and operating from the Bahamas. The handling of No 10’s response to the leak wasn’t great. I’m told that part of the problem was that Cameron was upset on behalf of his mother and sought to protect her because every time she looked at the TV, there were pictures of her late husband, who was being presented as part of a dodgy world elite in the same category as Vladimir Putin and sundry Middle Eastern ex-autocrats.

It may have been Henry Kissinger who originally said: “Anything that will have to be admitted in the end should be admitted at once.” Instead, we are at the end of a week when the facts appear to have been dragged out, and Cameron is only now, four years after first promising it, going to publish his tax returns.

But the main damage was done simply by reminding the voters that Cameron is much better off than they are and therefore lives in a different world. I refer the reader to the definitive academic analysis on this point, by Rosie Campbell and Philip Cowley, in 2013, who found that “voters … reacted negatively to financial success”. Given fictional biographies of candidates, they tend to prefer as MPs those with lower incomes.

This is not a problem for Cameron at the next election, as he will heed this week’s calls and Twitter hashtags for his resignation before then, but it is a problem for him in the EU referendum campaign. That campaign was already veering into a contest of the elite versus the rest, and a week of tax-baiting will have sharpened that antagonism.

And it is a problem for Osborne and Boris Johnson, two posh Tories vying for the succession. It is worth noting that, while the Prime Minister will publish his tax returns in the next few days, the Chancellor will not. Osborne also refused to discuss his family’s finances in a TV interview on Wednesday. Johnson published his tax return in 2012 when he ran for re-election as London Mayor, but whoever succeeds Cameron will have to work hard to release the Tory handbrake of poshness.