D J Taylor: Congratulations, Sir Cliff: yours is the soundtrack of Middle England

He has endured decades of being patronised, as has a large section of the nation. But, as his latest revelations suggest, he is a survivor. His is the very story of post-war Britain

Sir Cliff Richard's publicists must have had their work cut out over the past few days. At the start of the week, the gossip columns were aflame with news of the reissue of his cover version of a song entitled "Honky Tonk Angels", whose alleged subject-matter – prostitution – might conceivably "tarnish his reputation". Then came the disclosure, blasted all over the front page of the Daily Express, that the late Queen Mother's troubadour-by-appointment has spent the past few years sharing his house with a male "companion", a former Roman Catholic priest named John McElynn.

Throwing open the closet door to its furthest extent, but still not quite manoeuvring himself over the lintel, Sir Cliff declared that he and McElynn had struck up a "close friendship", which wouldn't necessarily prohibit a future marriage: "I may suddenly meet someone and feel differently, but right now I am not sure that marriage would enhance my happiness."

There was also a suggestion that the Church should "come round" to the idea of same-sex marriages, and some hand-wringing over the media's interest in his sexuality. "What business is it of anyone else's what any of us are as individuals?" he demanded.

All this, cherry-picked from a newly published autobiography, My Life, My Way, might look like the usual souped-up yellow-press salacity. In fact, as Sir Cliff – seasoned handler of the media as he is – no doubt realises, it is a victory of a sort: an example of the advantages, in a cruel and capricious business, of sticking to your guns. The most successful UK recording artist of all time? The only performer, along with Elvis, to register hits across six decades? The man whose records spent more time on the UK charts in the 1960s than the Beatles and the Stones combined? The first pop star to receive a knighthood? In all four categories, unbelievably enough, the answer is: Sir Cliff.

Set in motion as long ago as 1958, when the then 17-year-old Harry Webb and his band, The Drifters, scooped a contract with EMI and a residency on the new ITV pop vehicle Oh Boy!, the Richard cavalcade – Roman Catholic priests, cancelled engagements to Sue Barker, podium-sharing with Billy Graham – is more than a uniquely successful pop career, periodically extended by astuteness and an eye for the main chance.

Seen in the round, it is nothing less than a soundtrack to the last half-century of British history. Middle English history, that is: a fraught, uneasy time for the most part, when the majority of Middle Englanders could expect to be patronised or ignored by those above them on the social pyramid, but whose uncertainties would give way to eventual triumph. All of which, mutatis mutandis, might be said of Sir Cliff himself.

Anyone who grew up in the 1960s will remember Cliff: "Summer Holiday", "The Young Ones" and "Congratulations" blaring out from Family Favourites on the radio; that amiable chipmunk's face gazing out from your sister's pop mag to explain why he drank milk ("I like it. It's cool," our man sternly rationalised').

Even my father, a deep-dyed reactionary who considered rock'n'roll "the music of the sewer" and held Elvis responsible for the decline of Western civilisation, liked him on grounds of "wholesomeness" and singing songs you could whistle. All this, it turned out, was a mark of – well, it would not quite be accurate to call it Cliff's protean qualities, for the essentials of the package have stayed fairly constant – his ability to constantly re-invent himself, to maintain the popularity of the brand while moving purposefully off in whichever direction the public taste happened to be straying without getting swamped by the tide.

This talent for periodic redefinition is all the more startling when you consider the kind of figure Cliff cut at the time of his Macmillan-era launch. A grateful John Lennon maintained that "Move it", the 1958 debut single, was the first authentic British rock record. Columnists from the "must we fling this filth at our kids?" stable professed horror at the gyrations of his live outings. "His violent hip-swinging was revolting," wrote an anxious lady reviewer in the New Musical Express, after catching an early instalment of Oh Boy!, "hardly the performance any parent would wish her children to see." There were more complaints about the amount of eye-liner on display, while the Daily Sketch summed up the debate by demanding: "Is this boy too sexy for television?"

Ah well, Kingsley Amis (to take another representative figure from the time) was an angry young man once. However improbably sexy or implausibly mascaraed, Cliff was also a sharp enough operator to see which way the commercial wind was blowing and to appreciate that long-term careers were likely to be won on the back of mainstream "entertainment" rather than minority interest rock and roll.

At an unseasonably early age he was ready to line himself up as the standard-bearer (or casualty, depending on your point of view) of what George Melly, borrowing the phrase from a poem by Thom Gunn, christened "The revolt into style", that time-honoured and well-nigh inexorable process whereby the Biz fastens on anything remotely left-field or challenging and sanitises it for mass-market taste. Even by 1959, still a year in sight of his 20th birthday, Cliff was mellowing his tone. There had been a couple of number ones by this stage, not to mention some film work, and the sexual calisthenics of the live shown had been radically down-graded ("His erotic twitching turned into a bent-kneed shuffle," Melly reported, "not so much a sexual courtship dance as a suggestion that he'd wet himself.")

With sanitisation came the old family entertainment-style polish, which as the social historian, Dominic Sandbrook, notes, connected him to the older world of the inter-war variety halls (this was precisely why my father liked him: he could see him sharing a bill with Max Miller and Henry Hall.)

Banished from the equation, on the other hand, was any kind of credibility. Moving on from Family Favourites to the music magazines of the mid-1970s, I quickly discovered that no home-grown pop star was a riper candidate for satire than Cliff. He'd been on the Eurovision Song Contest. He was – ye Gods! – a Christian.

His music was as edgy as a vat of warm water. My 1975 copy of the New Musical Express Book of Rock summarises the charge sheet: "Steady flow of bad movies and records aimed at mass audience have smothered any early promise in a sea of corn," the Bible of any self-respecting Seventies pop kid telegraphically (and haughtily) pronounced.

Meanwhile, Cliff was re-inventing himself again, in line with the popular taste, gesturing at disco, nodding at the post-punk interest in social realism with vistas of bedsitter angst ("Carrie doesn't live here any more/ Carrie had a room on the second floor/ Carrie had a date with the wrong kind of guy" etc).

Come the Eighties it was saccharine (and bestselling) Christmas singles about mistletoe and wine, and furious rows with playlist compilers, in which the only guarantee of a record being heard on the radio was to issue it pseudonymously. All this illustrates an odd paradox in British cultural history – British history generally – in that its "accepted version" usually turns out to reflect the views and assumptions of tiny minorities.

The Sixties, most of the evidence insists, was not the decade of the Beatles and Mick and Keef, it was the decade of Cliff and the era's best-selling album, The Sound of Music. Exactly the same point could be made of the late Seventies, when the Sex Pistols (minority interest) got the cultural analysis and the Bee Gees sold the records.

Nowhere, it could be argued, has liberal orthodoxy made more of a fool of itself than in its presentation of Britain's last half-century, where the Sixties are seen as a period of mouth-watering social experiment, full of limitless opportunities for modish young people, and the Eighties as a kind of new ice age tyrannised by Mrs Thatcher and her sidekicks.

The average (and numerically superior) Middle Englander – that vast proportion of the populace whose very existence is a matter of pious regret to the average Guardian columnist – was, on the other hand, profoundly alarmed by the Sixties, terrified by the Seventies, which revealed the disorganisation of "organised" Labour, and, come the Eighties, prepared to greet Mrs Thatcher as a moral saviour, one of "themselves" rather than a grouse moor patrician or a TUC stooge.

The contempt in which Sir Cliff has nearly always been held by the nation's opinion formers stems largely from the fact that he represents a majority view, something which most of these commentators on our public life nearly always prefer to sweep under the carpet.

At the same time, no amount of live-in companions or paeans to honky tonk angels will ever tarnish the Cliff Richard brand, which is now so indissolubly intertwined with the constituency that sustains him as to be more or less inviolate.

Rather like Middle England itself, Sir Cliff remains – despite all the brickbats – a curiously potent and successful force whom no amount of routine disparagement or incidental snobbery can ever quite anaesthetise or subdue.