Howard Jacobson: So much more could have been done to liberate people from the confines of class

A funny thing happens to working classes when they read – they stop being working class

Not often I say this, but I'm feeling sorry for the BBC. Specifically for BBC2 whose series White – Is White Working Class Britain Becoming Invisible has drawn criticism from people of all colours, classes and visibility.

First the programme-makers were accused of giving whites a platform from which to spew racist hate, then they were accused of patronising whites, and now a public letter signed by senior educationalists claims they've missed an opportunity to "interrogate the global, national, regional and local economic relations and policy priorities that have created social immobility, joblessness and disenfranchisement for both white and minority ethnic working-class people" – which, to my ear, is just a less snappy (not to say less literate) way of asking Is White Working Class Britain Becoming Invisible. You can't please all of the people all of the time, but when it comes to class and colour you can't please any of them ever.

I have avoided most of the series so far because I don't like how thinking about the white working classes makes me feel. Is that because I was once a member of that class myself, and suffer the conflicting pulls of loyalty and shame, with a bit of guilt for leaving the sinking ship thrown in? No, is the short answer to that. No, is the long answer too. My great-grandparents came from Lithuania and Ukraine carrying their clothes in bundles on their back. The British white working classes were not particularly pleased to see us. Others who shared our ethnic origins but were already resident in this country weren't particularly pleased to see us either. The old problem of uncouth cousins turning up and embarrassing you in front of your new friends.

Immigration is a complex business. And gets more complex still when it comes to class. We were white, we were poor, we worked; but that still didn't make us white working class. We didn't look that way to the white working classes. And we didn't look that way to ourselves. It takes more than circumstance, income and colour to define your class.

And yet my father called himself a working man until he died. He voted Labour all his life. Many of his friends were communists and trade unionists. We took the Daily Mirror. We knew no Conservatives. People we thought of as middle class by virtue of their social ease drifted in and out of our lives but we were never comfortable with them. Fear of being looked down on was part of it. I go in fear of that still, to the point of wondering, sometimes, whether I am looked down on by myself.

So what were we? I didn't know then and don't know now. We missed out on being middle class because we didn't have the confidence of belonging. And we escaped being working class – I choose my words carefully: to me it felt like an escape – because its culture, or what we understood of its culture, was alien to us. And here we enter that dangerous terrain of condescension and caricature. If I say we were not drinkers and didn't like going to pubs, someone will remind me that not all white working class people drink. Many were Methodists or the like. But then there's another difference: we weren't Methodists or the like. Not that I knew what Methodism was, but it seemed entrenched, a system for keeping you as you were until life got better for you in another world. We wanted life to get better for us in this world.

If the spirit was not a manufactured solace for us, neither was the body. I saw several grown men fighting in the streets as I was growing up, always white working-class men. Their wives and children cheered them on. I couldn't understand why the spectacle didn't make them weep. We didn't do gladiatorial. Football, for example – at least in our house – mattered not a jot. Though my father did like to watch wrestling on television, which as far as I was concerned very nearly made him working class after all.

And he wasn't a reader – another mark against him, class-wise. I am not, of course, saying that reading is an un-working-class activity, but a funny thing happens to working classes when they do read – they stop being working class. Not stop working, but stop being conscious of the limits of class. Isn't that the way it's meant to be? Isn't the proletariat meant, at the last, to do away with itself, enjoy a transfiguration, not into the hated middle class – though I happen to think there are worse places to go – but into something else, unconfined, liberated, unamenable to class description?

Some illogical sentimentality attaches us to a culture whose virtue could only ever have been one of making the best of things. It isn't fun to be poor. It isn't fun to work in a factory. It isn't fun to be denied what others have. Three cheers for the hard-won vitality of working-class life. But four cheers for its emancipation from itself. An emancipation which leftist educational theory has done nothing but hamper for the past half-century or more. Among deeds of intellectual wickedness, this one ranks high indeed: refusing, on ideological grounds, deliverance to those made slaves to class. Denying them better, because better is an unacceptable concept.

In its own terms, this was the subject of the one programme in BBCs White season I have watched. White Girl. An excellently written, superbly acted and directed drama, about an 11- year-old white working-class girl finding in her Muslim neighbours what her own blasted family cannot give her. Some have accused it of idealising Muslim culture at the cost of stereotyping the white working poor. None of it true. We saw Muslims as the distraught girl saw them – sometimes literally so, the camera at her eye level – her seeing inflamed by the emotional deprivations she was suffering at home. Home? Ha!

This was disinheritance at every level, and we do the white working class no favours by refuting this depiction of them. The drink, the drugs, the violence and lovelessness, the prejudice, the aggressive illiteracy. Sure, it's not the whole story. Nothing's the whole story. But something has been annihilated out there – I don't mean the class, to hell with the class – I mean the humanity. The White Girl won back her humanity by looking around her. The first stage in any cultural escape: see what else there is. It augurs well for the future. The older generations cling on to what they hate and gives them nothing and call it loyalty. The young might break the cycle yet.