Yasmin Alibhai-Brown: Men are just as much victims of the workplace

My husband has moved to working four days a week this year, and it has changed our lives

I met Fiona Phillips at a dinner last week. The bright and skilled presenter on GMTV's breakfast programme is demob happy. After 12 years, she is quitting her lucrative and influential job. She wants, she said to me, more humane working hours, real time with her kids and to care for her elderly father.

Her decision was announced just as the nation was presented with the latest Equality and Human Rights Commission (EHRC) report, Sex and Power. The news was abysmal. In the drive up to the top, talented women had slowed down, stopped or gone into reverse. The old banger stalls more than it moves.

I hope Jeremy Paxman read the report avidly. It might assuage his paranoia that white men are now so washed up that they will perish, pining and untended on our shores. White men still rule, whether good, brutally bad, plain ugly or manifestly mediocre. And many wilfully and demonstrably discriminate against women and non-white aspirants, although of course they deny it. They would wouldn't they?

The EHRC seriously understates this active exclusion by powerful men. The chairman, Trevor Phillips, and the chief executive, the former civil servant Nicola Brewer, are inclined to favour the consensual model, to put the powerful at ease. Eschewing legal instruments and straight talk, the commission prefers careful words and to offer carrots. We wait and see if this strategy will deliver. I am sceptical.

The report, however, is right to conclude that gender prejudice is not the only cause of the differentials at the top of British society. Other important factors may be responsible. Ms Phillips, like many other successful women, decided to use more wisely her coming years, not to squander precious time by giving over such a disproportionate amount to her brilliant career.

Not many men could give such reasons for leaving top jobs. Our culture and their own masculinity does not allow it. A tiny percentage do but soon return to their natural place in the order of things. Think of Alan Millburn, who left his job as health secretary in 2003, ostensibly because he wanted to see his kids growing up. Norman Fowler went off for the same romantic reasons in 1990. Both were back prowling the corridors of power within two years. Either the kids of the leading classes grow up unnaturally fast or they couldn't hack it. Or, more poignantly, they do not have the same entitlements as women when it comes to work-life balance. They are victims of another kind of emerging inequality.

Earlier this year I spoke at diversity conferences in the City. We discussed the pressures of working long hours and the demands of children and ageing parents. Employers do try and facilitate their needs, up to a point. It isn't enough, and the sense of personal crisis is palpable.

Most carers of parents are female even in this group of supermen and superwomen, partly because that is their given role. One man said: "I can't tell my boss my mother is suffering from dementia and I have to see to her. It would be professional suicide for a guy to confess to that."

Within the family there are other pressures to conform. Most men in work do not feel they can abandon decently paid jobs to start up risky, small businesses and better balance their lives – the route taken increasingly by talented women. My husband has moved to working four days a week this year, and changed our lives hugely for the better. He feels liberated, but I feel a hum of worry. His regular job has enabled me to opt for a dilettante career. In that sense I have more rights than he has. You do find that behind many a creative and self-driven woman, there is a dependable bloke who steadies the ship.

In the end though, we have to blame white men for this sad status quo. Who made the rules? Who is responsible for the manic workplace where child-bearing or child-rearing women are kept down or out, where there is an institutional expectation that money buys a person night and day? Blokes like Alan Sugar and Gordon Brown did.

I sometimes wonder how much the PM sees of his sick, young boy. The current unremitting pressure on him must affect his family duties. Women can at least escape; men are stuck in the mud, flailing.

A downturn in the economy may correct, moderate our misshapen work culture. At the same time it may be too late for that. When women no longer want positions held by men, you know the system is irredeemably rotten. The glass ceiling will fall because the edifice is unsupported and unsustainable. Soon I hope. Then we can construct a clement world of work, and full of possibilities, for all of us.

y.alibhaibrown@independent.co.uk