No wonder ministers are exhausted. They're all trying to copy Tony Blair

Alan Milburn, the former secretary of state for Health, was hoist on metaphorical shoulders, beatified and canonised in the space of a few hours last week, merely for saying that he felt "a million times better" now that he had time to spend with his young children. So I should be careful about disagreeing with St Alan in his campaign against Britain's culture of working excessively long hours. Which, of course, I don't. Family-friendly employment practices and the life-work balance are the feel-good jargon of the moment, and everyone is in favour of them, including me. They are like motherhood (for which read fatherhood, as appropriate) and apple pie.

Yet there is something not quite right about the Milburn way to personal fulfilment, which he was honest enough to hint at in his speech on Tuesday. He admitted that, as an MP, he is still well paid; and that the people who really have a work-life imbalance are the working poor.

But there are other problems, too. If he could not combine a "24-hour-a-day career" with "a life", then what was he saying about the poor, work-obsessed saddoes he left behind in the Cabinet, including his good friend the Prime Minister, who has an even younger child? There is one important difference in the basic organisation of the Milburn and Blair households, of course, in that Tony Blair lives above his office, whereas Milburn's partner and two sons, aged seven and 12, live 250 miles from Whitehall. Even so, it cannot be right that the only way to reconcile the job of Health Secretary with what Milburn described as "anything remotely resembling a normal family life" in northern England is to give up the job altogether.

Responsibility for the National Health Service, one of the largest employers in the world, is undoubtedly a heavy burden. But other ministers have big jobs too. They do not even have to be in the Cabinet. Beverley Hughes, who was famous for 15 minutes for the manner in which she failed to make it to the Cabinet, was minister at the Home Office for "citizenship, immigration and counter-terrorism". Her successor, Des Browne, who has not yet been famous even for a quarter of an hour, is no doubt still reading up on his immigration and terrorism brief.

Some of Ms Hughes's defenders suggested that her job was "too big"; that she could hardly be blamed for forgetting a letter from a year ago about one-legged Romanian roofers when she had al-Qa'ida to worry about. However, as I am sure the best management handbooks put it, the secret of any job at or near the top of a large organisation is about three things: delegation, delegation and delegation. The skill of being a good minister must be to ensure you have to deal only with the absolute essentials.

Yet the job of being a minister is almost completely flexible. There is no measurable output, no benchmark of productivity, not even much of a fixed structure beyond the traditions of the House of Commons timetable and the ritual events of the minister's client groups. Even they can sometimes be dispensed with, as Charles Clarke has done today, saying he has better things to do on Easter Sunday than attend the National Union of Teachers' conference.

Because the job is so formless, many ministers compensate by working too hard, in a perfect demonstration of Parkinson's Law that work expands to fill the time available for it. Another consequence of the lack of structure is that ministers are easily captured by the civil service, who provide a ready-made shape to the ministerial day. This is despite the fact that they have all read Gerald Kaufman's classic, How to Be a Minister, which tells them how to maintain good relations with their private offices while asserting control over them.

The most formless job of all is the Prime Minister's. Each holder of the office defines it differently, as charted in Peter Hennessy's The Prime Minister: The office and its holders since 1945. But there has been a trend towards hyperactivism, of which the current holder is the culmination. Of course, Clement Attlee's interest in cricket was partly an act, as was Harold Macmillan's apparently languid lifestyle, but there is no doubt that Blair's commitment to a form of "total premiership" is unprecedented.

The sheer physical energy that he brings to the job is often underestimated, not least because he sometimes looks tired, which is hardly surprising given his schedule, especially the number of foreign trips. Margaret Thatcher certainly increased the expected workload of the office, with her ability to spend long hours at a level of detail not previously within No 10's reach. John Major carried on these workaholic tendencies with rather less rigour, and they have been ratcheted to a new level by Blair.

But there is nothing that says this is the only way to do the prime minister's job, or that of any other minister. Blair does prime ministerial activism rather effectively, although some of his personal excavations at the coalface of public-service delivery only emphasise how difficult it is even for prime ministers to "get stuff moving through the system", as he put it. It cannot be that the only way to succeed in politics, despite the modern media's increasingly voracious appetite, is by being available all waking hours and several sleeping ones.

There is no reason why a prime minister should not revert to the older model, which is to chair meetings of ministers charged with delivery, and to make ministerial appointments - these are still the only formal powers of the prime minister in the British constitution. Equally, there is no reason why a secretary of state for Health, for example, should not delegate nine-tenths of the work associated with that office to the department's five junior ministers. Everyone knows, even if they have not read Alan Clark's Diaries, that there are far too many ministers with not enough to do, and that a bit of job-sharing could benefit everyone.

Every ministerial memoir laments the lack of time to think creatively or at length, especially about long-term issues, while in government, but a confident secretary of state, with a firm grip of his or her department, should surely be able to make time for considered reflection? There were many reasons why Alan Milburn decided to leave the Cabinet, and part of the reason I am annoyed with him is that he was one of the few politicians of real quality at that level who was pushing for reform that was destructive of the NHS as we know it, which I thought was a good thing.

But this issue is not just about him, or even about British politicians generally. We all know, if we are honest, that we can work more effectively if we work shorter hours. So why don't we? Why do the British have the habit of working for longer than we need? Our average working week is 44 hours; in France it is just under 39 hours. Yet France is a richer country, and if we have recently started to narrow the gap it is not because we spend longer at the office.