How did Britain's prison system get so bad - and who is to blame?

The Big Question: The PM has decided prison reform is one of the "great progressive questions" of our time. But experts say his own policies have made the prisons system the worst they have ever seen it

Prisoners waving out their cell window in Brixton prison in South London (Rex)

Why are we asking this now?

David Cameron has set out his plans for a "truly 21st Century" prisons system by vowing to make prison reform one of "great progressive causes" that will mean prisoners are not simply regarded as "liabilities to be managed, but instead as potential assets to be harnessed". He reckons that the failure to rehabilitate prisoners costs £13bn per year because of "scandalous" rates of reoffending.

He said that the "current levels of prison violence, drug-taking and self-harm should shame us all".

Why this sudden interest in rehabilitation from the man who appointed Chris Grayling?

Frances Crook, chief executive of the Howards League for Penal Reform, welcomed the "brilliant rhetoric" of the Prime Minister's speech - but said Mr Cameron was making an implicit admission that the policies of Chris Grayling, who was Justice Secretary between 2012 and last year's general election, had brought the prison system to its knees.

Ms Crook, who has headed the Howard League since 1986, said: "I have been around a long time, and I don't think I have ever seen prisons in such a state ... [in terms of] overcrowding, murders, suicides, prisoner self-harm and assaults on staff."

The problem, Ms Crook says, is chronic overcrowding and understaffing of prisons, brought about by Mr Grayling's closure of 18 prisons and prison wings and cuts to the number of prison workers by around 40 per cent. A new, punitive, disciplinary regime imposed from Whitehall had the effect of riling already "bored, angry and frustrated" prisoners who had been locked in their cells for long periods.

As Ian Dunt of Politics.co.uk points out, the 2015 report of the Chief Inspector of Prisons demonstrates the disastrous consequences of overcrowding.

"You were more likely to die in prison than five years ago ... More prisoners were murdered, killed themselves, self-harmed and were victims of assaults than five years ago."

Cameron, it seems, has decided to heed these grim facts, to listen to Mr Grayling's successor, Michael Gove - who has stressed rehabilitation and reversed many of Mr Grayling's worst policies - by changing course.

Liberated by his promise not to contest another election - with an eye on his legacy and his pledge to make his time in office a "turnaround decade" for the country - Mr Cameron seems to have decided to ignore the tabloid concerns about "soft justice" and go with what works.

Isn't there a more cynical party political reason for backing Gove?

Well, yes, there is that.

What is the Prime Minister actually proposing?

• Six new ‘reform prisons’ to be created this year, with full autonomy over how they operate and spend their budget - modelled on Mr Gove's beloved free schools and academies - extended to all prisons within five years.
• The prison education system will be "completely transformed" with full control being given to ‘reform prison’ governors, while protecting its £130m budget - effectively ending Mr Grayling's puntitive, centrally imposed disciplinary regime.
• Intitatives to try to reduce the overall prison population by extended use of satellite tagging and community punishments. As the Daily Mail notes, some prisoners could be let out during the week, to return to their cells at weekends.
• New league tables of prisons, showing which are the best and worst at rehabilitation.
• Nine new prisons will be built at a cost of £1.3bn with a plan to sell off some of the old Victorian prisons which are especially difficult to refurbish.
• A Teach First-style scheme to recruit high-quality graduates to work in the prison service, run as a social enterprise by Lib Dem former minister, David Laws.
• Prisoners could be banned from drinking on their release, enforced by GPS tags that indicate whether they have consumed alcohol.

Will it work?

Frances Crook says that the "£50,000 question" is whether Mr Cameron is prepared to send fewer people to prison, and spend more on staffing. The current prison population is over 85,000; England and Wales has the highest imprisonment rate in Western Europe.

"We need to take the power away from magistrates to send prisoners to prison for a few weeks at a time, for things such as begging," said Ms Crook. "These people are often homeless, they are often drunk, they are a nuisance, they are not a danger to society. Prison is not going to stop them from begging."

On funding, she told The Independent's political editor Oliver Wright: "It is all very well giving prison governors autonomy to improve rehabilitation and education. But you are not autonomous if your prison is so full you can't let prisoners out of their cells. It is hard to have autonomy over an institution that is stuck in the 18th Century."

Prisons with excellent facilites such as libraries are not able to let prisoners use them properly because there are not enough prison guards to escort them there, she added.

Has Michael Gove turned the PM into a bleeding heart liberal?

As John Rentoul demonstrates here, Mr Gove insists there is a clear, Conservative (and conservative) case for prison reform and rehabilitation.