With this lot as neighbours no wonder Prince Harry let rip down at the pub

Good food, hunting, wife-swapping... they know how to have fun in England's poshest shire. And there's drinking, too, as one young royal has discovered. A look at the highs and lows of county life.

The thought of Prince Harry, third in line to the throne, drinking himself into a stupor in a shed at the back of a pub, shouting xenophobic remarks at French chefs, and smoking weed in the back garden of Highgrove has taken the shine off a certain idea of Gloucestershire.

Since the Prince of Wales established himself outside Tetbury and made the county smart, it has been the subject of a lot of tweedy myth-making by youngish men from The Daily Telegraph. When Madonna was house-hunting in the area, for instance, she was humorously advised to drive an ancient Land Rover, wear black wellingtons and a Barbour, take up hunting and go to dinner with titled people known to friends as "Bunter". P G Wodehouse did it better, and his world was just as imaginary.

True, there are a lot of grand people in the county, and quite a lot of rich people. "Royal Gloucestershire" includes not only Highgrove, but the Princess Royal's 760-acre spread at Gatcombe Park, near Minchinhampton, Prince Michael's place outside Stroud, and the ancestral homes of people such as the Duke of Beaufort at Badminton and Lord Apsley at Cirencester. Then there is Lord Vestey, the meat merchant, who has 4,000 acres at Stowell Park. It goes on. Prince Charles may live behind a 9ft wall, and Princess Anne at the end of a long drive, but these people have a presence locally. Charles, for instance, swallowed his tongue while playing polo at Cirencester. His niece Zara turned over her Range Rover at Bourton-in-the-Water when she was just 19.

They like to have fun. The most obvious example of this is hunting. It's hard to overestimate how keen the mystical prince, his sons and his paramour are on ridding the country of four-legged vermin. Luckily, David Somerset's 300-year-old Badminton estate, and his Beaufort Hunt, are keen to accommodate them, to the extent – if revelations last week are to be believed – of encouraging the vermin to breed so as to make exterminating them more interesting. The Prince says he came to Gloucestershire for duty. Highgrove, bought from Harold Macmillan's son Maurice in 1980, was conveniently placed for both London and the Duchy of Cornwall. But it was also convenient for Chippenham, where Charles was friendly with a young married couple, particularly the wife. Mrs Parker Bowles was keen on hunting, too.

Gloucestershire has always been handily placed. The West Saxon kings ruled from Pucklechurch, in south Gloucestershire, now famous as the place where Stephen Fry was incarcerated as a young offender. In the Civil War, the county's strategic importance was underlined by Charles I's siege of Gloucester. When that failed, he was effectively finished. Before that, though, he had found a little spot near Painswick, and called it Paradise. He was right. The best part of Gloucestershire is there for the taking. It's the countryside: you don't even have to own it.

The most remarkable hills are in the star-shaped group of five valleys centred on Stroud. Laurie Lee, who lived in a hamlet called Slad, rhapsodised about their quaint poverty in prose that sounds as if it had been translated, literally, from the Spanish. Physically the valleys have changed little, though the poverty is perhaps less attractive than it was.

Creamy-grey Cotswold stone houses, plain and strong, cling to the walls of these valleys. But down on the valley floor there's a mess of nasty, noisy, smelly industrial sheds, the residue of 200 years of decline. When the rest of the country started its industrial revolution, Stroud was already washed up. With hills for sheep farming, and plenty of fast-running water, the area produced cloth, notably the Stroudwater Scarlet worn by the British redcoats as late as the Zulu wars. Today those mills are sofa warehouses and double-glazing workshops. No one hunts in Stroud, although Joanna Trollope, our local author (Jilly Cooper is our local "writer"), once claimed that teachers had seen children scavenging from bins. She talked about "children going to school without underwear". This was not idle chatter – she is socially involved – but writers do make their homes suit their own purposes. Aghast at her Classic FM image, Trollope stresses Gloucestershire's gritty side. But she went too far when she compared tiny Aston Magna with Manchester's Moss Side.

Jilly Cooper, who lives in Bisley, has her own vision of Gloucestershire, in which the people with no knickers are adults. She considers that wife-swapping, rather than nine-pin skittles or sloping-pitch rugby, is the Gloucestershire sport. It is true that a swingers' publication recommended the lay-by at Birdlip, with its view of the Malvern Hills, for in-car entertainment. Sedate Cheltenham, with smart shops, a Raymond Blanc restaurant, theatre and infant university, was shocked last year to find it had a brothel, above a double-glazing shop.

When the news about Fred West emerged, however, people were horrified, but less surprised. Gloucester is the only urban place in the county, with a drifting and cosmopolitan population. It wouldn't have happened in Tetbury, which is like one of Prince Charles's watercolours of an ideal town.

Any ugliness is elsewhere. Gloucestershire is a county of Potemkin villages, where the conservation zone homes are inhabited by professionals working at GCHQ, or for the health service, and all the old villagers, with the funny accents bequeathed to them by the West Saxons, are in nasty council estates out of sight. There is an undercurrent of class hostility. People who live on the council estates of a town such as Cheltenham get literally nothing from its literary festival, classical music and posh ladies' outfitters. No wonder their offspring congregate in the high street, drinking themselves into a stupor (though not publicly – Gloucestershire's towns have a ban on open-air drinking), smoking weed and perhaps shouting xenophobic remarks.

Perhaps Prince Harry should be applauded for trying life as many of Gloucestershire's other teenagers know it.