News of the wold: What's the small Cotswold town of Chipping Norton really like to live in?

Oliver Bennett heads to Farrow & Ball country to find out

Chipping Norton is the kind of town that makes you go "hmm". It's pretty but not cloyingly so. It has a great bookshop and café combo in Jaffé & Neale, a high street with a hardware store called Harpers, a Dorothy Perkins, and a friendly mixture of shops that provide the backdrop to a pleasing atmosphere of small-town hustle and bustle.

But the epicentre of a pre-eminent global elite to match LA's Beverley Hills, Paris' 16th arrondissement and London's Belgravia? No way.

So it's peculiar that part of the fallout from the 'Murdochalypse' is the notion of the "Chipping Norton set", wherein David Cameron rides with Rebekah Brooks, before dining with Jeremy Clarkson, Elisabeth Murdoch and Matthew Freud.

This week I went to see why this Oxfordshire market town has become a power nexus. It's not immediately apparent. Those au fait with the area call it "Chippy", and as I drive into town there's a boarded up pub called The Off the Beaten Track and a fast-food shop, called Chippy. But of course. And what does its owner, Michael Richards, think of the Chipping Norton set? "Get out of here," he jokes. "It's all a load of hype."

So it's not a notion that delights everyone, and ex-Goodie and Chippy resident Graeme Garden recently decried the "Chipping Norton set" on stage at a fête. But if there's any idea that Chippy residents are chippy, it does not register today. "The fuss is putting Chippy on the map," smiles a local cabbie. "We're enjoying it."

I walk past the quaint theatre and down to one of Chippy's architectural highlights: a row of 17th-century almshouses in a picturesque row leading down to the fine church – testament to Chippy's past as a prosperous hub for the wool and tweed trade. There are some great houses here in the limestone Cotswolds vernacular, but Chippy is not twee.

As the historian and new Chipping Norton resident Dominic Sandbrook recently wrote: "Of all the popular Cotswold towns, it is easily the most unpretentious. You can find a period cottage for £150,000 and £250,000 here, which would be impossible elsewhere in the glorious Cotswolds."

I meet local lad Jonathan Bramwell of property hunters The Buying Solution at Jaffé & Neale. "Chippy's not as touristy as Stow-on-the-Wold, Woodstock or Burford," he says.

The latter, a Cotswold pearl full of kitchen shops and Japanese tourists, is where Matthew Freud and Elisabeth Murdoch live, a power marriage to match Henry VIII and Catherine of Aragon, complete with local courtiers – Cameron's head of strategy, Steve Hilton, lives nearby in a barn conversion.

Even so, Bramwell, who grew up in nearby Churchill – where Brooks lives – says Chippy has slowly become glossier.

"In the past five to 10 years, it has acquired a bit of a London finish," he says. He means décor shops. Farrow & Ball paint. Labradors. Hunter wellies. Cupcakes and bunting. All these are there.

But Chipping Norton remains a working town. "It's not the most glamorous place in the evening," admits Bramwell. "Come after dark and there's a kebab van here."

Incomers tend to prefer the surrounding villages to Chipping Norton, but there's a mature, arty crowd in the town. The local estate agent Toby Harris, of Strutt & Parker in Moreton, calls them the "arty ladies with hair"; bohemian femmes with bicycles, and period cottages crammed with pottery. Nothing wrong with that.

Bramwell says the old Parker Knoll factory has been turned into housing, possibly to include a (contested) Sainsbury's supermarket, and that the town is expanding. But he agrees that the Chipping Norton set and their ilk don't themselves live in Chippy. "The rich second-homers want the country, not market towns," he says.

These mid-life power movers are augmented by the hedge funders moving westwards within London – this is an area that co-ordinates to Notting Hill, not Islington – as well as the commuter train from nearby Charlbury and proximity to Oxford.

There are a handful of £1m houses in the town itself, although as Harris says: "Chippy is a stepping stone; a bargain compared to the countryside around, quite middle class, a really good comprehensive and old-fashioned shops, like the butcher Trev Beadle."

I go up to Georgian House Antiques, a delightful, 17th-century treasure trove. The owner, George Wissinger, doesn't know much about the fabled "set", but as far as he's concerned, bring them on. "The attention is great," says Wissinger, who laments that the traditional tourism market – Americans – have deserted the Cotswolds, to be replaced by British ramblers, Germans and Dutch visitors, none of whom are great antiques buyers.

Indeed, Wissinger and his friend Roy Burton, a retired local, aren't against the Chippy set as such. They particularly admire Clarkson, who lives nearby, and who apparently hosted the party where Brooks met her second husband, old Etonian Charlie, the racehorse trainer-turned-thriller writer. "Clarkson's good to the town," says Roy. "He gives lots of money for various things like the swimming pool via auctions of promises, and he uses the shops and talks to everyone."

The pair have seen the Prime Minister around a lot too. "He uses the supermarket, and I saw him in the charity shop," says Burton, adding that he too "speaks to everybody".

I leave Chippy and drive out to the surrounding villages with Harris. As we drive, the Farrow & Ball index creeps up. There's the amazing Bliss Mill, an old woollen mill like a castle and a bolthole for international seniors, where three-bed flats are half a million pounds, then we arrive in Churchill – a pretty, honey-stone village with a handsome church. This is where the Brookses attend to their privacy in a luxury barn conversion, and it was here that Cameron was a Christmas guest along with James Murdoch. Imagine my delight when I see that there is a Hackers Lane in Churchill.

Harris drives on, through ruralscenery that is pretty rather than jaw-droppingly awesome, and we arrive in the next village, Kingham. This is the territory of renaissance man and farmer Alex James, and the place runs on organic cheese. Food, festivals (the Cornbury and Alex's own) and a primary school has made it attractive to yummy mummies, according to Harris, who shows me a cottage for £320,000 – several thousand more than a similar property would be in Chippy itself.

We stop at The Kingham Plough, the discreetly boutique'd gastropub, run by Bramwell's sister in law and Heston Blumenthal protege, Emily Watkins, and look around.

A sign on the green says that Country Life magazine voted Kingham the finest village in England, while a memorial to the fallen men of the village from the two wars reads: "This quaint corner of England". As I stand, welling up, one of Harris' arty ladies cycles by on an upright bike.

Nearby is Daylesford, home of the famous organic farm and shop. We're starting to get deep into the Cotswolds now, and the red trouser crowd is more in evidence in the well-stocked car park.

"It's quite hunting, shooting and fishing out here," confirms Harris, and we drive back to gritty Chippy, where houses are counted in the tens of thousands.

Chippy will survive the fuss, as it seems a sanguine kind of place with deeper concerns than flash-in-the-pan celebhood. Indeed, last week, there were a lot of photographers in Chipping Norton. Were they looking for the Chipping Norton set?

"Not at all," says a man in Jaffé & Neale. "It was a rare oriental turtle dove. We couldn't move for twitchers."

The Chipping Norton Set

* Rebekah and Charlie Brooks, live near Churchill

* Matthew Freud and Elizabeth Murdoch live in Burford

* David Cameron's constituency home is in the hamlet of Dean

* Steve Hilton, the PM's head of strategy live nearby in Burford

* Jeremy Clarkson, whose family home is south of the town

* Charles Dunstone, the Carphone Warehouse boss, also lives near the town

* Alex James, Blur bassist and now cheesemaker, lives near Kingham

* Property developer Tony Gallagher lives in 17th-century Sarsdon Manor