Edward St Aubyn: Stripping off and cavorting at New Age retreats, all in the name of research

Best known for the haunting, autobiographical Melrose novels, Edward St Aubyn has a lighter side

Edward St Aubyn's interviews in France, one senses, are a bit different from over here. "I wasn't warned by my publisher," he says with some amusement, "but we went to the Hôtel Crillon, and suddenly I was surrounded by this horseshoe of journalists with microphones and television cameras and radio mikes and so on. I'd only ever seen that in films and generally, you know, it's the head of the LA police department or something, explaining how they have the situation under control. So I felt..." he gives a bark of laughter... "very odd."

His last novel, the scathing, hilarious and universally acclaimed Mother's Milk, recently won the Prix Fémina Etranger, France's most prestigious literary award for foreign literature. Hence the interest. He sighs. "And then I went on the radio. France Culture. So intimidating: whenever I'd listened to it it'd been, you know, Susan Sontag talking about her production of Waiting for Godot while the bombs were falling on Sarajevo, or Malraux talking about Greek tragedy or Borges speaking in flawless French about Beowulf, and I thought, I just can't do this. I didn't sleep for three weeks, I thought they were going to ask me about Heidegger or something. And then finally they said, you know, Monsieur Saint-Aubyn" – his French, like his English, is beautifully enunciated – "Do you think you're in the tradition of PG Wodehouse?" He gives an exultant shout of laughter. "PG Wodehouse! Ha! Ask me about Heidegger!"

It's hard to imagine anything further from the manicured lawns of Wodehouse than St Aubyn's work. In Mother's Milk, for example, his regular protagonist Patrick Melrose spends a succession of joyless summer holidays sunk in a mist of self-hate, stoned on Temazepam, drinking heavily, sleeping with his mistress while his wife's asleep and being generally vicious to all and sundry.

"Yes," St Aubyn continues. "Evelyn Varg was the other one. I said there was peut-être some Evelyn Varg there" – though he later owns up to disliking Waugh, decrying his "emotional poverty" – "but aucun lien avec PG Wodehouse."

He's rather more wary of English interviewers, and who can blame him? When he found himself in the running for the Man Booker Prize last year, the organisers helpfully issued a press release indicating the direction in which subsequent publicity would run. St Aubyn, it said, baldly but accurately, "was raped by his father as a child, abuse which continued until, at the age of eight, he confronted him. At the age of 16, he became a heroin addict and this habit continued at Oxford University. At 28, he contemplated suicide."

The author, who'd already written three novels based around these experiences, appeared nonplussed at being grilled on such formative brutalities by a succession of unembarrassable hacks. Recalling it now, a hunted look crosses his face. "It is a cliché," he says slowly, "but it's true; a novel is something that you kind of catapult over the ramparts and people will make of it what they will. You stay behind. But now you have to catapult yourself over the ramparts and then paraphrase it and... wurggh." He picks miserably at a nearby plant, doing for a moment a good impression of a man being dragged to the guillotine.

Fortunately, the novels we're here to discuss are rather cheerier propositions, and St Aubyn – now a serene-looking 47 – soon brightens up again. On the Edge and A Clue to the Exit (Picador), which he calls his "middle novels", were written between 1993 and 2000, the result of an experimental holiday from the Melrose family's vexed existence. They bounce back into print this month thanks to the success of the last book, and each is guaranteed to surprise.

Readers of Mother's Milk, remembering the ghastly fake shaman Seamus who bilks Patrick of his inheritance in the middle of his "rather awkward mezzo del cammin thing", will find On the Edge an intriguing proposition. Not only is it an entire book about the New Age movement, set at Esalen, the retreat centre at Big Sur in California, it's also a satire with genuine warmth and compassion. "Seamus," says St Aubyn, smiling, "might be regarded as a particularly inadequate representative of – what do shamanic people call it? – non-ordinary reality. Classy phrase."

"I'd finished Some Hope in 1993," he explains, referring to the third Melrose novel, "but I already knew I wanted to go among the seekers." He leans back in his chair, pushing up the sleeves of his sloppy jumper. "I just felt that it was permeating life in a fairly unexamined way – you know, that group therapy and psychobabble and the influx of Eastern religion was becoming a part of the fabric of our culture." He doesn't mention that his mother, Lorna, the model for the Mrs Jellyby-like Eleanor Melrose, was a hardcore Aquarian herself, the author of such resonant titles as Dancing with the Wind and Healing.

St Aubyn himself spent "three, perhaps four very enjoyable years going to Esalen a lot. Very soon I got drawn into being a participant. Anyway, you're not really qualified to say anything about it if you're on the outside: if you don't, you know, get in the hot tub and take your clothes off and dance, then what're you going to write about anyway?"

The full range of his research gets into On the Edge, the most sprightly and cheerful of his books; from Keith, the faux-guru who "studies" baseball on the television while developing his catch-all philosophy of Streamism, to Peter, the strait-laced banker in search of the lost love of his life, who suffers a mystical epiphany after sloshing about in the Esalen hot tubs. It all culminates with a mass climax after the Tantric sex workshop.

"The purely feeble-minded wasn't worth writing about," he says. "I was interested in the border between the mystical and the fraudulent, between the absurd and the generally liberating and revealing.

"But as I was subjected to all these wild claims about the nature of consciousness and the extraordinary global optimism based on that," he continues, "I couldn't help thinking, well, what's the latest scientific and philosophical position in these things? I wanted something much harder and more rigorous." So, in between his trips to Esalen, he began attending conferences. "The Science and Medical Network, Towards a Science of Consciousness, all that stuff. And that all became the subject of the next book, which I wanted to do in a very different way."

The result was A Clue to the Exit, the most cryptic and overlooked of his novels. It follows Charlie, a successful Hollywood scriptwriter (biggest hit: Aliens with a Human Heart, a good description of many of St Aubyn's characters) who learns that a liver condition has left him only six months to live. He goes on a spending spree, living in expensive hotels in Monaco and pursuing a doomed relationship with a gambling-obsessed femme fatale. He also writes a novel, a sort of Platonic talk-piece about consciousness involving a number of characters from St Aubyn's fiction. Simultaneously fascinating and frustrating, it's the most out-there of his books, the only one to employ first-person narration, and probably his favourite.

"I wanted to include a first-person novel and a third-person novel," he says, "because on one level the subject of the book is the inability of science to include consciousness. The language of experience, the first person, and the language of experiment, the third person, can't be translated into one another, so the novel embodies some of those difficulties. I suppose I found it extraordinary that consciousness, which is the only thing we know we have, can't be described by science, which claims to be on the verge of evolving a theory of everything." He laughs. "I find that quite an exciting, an anomalous, thing."

Whatever his misgivings about the interview process, St Aubyn proves a charming conversationalist, incisive and self-deprecating by turns. He tells a story of the completion of Mother's Milk, which he'd originally written with different characters, but "by the time I'd finished I realised I'd just created a simulacrum of the Melroses and just given them different names. So I decided to make it a Melrose novel, which ought to be sublimely simple, I thought, because I just had to change the name of the protagonist, Mark, to Patrick.

"But I'm not very adept with computers, he goes on ruefully. "I was told there was a 'change all' function, so I just typed in Mark and asked it to change to Patrick, pressed, you know, return or whatever, and the next thing I knew, people were going to a superPatrick, and it'd say 'he rePatricked bitterly' or 'you Patrick my words'. And so this word-processing facility turned into a kind of agonised weeding session. I couldn't have chosen a worse name."

Right now, though, he isn't writing at all, and he's hating it. Not that he much enjoys it in general. "No, I don't have any fun," he says with absolute certainty, looking into space. "I just have this sense of obligation, urgency, restlessness, drivenness... extreme tension... and... um... some passages of total despair, some of great excitement. And there are these little... not very often... these little moments of joy, when I feel I've surprised myself, escaped from my own mental habits and seen or said something I could'nt've said or seen if I hadn't submitted to the process of writing a novel. That's the reward. And it's nice when it's over, for about three weeks, then there's the feeling that one should start again. But it's been a long time now, it's been much more than three weeks. I feel very disoriented."

How long exactly?

"Oh, long. A long time. I don't want to name the number of years. But it's all about to end." In fact, he's heading off the following week to a cabin in the woods – "in a secret location" – armed with just a notebook and a pen. I wish him luck.

"Yes," he says with gloomy relish. "I'll either emerge bearded and raving, or a productive author again."