'On stage I'm just me having a bad day': Nick Cave on 40 years of music and mayhem

What's Nick Cave, pop's Prince of Darkness, doing throwing himself into the crowds in 100F weather in California? And, asks Craig McLean, will he be bringing his newfound sunny disposition to Glastonbury next month?

Cave says: 'I was always taking heroin. But that Berlin amphetamine is the most extraordinary stuff – if you kept taking more, you never slept. So that created some extraordinarily lengthy days and nights at the typewriter' (Steve Schofield)

Britain's greatest living Australian (pace any UK-resident Murdochs, of course) is leading his band in a boiling, surging, argy-bargy-blues set. He sings "Honey Bee (Let's Fly to Mars)" with a perma-frown, his equine nostrils aflare to his ears, his flouncy shirt agape to his nipples.

"I stick my fingers in your biscuit jar and crush all your gingerbread men!" he leers at a woman in the front row with whom he appears to have forged an intense, in-gig relationship. He cranks and saws at a feedbacking guitar. He plays a Korg synthesiser like a collie worries sheep. Suited and booted, the medallion man makes regular forays into the crowd, bodily launching himself on to upstretched arms.

"This song is called 'No Pussy Blues'," the 55-year-old rock'n'roller declares, whipping a hand through his raven-black hair. "It's not about not getting any pussy," Cave clarifies. "It's about not having a pussy." That's all right then.

If it's Friday at the Coachella Valley Music and Arts Festival 2013, it must be a Grinderman show. And if it's Sunday, it must be a Bad Seeds show. For the first time in a four-decade career that has darted in umpteen diverting directions – novels, poetry, screenplays, soundtrack scores, a duet with Kylie Minogue, an album (The Boatman's Call) inspired by one-time lover PJ Harvey, heroin addiction – Nick Cave is playing with both his day-job bands at the same festival.

"I'M THE GRINDERMAN!" hollers the frontman as their set reaches a torrid climax. It's all over bar the kicking – Cave aims a well-shod foot at Warren Ellis, the Dickensian-hairstyled guitarist who seems to have got caught up in bashing a single maraca on a single cymbal. And then, crash, it's all over. "You've been fantastic," says Cave to the rapturous applause of crowd.

It's the morning after the revelational gig the night before. Freshly suited Nick Cave is pulling up a chair on the tiny hotel-room balcony of a resort complex in the Californian desert, a few miles from the Coachella site. He sets about replacing his depleted potassium levels with a tactical banana, and zealously recaffeinating. Room service is on its way with a pot of coffee, but he politely asks his publicist for a Starbucks chaser. Three sugars, please.

"Three sugars?" I query.

"Between you and me," he says with a conspiratorial twinkle. "We don't wanna unduly influence the kids."

From beneath us, the tinkle of fountains and the laughter of children. Above us, the sun glowers, well on its way to a 100F high. All around, holiday-makers and resting festival-goers perambulate in swimwear and vests. It would only be a less Nick Cave-like environment if the prodigiously talented, cheerfully saturnine artist had booked us a mani-pedi and a post-brunch 18-hole round. It's fair to say he's not a fan of this temporary accommodation, his domicile since Coachella's opening weekend seven days ago (each weekend has exactly the same bill). "Golf gulag," is his pithy summation.

Still, with typical Stakhanovite vigour, Cave is here to work, not play. I praise him on the Grinderman show. His recent Nick Cave and the Bad Seeds album, Push the Sky Away, is an atmospheric, spacious, melodic triumph. But that Grinderman gig – that was something else. It was a theatrical tour de force, a transporting, immersive experience – one in which the crowd went willingly into the dark night.

"Yeah, the crowd were great last night. Last Friday they were a little… confused," he smiles, his Australian accent undimmed by 30-plus years exile in London, Berlin, São Paulo and now Brighton. "Last night they seemed prepared. Maybe they were on different drugs or something like that. The first night, there was a kind of terror in the audience that was really exciting. But it was great last night, too."

It was, in all the right ways, so brilliantly stagey that when, at one point, Cave gestured to his sound engineer, it was like an actor breaking the fourth wall and addressing the camera. "Oh yeah: my guitar wasn't loud enough," he grins, which is hard to fathom given the cataclysmic Grinderman racket. "It's always that."

"And," I ask, "what did Warren do to deserve a kick up the arse?"

"Um, I have to move Warren along a little bit. He gets into the moment for… extended periods. And I'm like, come on… So, yeah, I kick him along. I kick him more in Grinderman than in the Bad Seeds."

Are the shows terrifying for Cave, too? He certainly looks like he's lost the plot.

"No, it's not terrifying," he says evenly. "It's a real pleasure to do the Grinderman things. You are transported," he agrees. "You can't even imagine the concert before you go on. And then something happens.

"There's always an element of risk, because it's very confrontational for the audience, and especially at festivals you never know what's gonna happen," adds this dogged, committed performer who, next month, plays Glastonbury for the fourth time. This, he says, is why he "does that thing" of stepping bodily from the stage and into the throng. "It's because it just throws in this… um… you just don't know what's gonna happen."

A sense of jeopardy?

"A sense of jeopardy," he agrees. "Can you have me say that? And I'm not trying to encourage kids to crowd-surf," says this father of four sons (including 12-year-old twins with current partner, model Susie Bick – that's her naked on the cover of Push the Sky Away). "It's more that you can get some kind of mad energy out of the crowd if you're in there for a bit."

The Grinderman airing at Coachella aside, this current American tour is all about the Bad Seeds, the band he formed in London 30 years ago, and their aforementioned 15th studio album. Cave ordinarily opens their set with "Jubilee Street", the six-and-a-half-minute epic at the heart of the album, a gently climaxing, strings'n'guitars, mid-tempo ballad that features a children's choir. Wherever possible he's drafted in local kids to expedite the rapturous denouement.

At Coachella, he's been helped in this regard by Flea. The rubber-limbed Red Hot Chili Peppers bass player co-founded the Silverlake Conservatory of Music in Los Angeles in 2001. Some of his students were only too happy to sing with Cave. But on other stops of a tour that began in Australia – where the album gave Cave his first homeland Number One, 40 years after he formed the band that turned into the legendary Melbourne punk-goth outfit the Birthday Party – sourcing willing kids has been more problematic.

"Sometimes you have to have three or four goes at finding the choir. They come back going, 'No, no, no…' Especially in this country."

Parents, it seems, have been doing their own Criminal Records Bureau-type checks, looking up Auld Nick's ripe back catalogue. They might have alighted on the titles of his 1986 albums Kicking Against the Pricks or Your Funeral… My Trial. But what does he think has been the biggest red flag?

"Well, there's the obvious things – 'Stagger Lee' and stuff," he shrugs, referring to his reboot of the folk song about the killing of a pimp on his 1996 album Murder Ballads. And even when he secured young guest vocalists, "There were certain songs they wouldn't sing. They won't do 'Mermaids', for instance, because of the lyrics… Personally, I find that ridiculous. But maybe I'm more liberal with things," he notes dryly while giving the palm-tree'd middle distance a hard stare. "Anyway, often people feel it's inappropriate, and they don't want their children round words like 'snatch'."

"Mermaids" might be a gorgeous, rippling reverie, but it does begin: "She was a catch… I was the match that would fire up her snatch." This is the song Cave has chosen as the Bad Seeds' next single. Is he hopeful of much radio play?

"Um, I just think we took the front off it," he says bemusedly. "Did we? Something like that. But to be honest, that decision was something that came from elsewhere." Another shrug. "I dunno, we've done what we wanted to do with this record, and it's doing really well," he says, quietly pleased at Push the Sky Away hitting Number One in eight countries. "And people are minded to extend it… to keep it out there. So they're doing things like that. But I have a feeling we just chopped the front off!" he laughs.

As a man who loves words, who crafts every syllable of every word of every lyric, how does he feel about that crude butchering? "In this instance I couldn't give a fuck, to be honest. I've done the song, it's on the record, it doesn't really feel like a compromise to me. It's not like the shame that both I and Warren feel for 'Breathless'," he says. Prior to the release of the 2004 single, the pair were sat down by Daniel Miller, boss of their then-record label Mute, and given a lesson in music-industry realpolitik.

"He said that the radio would play 'Breathless' if we lost the out-of-tune flute that Warren played on the start of the song. And we said, 'OK, take it off.' And both me and Warren have never…" He sighs. "You know, it's those little things where you know you've kinda…"

Caved in?

"You cave in," he nods. It's those moments of weakness, he admits, "that come back to you. They make no fucking difference anyway. Yet they come back to haunt you."

Nick Cave has plenty of experience of the back and forth nature of the creative process. His two filmed screenplays, The Proposition and Lawless, endured varying degrees of rewriting. And over the years he's had several approaches to make a film of his first novel, the brilliantly overwrought and biblical Southern gothic yarn And the Ass Saw the Angel (1989). He wrote the book when he was living in Berlin and was out of his mind on drugs. When I interviewed him in 2004 at his then-office in Hove (a short on-foot walk from his Brighton home, a commute he diligently made every day), he fondly recalled the creative process.

"I was always taking heroin. But that Berlin amphetamine is the most extraordinary stuff – if you kept taking more, you never slept. So that created some extraordinarily lengthy days and nights, days and nights, at the typewriter. The [original] book was 500 pages long, twice as long as the published version. The speed probably had something to do with that."

Now, he says, And the Ass Saw the Angel "could be a film. It's got a story in it, buried underneath all that language. And a lot of people come asking to do it. But they don't have enough money to make it. So we haven't given the rights. It's not a little film."

Conversely his second novel, The Death of Bunny Munro (2009), began life as a script. The story of a sex-mad Brighton-based door-to-door unguents dealer – if you like, an Avon lad with an erection – it was written with Ray Winstone, star of The Proposition, in mind.

"Initially, yeah. The idea came up to write a book about a travelling salesman," Cave elaborates. "And I'd been spending time with Ray during the rehearsals before filming The Proposition. And Ray has this magnetism, where he just sits there and people are drawn to him. Especially women. There's just an excitement around him and he knows [it]. And he's a very funny guy, always pushing the edge of decency, seeing how far he can go," he says approvingly.

But script became novel, and the new, more rabid, less cuddly Bunny was less of a Winstone homage. Nonetheless, when I interviewed Winstone last year he was listening to an audio version of the book, with a view to starring in an adaptation. I ask Cave if that might still happen.

"Yeah, one day," he answers, not altogether convincingly. His hesitation is understandable. His screenplay for The Proposition (2005), a Victorian-period Outback drama, was a great success. Directed by fellow Brighton-dwelling Australian John Hillcoat (The Road), the film starred Winstone and Guy Pearce and was a pungent, sun-blasted, fly-blown gothic Western.

But his second collaboration with Hillcoat and Pearce, 2012's Prohibition-era thriller Lawless (which also starred Shia LaBeouf and Tom Hardy), was a less edifying experience. Cave adapted a fact-based book about Virginian bootlegging brothers the Bondurants, but found his words being watered down by Hollywood committee. Was that ruinous to his psyche?

"Yeah," is his flat reply. "Lawless wised me up to what scriptwriting really is about. Somehow The Proposition, even though there was some rewriting done on that, it just got through without the intent of the movie altering too much. But Lawless really put me off writing scripts."

Never again?

"It's not 'never again'," he says, later adding that he's currently reading George Dawes Green's thriller Ravens, sent to him by film- makers interested in a screenplay adaptation. "But I just wonder what I was doing. I have another job," he smiles. "Oh, look, the bingo," he says perkily, looking down over the balcony at a passing hotel cart toting a box of multi- coloured balls. "That's my night set."

He might not be joking. He has 36 hours to kill before the final of his four Coachella shows, and he's without his family. Last weekend his twin 12-year-olds Arthur and Earl were here to see him at work. They saw both shows, although he professes not to know which version of dad they prefer, the Bad Seeds' frontman or Grinderman's. Still, do they see Nick Cave in full performance flight and wonder who that strange man is? "Nah," he sniffs, "on stage I'm just me having a bad day."

The single 'Mermaids' is out on 20 May. Nick Cave & the Bad Seeds play Glastonbury from 28 to 30 June