The dark knight rises: Perhaps Michael Palin isn’t the nicest chap in Britain after all…

Legend has it that when Michael Palin answers his front door to a stranger, he will pretend, if necessary, to be his own brother, in order to escape.

It's not impossible to imagine that some concealed room in his house could actually be occupied by his less congenial twin: a man who, once unobserved, eagerly submits to a comedian's traditional demons – depression, alcoholism, avarice and rage. Nobody – some have suggested, having watched Palin patrol the globe in his familiar uniform of beige chinos and neatly pressed sky-blue shirt – nobody could possibly be, to use his least favourite adjective, quite that nice.

The truth, Michael Palin tells me on the phone, a few days before we meet, is rather less melodramatic. His life, he insists, is largely unremarkable. He's surprised that anybody should want to know more about it, so free has it been of spectacular incident.

"Apart from that guy you shot in Spokane."

"Spokane," Palin laughs, "was an unfortunate misunderstanding. It could have happened to anyone."

None of the livelier recreations Michael Palin has embraced in his comedy – things like cross-dressing, inappropriate use of explosives, group sex with prostitutes, criminal damage and, indeed, homicide – has ever besmirched his own CV. Palin is that most unusual of things: an internationally recognised icon who longs to sound more ordinary than he is.

The 69-year-old, who has just published his second novel, The Truth, lives quietly, as he has for the past four-and-a-half decades, with his wife Helen, in a side-street in north London, close to Hampstead Heath.

He leads the way up to his study, where he makes tea.

If England ever requires one figure to rally all of its citizens, that man, I suggest, might be Michael Palin. He has somehow come to embody our notions of fairness and decency; and these are the two attributes that the English – whether they sympathise with Billy Bragg or Nick Griffin – would most like to claim as their own.

"I don't see why it should be remarkable that you can acquire a reputation for fairness and decency," Palin says. "Those are qualities shared by so many people. And the great majority of people I meet are decent people, just trying to navigate their way through the world without causing too much trouble."

Palin is one of the few men I've met who visibly recoils from a compliment. "Which is a shame, really," I tell him, "because there's some base part of us that prefers to think of all famous people as behaving like demented egomaniacs."

"Fame changes everything. When you're well-known, you're expected to be different. Some people assume you must have a yacht, and four homes. Or that you're famous because you are 'A Decent Man'. Just think of the number of people who do selfless work in this country every day; nobody has even heard of them."

The unaffected modesty that underlies this last remark has been vital to Palin's emergence as a peerless presenter of travel broadcasts. His journey by dhow from Dubai to Bombay, in the third stage of his 1989 series Around the World in 80 Days, remains one of the most inspirational programmes ever made.

He was not the producer's first choice. A gruelling assignment, involving possible shipwreck and certain gastroenteritis, it had been declined by, among others, Alan Whicker. Whicker, Palin's predecessor as doyen of the travel documentary (and a figure mercilessly parodied by Monty Python) explored the world much like a Home Counties bank manager who was vaguely perplexed to encounter indigenous peoples whose costumes would radically contravene the dress code at his local golf club. Palin, whose gentle curiosity places him more in the tradition of the late American broadcaster Studs Terkel, approaches people with unfeigned empathy and two other, seemingly incompatible, qualities: diffidence and courage. The presenter of Pole to Pole, Sahara and New Europe has just completed a new four-part series, Brazil, which will be shown on BBC towards the end of this year.

"Where haven't you been?" I ask him.

He thinks for a moment, then replies: "Middlesbrough."

Wherever you travel, Palin explains, "people are fascinating. You want them to like you and you want to like them. I find myself especially drawn to people who might tend to be sneered at. The thing I remember from Sheffield, when I was growing up, was characters. People who looked at life in a different way. I was always attracted to people like that."

Ever since he was first interviewed, back in the 1960s, Palin has been talking about writing fiction. "I saw novelists as being admirable people and I thought…" Palin hesitates. "…I thought… maybe, one day, I could be one of them."

Some might question whether publishing houses' insatiable appetite for celebrity authors has enhanced the quality of contemporary British fiction, but Michael Palin, like Hugh Laurie, is one of those who really can write. His first novel, Hemingway's Chair, published in 1995, was, among other things, a prescient critique of large corporate retailers, and of their devastating effect on local communities. That book proved much more popular with readers, including this one, than it did with some literary critics. Palin is doomed never to be fashionable as a writer. Time, in his stories, is never likely to go backwards, nor will his narrative voice ever lose its compelling simplicity of style. Which is not to say that he isn't capable of an unforgettable phrase.

"She was a woman of such epic and ineffable unselfconsciousness," he writes of one character in Hemingway's Chair, "that, if born poor and unwelcome, she might well have been considered mad."

The Truth is a beautifully written, elegantly constructed novel which transports the reader from Sumburgh in the Shetland Isles to Bhubaneswar in India. Its hero, a maverick environmentalist called Hamish Melville, observes, at one point, that: "A book of my life, and close scrutiny, are the things I am most eager to avoid." That, I suggest, sounds somehow familiar.

"I am certainly more interested in interviewing than being interviewed," Palin replies. "Sometimes you find yourself attacked from the start."

I have to declare an interest at this point. I've known Michael Palin for some years, through a mutual friend; not exactly as a close confidant, but well enough to have an idea of the man when he is not facing a camera, or speaking for publication.

The first time I had dinner with Palin, I tell him, I remember being quickly aware that there was a little more steel in him than you might imagine from watching his broadcasts. He somehow transmitted a sense that, if you crossed a certain line, he wouldn't hesitate to let you know.

"I have never claimed to be the nicest man in the world," he says. "That's a cliché that has somehow come to be widely accepted. It drives Helen mad. As she and my children [Tom, Will and Rachel, now all grown-up] will testify, I have a short fuse over certain things. Like if the one-inch nails are not where they should be in the box, and they've been moved to the three-inch section. [Pantomime bellow] Who did this?"

Michael Palin is the only man I've ever met who has stormed out of his own house during an interview. He was talking to the journalist Edward Whitley, then an undergraduate, who had not enjoyed Palin's 1982 film The Missionary. (The story of a clergyman who takes his mission to comfort and bring joy to London's prostitutes a little too literally, it stars Palin, Maggie Smith and Trevor Howard.)

"Whitley has described how you flounced out, after confronting him. As I recall, the veins were 'pumping in your forehead'. Your face was pressed against his…"

"There was a bit of poetic licence at work there. I don't think my veins pumped. I suppose he was trying to portray me as mad. But he was a little shit, I have to say." Palin, characteristically, is instantly visited by an awareness of the need for balance. "I mean that he was like that at the time. Two of them came, from the Oxford University newspaper. You have to accept that not everybody will like something you have done. But there are ways and ways of saying you don't like something. They were so incredibly patronising and condescending."

"Just possibly with a view to winding you up?"

"I suppose so, but why? I can see the weaknesses in everything I've ever done, sometimes to an almost alarming degree."

"What happened after you'd locked yourself out? Did they ransack your underwear drawer?"

"No. Helen showed them out. Someone told me later that Whitley was actually a lovely man, at which point, of course, I began to feel guilty."

Absurd as the word may seem when applied to a man in his 70th year, "boyish" is an adjective that still suits Palin. Though his Sheffield accent is now barely detectable, it returned with a vengeance when he went back to the city for a BBC documentary, Comic Roots, in 1983, and visited the engineering factory where his father, Ted, was employed. Friends describe Edward Moreton Palin as a somewhat intimidating figure; a man whose frequent irritation was exacerbated or even caused by, his painful stammer. He is perceived by some to have been less encouraging than he might have been towards Angela, Michael's only sibling. Nine years his senior, Angela committed suicide in 1987.

Michael was sent to Shrewsbury School, where he encountered the late John Peel (then "Ravenscroft JPR") and his brother Francis. He went on to read modern history at Brasenose College, Oxford.

I tell Palin how I've always felt that his manner betrays both the deference that can be a legacy of a Northern industrial upbringing (where artistic ambition, in the 1950s, might have risked inviting ridicule) and the robust self-belief instilled by a boarding-school education. "Shrewsbury – curious place though it was – certainly didn't let you forget that you were somewhere special," Palin says.

His father sounds rather frightening. ("Quite dour," recalls Robert Hewison, a friend of five decades' standing, while Palin's mother Mary was "wonderful. A tiny woman, but so full of love, and laughter. I think a lot of his character comes from that.") "I couldn't say that I was frightened of my father," Palin says. "But I never felt totally comfortable with him. Perhaps because of his stammer. When you just can't get the words out, it distances you."

"And it placed you – as an articulate child – in the awkward position of being able to do something which he, the adult, couldn't. I imagine that could be enervating."

"That might have been it. My father did have a slight tendency to put down anything I did. I don't think he meant it." (Palin, who drew on childhood memories to play Ken Pile, his speech-impaired character in A Fish Called Wanda, has, for almost 20 years, lent his considerable support to The Michael Palin Centre for Stammering Children, based in central London.)

Palin Senior had also attended Shrewsbury, then gone on to Cambridge University. He had hopes of becoming a church organist, but appears – after a bright start – to have felt that his life failed to proceed as he'd wished.

"He was always confronting people," Michael once told the writer Lynn Barber, a friend and contemporary at Oxford. "Bus conductors, waitresses: he felt everyone was laying traps and should be treated with suspicion. There was always tension when he was around. I found it deeply embarrassing. That's why I hate rows and try to avoid confrontation."

While Ted Palin had little enthusiasm for showbusiness, each of his children embarked on a career in entertainment: Michael in comedy, and Angela, unsuccessfully, as an actress. (His sister would settle for production work, such as studio management at the BBC World Service.)

"What did your father want you to be?"

"He knew I couldn't be an engineer like him. I think maybe something in publishing, or journalism."

At Oxford, he was encouraged by his tutorial partner Hewison ("pushed", to use the latter's word) to participate in student revues, first with Hewison, then, increasingly, with their friend Terry Jones, who would remain his closest writing partner throughout the Monty Python years.

Palin, who has described himself as "one of that cursed generation doomed to take nothing seriously", then became a protégé of David Frost, who first saw him, with Jones, at the Edinburgh Festival. (When you speak to Frost about this period, even today, he can't conceal his pride at the fact that he was the first man to gather all of the Pythons in the same room, as writers on The Frost Report.)

Palin had met Helen Gibbins when he was 16, while on holiday in Southwold, Suffolk: a life-changing encounter that would inspire his poignant 1987 television drama East of Ipswich. While he was at Oxford, Helen was training as a teacher in London. Their relationship, then as now, survived his frequent periods of absence. They married in April 1966, and moved to their current address two years later. (The Palins have since bought the two adjoining properties in the road, and have knocked through into, and also occupy, the next-door house.)

He had a major breakthrough in late 1967, when he appeared with Jones, Eric Idle, David Jason, Denise Coffey and Vivian Stanshall's Bonzo Dog Doo-Dah Band, in Britain's first anarchic children's show, Do Not Adjust Your Set. Watching that footage now, you're struck by how much of the material – Palin's recurrent role as a surreally incompetent shopkeeper, for instance – anticipates Monty Python. Michael Palin's ideas – back then, as so often in his career – were considerably ahead of their time.

"You've said that it was working with Cleese, Graham Chapman and the rest of the cast of Monty Python that allowed you to overcome your diffidence."

"Yes. Until then I had been a very obliging kind of a guy. Having to defend your material, in that company… you just couldn't be diffident. Also, I realised that I was quite good at it. They gave me great confidence."

Though he was principally writing with Jones, Palin's screen performances ignited when he played opposite John Cleese. "I just loved acting with him. Perhaps because there was no competition between us. I was short and he was tall. He was imperious and I was defensive. He needed me and I needed him. It made those shop sketches, like the dead parrot, extremely enjoyable to do. Me trying to be endlessly obliging, him being increasingly aggressive."

"Did you feel subservient in your wider relationship with Cleese?"

"No, because I met him on The Frost Report, after which he was offered all sorts of things, but he rang me up instead, and decided to do Python. Which was quite a risk, for a man so driven by success. When we were collaborating, the only question was: does this work or not? He was very funny, and he appreciated good writing from others. So that was fine. Until the third series, when John clearly wanted to go, and people started to ask why. Whenever humour is taken out of the equation, the Pythons don't necessarily get along that well. Comedy brought us together."

Palin, despite the formidable demands on his time, still extends great generosity towards younger writers, to the point that he's very occasionally prepared to look at an unsolicited manuscript of a 300-page novel and, if he likes it enough, offer a written endorsement. I know about this, because he did it for me. Cleese, by contrast, famously responds to such requests by sending a withering standard letter which begins along the lines of: "Dear Very Nice Person, I have still not read many great books by major authors." Once he's finished those works, he informs the supplicant, perhaps he will find time for them.

"I think you can make that kind of a rule," Palin says, referring to Cleese's rejection note, "if you are prepared to break it. Otherwise, how are you ever going to be taken by surprise? That's just John controlling his world. But if you put the shutters up against the outside world, I think you can live to regret it, eventually."

"These days John Cleese seems increasingly unnerving, at least when viewed from a distance. Was he always like that?"

"Something about John was always very unsettled, I felt. There was always something else he wanted to do. He seemed constantly driven by this sense that there was a nirvana somewhere; some unique place where mind, body and soul would be utterly satisfied."

"That sounds like Middlesbrough. You wouldn't know, because you've never been there."

Palin laughs, graciously. "Well, John, of course, is from Weston-super-Mare. So he knows all about pleasure and fulfilment. I think he set himself a very high standard of achievement and possibly feels he never quite attained it. He's always moving: first to New York, then to California, now Monaco. Where next? I always wanted to say to him: 'John, you're so talented. You have a lovely wife and kids; just relax.' But there was always something more that he wanted, to a point that was almost destructive."

"There's a Monty Python boxed set, where you each apparently compiled a DVD of your favourite scenes …"

"I think I got last pick."

"Cleese's selection is introduced, as you'll know, with specially shot footage in which he plays a very rich, very rude English expatriate, bossing people around at his swimming pool in LA, or somewhere. I remember watching that and thinking: where's the irony there?"

"I should say that I am still very, very fond of John. But the image I always have of him is of this 747 coming in to land. Teams of people, pipes underneath, doors open, people scurrying to service it. Then off it goes again, to the next destination."

"Now there's a man who really does conform to the stereotype of anguished comic genius."

"Yes. Let's call it ACG for short." Palin laughs. "But with a genius, you know," he adds, with no irony, "you can put up with a lot."

Adopting brace position, I explain to Palin that, much as I love almost everything he has done – including The Missionary, and Ripping Yarns, the magnificently deranged series of comedies he wrote with Jones in the 1970s – I belong to a minority of people who can't quite bring themselves to elevate the original Monty Python television shows to the pantheon of truly epic comic productions, such as Hancock's Half Hour, say, or the finest incarnations of Steve Coogan's Knowing Me, Knowing You.

"My problem," I explain, "is that I've always felt that the greatest comedy is accessible to everybody. And, stunning as the highlights of Monty Python are – the Spam sketch, the Lumberjack song, or the parrot – when you watch a typical show, there's a cliquish intellectual quality to much of the humour."

"To that," Palin replies, "I'd have to say, guilty as charged. We did enjoy writing sketches about Marcel Proust. And we were actually trying to debunk that sort of elitism. But you do have to know about something, in order to debunk it. Monty Python has suffered from very high praise. There were some things that quite laboriously didn't work. The television shows were, as you suggest, uneven. Really good material was in there, among a lot of dross. I still think some of the obscure stuff is good, and often needed to be there. But basically I'd agree with you."

"Did your father like the show?" (Edward Palin died in 1977.)

"No, I don't think he quite… got Python. Also, he was developing Parkinson's and the medication was giving him hallucinations. Graham Chapman was a medical man, as you know. He was fascinated by the fact that my father was seeing hamsters running up his trouser leg. In fact, my mother would have to retrieve them and put them in a bag."

I struggle to keep a straight face at this point.

"We shouldn't laugh," I tell him.

"We shouldn't I suppose, but I was grateful then that we could laugh, and Graham found it extraordinary that it was so surreal and Pythonic. My mother, bless her, defended Python. I don't know how much she understood it."

I've always wondered why the Monty Python team didn't give more prominence to its women contributors, especially Carol Cleveland and Connie Booth. And just how is it that Palin (a man who, years later, would conduct so sensitive and uninhibited an interview on the subject of female genital mutilation, in Sahara) could have participated in a Monty Python sketch which included the line: "My very last offer, Mrs Scum… a dagger up the clitoris"?

"That line was Graham's." (Chapman provided the series' more flagrantly misogynistic moments.) "Any references of that kind would have come from him. I'm afraid that was the price of keeping us all together. You had to allow people their own indulgences."

Palin is famously the only Monty Python member who has constantly remained on speaking terms with all the others. Yet, equable as he may be, the youngest Python is not immune to the occasional rush of blood. Terry Jones recalled his friend losing his temper when was asked to do a sixth retake of a scene in Monty Python and the Holy Grail, which required him to get down on his knees and eat mud. ("I just exploded," Palin says, "and started to rant. The whole cast applauded.")

"Michael's children," Jones says, "told me that they used to call him Mr Grumpy. He can get ratty, as anybody can, especially when he's not eaten. When I watch his travel programmes, I can actually tell when he's hungry."

In Diaries 1967-1979: The Python Years, the first of Palin's two published journals, there's a moment where he finds himself in the Berkshire village of Streatley, his blood sugar low. No establishment would serve him dinner after 9.30. "This is Southern England with a vengeance," Palin writes. "We feel like lepers as we walk down the pretty, the fucking pretty, little main street, clutching the crisps [a barmaid] was good enough to let us have."

As befits the trainspotter he once was, Palin's diaries are distinguished by an acute attention to detail, and an unusual determination to get things right. His precise style remains unaltered even when he is recalling the suicide of his sister Angela, at the end of Diaries 1980-88: Halfway to Hollywood, the second and latest volume currently available. She had been staying with Palin, here in north London, hours before she took her life. His journal describes how she left in her "hired" Volvo at "about" 11.15am before returning home to Suffolk and attaching a tube to the vehicle's exhaust. She is survived by three grown-up children.

"It feels intrusive even to mention Angela's name," I tell him. "Not simply on account of her suicide, but because, in your newspaper cuttings file, you can sense so many writers thinking: bingo. Here it is: the smoking gun. He says he's happy enough, but he's a comic writer, and they're all depressives. His sister was depressive; it's genetic – QED. Let's see him smile his way out of this one."

"That's why I included it in the diary; my feelings about Angela, and the details of her death. I was fed up with people going terribly silent."

"Angela was nine years older than you; was she the focus of the same kind of paternal ambition?"

"I think it was directed in a different way. I know that she was extremely bright, and she could have gone to university but didn't. She regretted that."

"Why didn't she go?"

"Perhaps because of money. And then she so much wanted to act. She never got big parts; it didn't really work out for her. But I think she had depression going way back. She was gifted, intelligent and very funny. But for some reason she just couldn't value herself enough."

"All of this brings us back to the notion that you, being her brother, and a comic writer, must also be a secret depressive."

"People look for patterns in everything. It's what keeps us sane, I suppose. I struggle to see any patterns in my life. I think I can understand depression a bit because of my sister. My own feelings of…" Palin hesitates. "I'm aware that, if you feel down, it can be strangely unrelated to circumstances around you. That's just the way life is."

"Of course you must have witnessed depression in others, like Spike Milligan."

"I loved Spike, yet I never realised, until I got to know him, that this person, who made me so happy, was desperate, and that writing the comedy I adored was tearing his life apart. When I interviewed him for Comic Roots, I asked him what it felt like to work on The Goon Show. He said, 'It was like one good summer.'" Palin pauses. "What a moving phrase that is. 'One good summer'. Something fleeting, that you can never recapture. That sense of: was it all just a dream? I think it was extremely perceptive of Spike to say that."

If Palin has enemies (and, while researching this article, I failed to bump into any), I imagine that their chief source of resentment, apart from his maddening calm, would be the enviable versatility of his talent. In addition to his distinguished career as a humourist, comic actor, novelist, screenplay writer and director, in 1991 he astonished many people with his bravura performance as a straight actor in Alan Bleasdale's television epic, GBH.

"Do I remember you saying that was the best part you ever had?"

"Yes, and I would still say that. I don't think I have done anything that has approached GBH, either in terms of the writing, or its size and scale."

Palin's portrayal of Jim Nelson, a principled Lancashire schoolmaster committed to defending natural justice in the face of violent intimidation was simply stunning – and courageous, given that many actors might have hesitated before agreeing to conspire in so politically combustible a drama. GBH managed simultaneously to satirise and eviscerate MI5, the Conservative Party, the right-wing press, and the less mentally grounded elements of Labour's support. Palin's acceptance of the role rendered ridiculous the inference in John Cleese's ungenerous assertion that, "Michael is not a man who will ever die on the barricades, but he will die with thousands of friends."

In a stellar cast that included Julie Walters, Robert Lindsay, Lindsay Duncan, Daniel Massey and many others, Palin shone, with a performance that was moving, passionate, and – in the scene where Nelson, a hypochondriac with a terror of road bridges visits his dying, alcoholic doctor, played by John Shrapnel – painfully funny.

"Weary as Michael Palin might be of hearing this," Alan Bleasdale told me, "he is an inherently and instinctively decent person: very bright, very funny, and a remarkable actor. He has had so many roles where you are clearly meant not to believe in a character that he is playing. Somehow in GBH, without ever having trained as a Method actor, he just became that man."

A friend of Bob Dylan's once remarked that the singer finds it absurd that, "People approach him as if he were a cipher they're required to decode." Palin, while he might not suffer quite such widespread scrutiny as the small man from Minnesota, hints at a similar sentiment. "When I read profiles of myself," he says, "I sometimes think: I have spent my whole life struggling to understand my motivations and impulses, and I've never quite sorted them out.

"To be absolutely frank with you, I'm still not sure I understand them. Then these people wander in, and suddenly they've cracked it in half-an-hour. I always assumed that, with age, I would understand myself better. Unfortunately," Palin laughs, "it's proving to be quite the opposite. I'm no closer to defining what it is that I really am than I ever was – other than somebody who is intensely curious about life."

It may be no coincidence that, in GBH, Palin slipped so effortlessly into the skin of Jim Nelson: an ostensibly passive man who is quietly but stubbornly determined to see that things are done the right way, and is prone to rage when he finds they aren't. I'd imagine that such instincts would have some resonance in Palin's life, whether we're talking about somebody putting his one-inch nails in the three-inch box, or rampant social injustice, or racial bigotry. Towards the end of our conversation, he expounds on the last two subjects at some length.

A phrase I found kept occurring to me in connection with Michael Palin is the French expression: "Il gagne d'être connu," or, roughly translated, he improves with acquaintance. You sense that he still deeply misses his intimate friend and loyal co-conspirator George Harrison. Surviving allies such as the writer Ian Davidson have remained close to him for decades. Robert Hewison had told me that, even after 50 years, "I would love to be able to see Michael every day." Unfortunately, the demands on Palin's time render such a suggestion unfeasible. Which may help explain why intrusive strangers risk being greeted at the door by his evil twin.

"Fame really is a trap," Palin tells me. "When I start complaining about this, Helen quite rightly says: 'Well, here's an idea: don't make another 10-part television series.' But fame would cut anybody off from the kind of things that I like to do."

"Such as?"

"Oh, quite ordinary things. Observing the world. Learning about trains. Discovering new music. You also find yourself bearing other people's expectations. I don't want to bear anybody's expectations. I just want to do… what I can do. And to be judged on that. All of this comes back to what Ernest Hemingway said: 'Don't talk about writing; just write.' And I sometimes tend to think: 'Don't talk about living; just live.'"

'The Truth' is published by W&N, priced £18.99