His dark materials

He is Denmark's Shakespeare, the creator of a body of work so powerful that it's become a part of our everyday language. Yet Hans Christian Andersen was no cuddly weaver of fairy tales for nice kids - oh no, not beneath the surface. His biographer Michael Booth reveals that there was an awful lot more to the great fabulist than the Emperor's New Clothes

Hans Christian Andersen, the "greatest of Danes" (as recently voted by his countrymen in a poll that gave new meaning to the term "foregone conclusion"), was one of the great literary innovators of the Victorian era. This revolutionary genius dredged himself up from the "swamp" of life - as he called it - struggled to make a mark as a poet, novelist and a playwright before, finally, turning to the eventyr, or fairy tale, a genre that could not have been better suited to his strengths, weaknesses, neuroses and obsessions.

In Denmark they mention Andersen in the same breath as Shakespeare and Goethe, and not without reason. Unlike the brothers Grimm who were merely collectors of folk tales, the majority of Andersen's 156 tales were entirely of his own invention. Today stories like The Little Matchgirl, The Emperor's New Clothes, and The Princess On The Pea (it's never "and" the Pea, by the way - that's just one of many mistranslations to have dogged his works over the years) remain potent symbols of the human condition, their titles alone used the world over as a short cut to a rainbow of human frailties.

Without Andersen children's literature would still be stuck in the nursery, and every fairy tale would end happily ever after. The true malevolent power of the genre might never have been realised, and some of the most haunting, instructive and, at times, anarchic children's stories might never have been told. There would be no talking tea pots (Andersen virtually invented the notion of animated inanimate objects), or lovesick mermaids; ugly ducklings would be nothing more than pre-pubescent signets. There would be no Harry Potter, no His Dark Materials, and quite possibly no Roald Dahl either. Shrek is pure Andersen, right down to the morally questionable anti-hero. And without Hans Christian Andersen to lead me astray, I would never have visited a prostitute.

Her name was Sandra, and she worked just off the Reeperbahn in Hamburg. She was one of an unlikely cast of characters I encountered on my journey across Europe researching a book about Andersen, his life and his ceaseless, nomadic wanderings, that also included the first ever female whirling dervish, the Danish ambassador to Rome, Greta Scacchi and the Pope (though not all at the same time). Reading various Danish language texts, many of which have never been translated into English (I learned Danish when I moved to Copenhagen a few years ago), I discovered a figure often at odds with the dreamy man-child of popular perception. The Andersen that emerges from his diaries, almanac and letters is wracked with sexual confusion, driven by fame, obsessed with his own reputation and tormented by insecurities. He was the prototype drama queen who once threw himself face down on Charles Dickens' lawn after reading a negative review and cried his heart out. ("Hans Andersen slept in this room for five weeks - which seemed to the family AGES!" read the note Dickens pinned to his door after he had left Gad's Hill in 1857.) He was a pathological narcissist with a colourful portfolio of neuroses and phobias. He was afraid of dogs, would not eat pork for fear of contracting trichinae and, when he travelled, he carried a nine-metre rope for fear of finding himself trapped by fire. He positioned a note beside his bed every time he slept which read: "I only appear to be sleeping", in case anyone thought he was dead and buried him alive.

The Danes tend to gloss over some of the more problematic aspects of Andersen's character; he is, after all, the sacred cash cow of Danish culture and tourism. They are still in deep denial over his relationships with men, for instance. The fact that Andersen regularly expressed his dislike for his fellow countrymen also tends to be forgotten. "No nation has more prejudice, I think, than the Danes... to deride and sneer, to watch for the weak points in our neighbours, that is our evil nature," he once wrote, and this was far from a one-off rant. In another letter to a friend, while travelling, he railed: "I wish my eyes may never again see the home which can only see my shortcomings... The Danes can be evil, cold, satanic! A people well suited to those damp, mouldy-green islands..."

This whitewashing of Andersen is misguided - after all, rarely has one man's life and work been so intertwined. It is precisely his many personal conflicts - his social and sexual alienation, his doubts, resentments and longings - that give his stories their distinctive, troubling edge.

If Andersen hadn't spent his life in restless pursuit of glory, the Fir Tree would have been content to stay in the quiet forest instead of yearning for the bright lights of the city; if he hadn't been such a neurotic, the Princess would have slept soundly; if he had not been troubled by a deep-seated nihilism The Shadow would have been just another doppelganger farce; if he hadn't felt himself such an outsider, we would never have had The Ugly Duckling; and if he hadn't wrestled with a sexual duality and known countless unrequited loves then he could never have written The Snowman. The Little Mermaid would have been an everyday tale of boy-meets-fish. But still, as I suspect will be the case with the gala concert in Copenhagen on 2 April (featuring an "eclectic" cast of "Hans Christian Andersen ambassadors" including Pele, Elizabeth Hurley and Roger Moore), the Danes prefer to go along with Andersen's account of his life.

In this version Andersen is the innocent, cosseted son of a washerwoman and a shoemaker. Dad nurtures his creative talent with puppet theatres and readings from The Arabian Nights; mum struggles to support them as best she can. After his father dies Hans Christian collects a small sum of money by singing in the drawing rooms of Odense's grander addresses and, true to the fairy-tale cliché, leaves to seek his fortune in Copenhagen aged just 14.

His dream is to enter the Royal Theatre but finding the door slammed on his singing, dancing and acting talents, the freakishly tall, beaky-nosed young man faces destitution. At the last minute he is offered a royal award to attend school in preparation for university, but is relentlessly bullied by a cruel headmaster. Rescued by a tough but compassionate benefactor, Jonas Collin, Andersen returns to Copenhagen where he has some poems and stories published and several works staged at the Royal Theatre. A novel set in Italy - inspired by a pivotal sojourn there in 1832-33 - sells well in Scandinavia and Germany and is followed in 1834 by his first Eventyr, Fortalte for Børn, Første Hefte (Fairy Tales Told for Children, First Instalment). The stories are a great success and, with the exception of various unconsummated love affairs with women, and periodic bouts of excruciating dental pain, all is plain sailing from there on in and Andersen's life continues on an upward curve towards wealth, success and fame until his death, aged 70, from liver cancer. "My life is a lovely story, happy and full of incident... There is a loving God, who directs all things for the best," he once wrote.

The truth is that Andersen's autobiographies are a mix of fantasy, self-justification and self-deception, and there are numerous glaring omissions.

Let's begin with his childhood. Andersen's mother, Anne Marie, was the illegitimate daughter of a woman who may well have been a prostitute. Anne Marie already had a daughter out of wedlock when she became pregnant with Hans Christian two months prior to her marriage to a man 10 years younger than she, who may or may not have been his father (that said, the far-fetched rumour that Andersen was the illegitimate son of King Christian VIII, though given yet another airing in The Times just the other month, was comprehensively disproved some years ago). After his death Anne Marie married again and Hans Christian had little further contact with her during her slow decline into alcoholism.

The image Andersen presented to the world was of the classic poor-born Romantic hero - an innocent genius who fought against insurmountable odds to triumph thanks to a God-given talent. It was a story he retold time and again in his novels - most of them thinly veiled autobiographies - but the truth is that Andersen was a deeply ambitious, driven man who craved approval, attention and success to a degree that verged on the pathological.

To this end Andersen was an incorrigible toady, particularly where royalty were concerned. He had a remarkable gift for making friends in high places, including an impressive tally of European monarchs, crown princes, dukes, counts and lords. The English novelist Mary Russell Mitford was not alone in being "completely disgusted" by Andersen's social mountaineering. "[He] uses fame merely as a key to open drawing-room doors, a ladder to climb into high places," she snorted.

This kind of all-consuming self-dedication had a deleterious affect on his personal life. Andersen recognised early on that his career and ego could never co-exist with a wife: "I shall never be engaged," he wrote after his first rejection, "and it would be a great misfortune if it were ever to take place."

Nevertheless, publicly he courted numerous women - most famously Jenny Lind, the so-called Swedish Nightingale - although he invariably selected targets he knew were unobtainable (women who were either engaged, or about to be engaged or, alternatively, the daughters of friends who would never have allowed the marriage). On the rare occasion when one of his wooings went "wrong" and marriage appeared to be on the cards, Andersen would rapidly extricate himself by pleading poverty. "Beloved Sophie, you will never know how happy I could have been with you, if only I had the money!" he wrote shortly after an "affair" with the daughter of H C Ørsted. As his wealth grew, he simply raised the bar on the income he felt was required to pop the question.

Privately his relationships with men were far more heated, heartfelt affairs. Andersen experienced epic infatuations with the likes of the Grand Duke Carl Alexander of Weimar, Harald Scharff - a dancer with the Royal Ballet, and 30 years younger than he - and, most significantly of all, Edvard Collin, the stand-offish son of his benefactor Jonas Collin. "I long for him daily," Andersen wrote at the height of his whirlwind romance with Scharff. Of Carl Alexander he wrote: "I quite love the young duke, he is the first of all princes that I really find attractive." Theirs was an intense, intimate and unusual relationship; in his diary Andersen almost makes it sound as if he is the heroine who finally gets her prince in one of his stories: "The Hereditary Grand Duke walked arm in arm with me across the courtyard of the castle to my room, kissed me lovingly, asked me always to love him though he was just an ordinary person, asked me to stay with him this winter... Fell asleep with the melancholy, happy feeling that I was the guest of this strange prince at his castle and loved by him... It is like a fairy tale."

Despite this, I don't believe, as some have suggested, that Andersen was 100-per-cent homosexual. The crosses that he used in his diary to mark bouts of masturbation occur as often in connection with meetings with women as with men, and any man who takes the trouble to note in his diary when he spots an attractive nun, as he did on one visit to Italy, can't be all "lavender aunt".

There is one other reason to believe Andersen was as interested in women as men: the frequent and vivid diary descriptions of his encounters with female prostitutes in many of the cities of Europe. Andersen had an uncanny knack for locating a city's red light district and, after many years of loitering in Rome, Naples, Athens, Vienna and Hamburg, he eventually plucked up the courage to visit brothels during several visits to Paris in the late 1860s. He describes these visits as entirely innocent - he claims he just went to talk (which, incidentally, is why I went to see Sandra: to find out how normal this was) and we have no reason to doubt him. But the less well-known visits he made to a Madame in Copenhagen might well have been more "hands-on". We will never know for sure what happened as these visits don't appear in his diary - he only refers to them cryptically in his almanac - but we do know that he was flustered afterwards, not least because he was concerned he might have caught something. I doubt that all he had was a massage and a spot of aromatherapy but Andersen protested his sexual innocence throughout his life and, lacking any hard evidence to the contrary, we must conclude that he was, essentially, a bisexual who never had a homosexual or a heterosexual relationship.

Andersen's dealings with prostitutes were just one aspect of a lifetime's ceaseless travelling that particularly fascinated me. Considering how debilitating his everyday neuroses were - even at home he was a-quiver with anxiety most of the time - it was hard to imagine a less likely transcontinental nomad than Andersen. I often asked myself how this towering neurotic, a man who was afraid of his own shadow, mustered the courage to cross the road, let alone a dangerous continent in the midst of turbulent, revolutionary times. But the more I read about him, the more I came to realise that Andersen's constant need to travel was not merely a symptom of writerly curiosity (although he did publish five excellent but today largely forgotten travel books, the best of which, A Poet's Bazaar, I used as my route finder to the Orient). Rather, his travelling was inextricably linked to, and to a certain extent a self-treatment for, many of his psychological ills: his bitterness towards Denmark, his anxiety about his career and his confused sexuality.

Andersen's relentless social climbing made him a permanent outsider in his homeland - he could never return to life with the washerwomen of Odense, but he remained patronised and looked down upon by the bourgeoisie of Copenhagen. Travel eased this sense of dislocation. If others travel in hope, Andersen travelled to cope. "No son, no brother, can suffer more than I do," he wrote to his benefactor Jonas Collin on the second day of his most extraordinary journey, to Constantinople in 1840, "but it is just as well I am leaving: my soul is unwell."

Though travel was more stressful for Andersen than for most, his customary worries about losing passports and catching things from loo seats were infinitely preferable to dark nights spent nurturing a persecution complex in his rented rooms in Nyhavn, convincing himself that his career was on the skids and that, one day, he might be exposed as a homosexual and drummed out of Denmark. And so this unhappy, lonely, restless man was doomed, like Karen Marie in The Red Shoes, to an eternal dance - across Europe and beyond in a desperate search for love and acclaim.

'Just As Well I'm Leaving - To the Orient with Hans Christian Andersen' by Michael Booth is published by Jonathan Cape in July

10 things to do in Hans Christian Andersen year...

Visit the birthplace

Andersen's hefty scissors are among exhibits at the impressive museum in the Danish city of Odense built around the meagre place where he was born - or somewhere very like it - on 2 April 1805. The bijou attraction avoids sentimentality while housing at its heart a library of the author's writing in around 150 languages. This is the place to go for The Little Mermaid in Nepalese and The Ugly Duckling in Urdu. The bicentenary of Odense's most famous son kicks in on Saturday with a programme of guided tours, film, music, a flower festival (24-28 August) and an opera in September.

Hans Christian Andersen Museum, Odense (+45 65 51 46 20) www.odmus.dk

See 'Thumbelina' the ballet

At Tivoli, the pleasure gardens in Copenhagen that Andersen visited and which inspired his story The Nightingale, two ballet programmes designed by Queen Margrethe run in repertory throughout the summer. Thumbelina is a new piece by Dinna Bjorn, and Love in a Dustbin, first performed in 2001, is inspired by The Shirt Collar, The Darning Needle, and The Sweethearts. Tivoli will also stage a nightly spectacular, a Disney-esque fanatasy of parades, puppetry and pyrotechnics from 14 May.

Analyse the manuscripts

At Rosenborg Castle, a summer-long interactive exhibition takes place in a picture-book shaped building. "The Greatest Fairytale" includes artefacts such as the original manuscript of The Princess On The Pea. Also on display is a vast collection of Andersen book illustrations - the author's distinctive world, one of conspicuous wealth, spectacular poverty and unusual protagonists has been a gift to artists, particularly those with a sense of the absurd or bizarre, such as Dali. "The Greatest Fairytale" opens in May. From 1 October to 29 January; its exhibits go to the City Art Centre in Edinburgh, its only UK showing. Rosenborg Castle, Copenhagen (+45 3315 3286), www.rosenborgslot.dk; City Art Centre, Edinburgh (0131 529 3993)

Check out his paintings

Andersen's liberating travels in Italy, where he was embraced by other Danish artists, are recalled at the National Art Gallery in Copenhagen until 12 June. Among the appreciative and romanticised views of rustic and city street scenes are Andersen's own tiny thumbnail pen and ink sketches of favourite rooms and views. Like his intricate paper cutouts (shown below), the sketches are miniatures and served as aide-memoires when he came to write up the extensive observations of his compulsive journeys. Hans Edvard Norregard-Nielsen, curator of the exhibition, says "Denmark was considered a little, flat and stupid land - not beautiful like Switzerland. Andersen is important because in Italy he learnt to see beauty. He came back to Denmark and saw beauty there too."

National Art Gallery, Copenhagen (+45 3374 8494/www.smk.dk)

Hear Elvis Costello sing

At Copenhagen's dramatic new opera house on the old docks, Elvis Costello will perform new songs inspired by the stories, dovetailed with other pertinent numbers. The new songs are destined to form the basis of a full-scale opera in 2006. "The songs will tell a story that I have imagined existing between the lines of Andersen's biography and some of his most famous tales," Costello explains. "They speak of a misfit's love for an unattainable woman and a struggle between a huckster and someone who composes music in secret."

Enjoy a cantata

Classical music has responded to the bicentenary with a commission for 10 Danish composers whose works will be played in concert halls worldwide. The City of Birmingham Symphony orchestra will perform Per Norgard's Will 'O the Wisps on Saturday, in Birmingham, with Simon Callow narrating. Norgard chose the story of The Marsh Witch and Will 'O the Wisps from a late collection of stories dedicated to fellow Dane and dancer August Bournonville. The 45-minute cantata opens with the Overture 1864 which depicts Denmark's destructive sortie into warfare, then tells the story of a writer who finds all the ingredients for stories trapped in the Marsh Witch's store.

'Will 'O the Wisps': Symphony Hall, Birmingham (0121 780 3333/www.cbso.co.uk), Sat

Or a tone poem...

Danish soprano Lene Sahlholdt has commissioned British composers, including David Matthewsand Matthew Taylor, to write pieces inspired by their own choice of Andersen work. "I've never been approached for a commission in this way," says Taylor. "It's much less finite than being asked to write a symphony or a string quartet. I'm rereading the fairy tales and thinking along the lines of a mini tone poem." The finished suite of pieces will be recorded as a bicentenary tribute and performed in London in the autumn.

Visit his room

Enter Copenhagen's department store Magasin du Nord and turn left at the Hans Christian Andersen commemorative plates to find the writer's modest lodgings. For the stagestruck writer the proximity of these digs to the Royal Theatre made them highly desirable.

Consider his legacy

Andersen's doomed friendship with fellow theatre-lover Charles Dickens is the subject of one section of the British Library's bicentenary exhibition from 20 May to 2 October. The show examines Andersen's debt and contribution to English literature as well as the impact of his work on theatre, opera and film.

British Library, 96 Euston Road, London NW1 (020 7412 7332)

Eat a cake...

In Copenhagen you can now dine in the restaurant Olsen at the table where Andersen entertained and possibly wooed the Swedish soprano Jenny Lind. You can buy a Hans Christian Andersen umbrella decorated with cut-out figures or stock up on themed table linen from Georg Jensen. At the patisserie La Glace, which Andersen probably patronised, you can munch your way through a different fairy-tale fancy for each month. Or you can cut into a slice of raspberry butter, lemon mousse and hazelnut cake with white chocolate cut-outs. Like most Andersen creations, it's bitter-sweet.

Claudia Pritchard