Douglas Adams: How a new biography sheds light on his genius

In the 13 years since he died, Douglas Adams' fictional universe has lost none of its appeal. His biographer Jem Roberts explains how he came to tell his story – and the treasures he found

Evil eye: Douglas Adams in 'mad genius' pose (Corbis)

Comedy matters. Arguably the toughest and most widely loved art form we have, comedy is still finding its rightful place within our national heritage, with a Comedy Museum newly opened in the capital, an impressive landmark crafted from catchphrases in Blackpool, and spods like me devoting their lives to preserving and documenting the lives of our best comic minds – and, indeed, their jokes.

For my first book, the official I'm Sorry I Haven't a Clue Bible, I was lucky enough to interview two of the greatest comedy bosses of all time, Sir David Hatch and Geoffrey Perkins. Their early deaths made all the more clear the importance of researching and celebrating the creation of classic comedy, while those who were central to the process are still around to share their memories. But once that is no longer an option, what is left to give comedy fans some insight into the provocation of laughter?

The True History of the Blackadder provided my first experience of digging into private comedy archives, when Richard Curtis handed me the script for a "lost" Christmas special, "Blackadder In Bethlehem" (http://tiny.cc/blackbethlehem), which nobody outside the Adder coterie had ever got to see. There was a certain amount of quibbling over the media's use of the word "lost", seeing as Curtis had simply found the file on his desktop and printed it off for me, but it seems a reasonable shorthand for "something you probably never even knew existed until now". Although I never had any ambition to become a biographer, being subsequently accepted as the new official chronicler of the career of Douglas Adams was to provide a whole new level of archive exploration.

Adams' sudden death in 2001 inevitably left a tangled web of unfinished work, novel drafts, random jokes, diaries and notebooks packed with the kind of galaxy-shatteringly unique or ridiculous concepts that made his fortune with the success of Hitchhiker. Within a year of his passing, the digital trail salvaged from Douglas's numerous Apple Macs was edited and selectively published in the collection The Salmon of Doubt. But nobody had considered the paper trail, which must have been equally rich with fascinating, insightful and, above all, funny material. The great man's gadget-packed study remained as it had been for many years before all the documents were collected into cardboard boxes; and in time his widow, Jane Belson, remarried. Jane herself died only 10 years after Douglas, leaving their daughter, Polly, and her stepfather George as custodians of – well, of what, precisely?

It was Polly and George's decision to donate the entire paper archive to Adams' old Cambridge college, St John's, which finally opened up 25 years' worth of unpublished work for academic exploration. Thanks to their kindness, I was the first writer to be given access to boxes of comedy gold liable to send any of Hitchhiker's millions of devotees (myself included) far over the moon. Although the narratives of both my previous books had started in Cambridge, I blush to admit that this was my first time in the city, where I was given rooms in St John's for my weeks of research. The Adams archives, as arranged by the pleasingly monikered archivist Mandy Marvin, provided even more wonder than my medieval surroundings.

My gingerly perusal of a few old radio scripts (already seen at the BBC Archives in Reading and the Science Fiction Museum at the University of Liverpool) was quickly curtailed when the librarian casually mentioned that "there might be a few notebooks worth examining", and from the moment my unhallowed mitts cautiously prised open the first 35-year-old exercise book, packed with scribbled scripts, abandoned sketches and red herrings, I was wrapt – not just by the genius flowing out of every page, but by the heart-warming ordinariness on display, of a frustrated writer having bad days, the dead-ends with which any creative person would immediately sympathise.

Of course, the announcement that unpublished material is being used in a biography triggers questions about the morality of sharing a writer's raw work with the general public – especially if that writer is a notorious perfectionist like Adams, who liked to hone and tweak every paragraph long past any reasonable deadline. Besides assuring any outraged fan that the Adams Estate and associates – his daughter Polly, his agent Ed Victor and his friend and collaborator Robbie Stamp – have been rightly obsessive in their care for Douglas's memory and reputation, I should emphasise that, of the material I read, maybe one third was unpublished but arguably publishable, and the material included in The Frood represents perhaps less than a fifth of that.

This is not a case of cashing in on shopping lists or printing Marvin the Paranoid Android jokes with an extra word added. Anyone expecting to plough through pages of hastily abandoned erotica will be disappointed. The Frood strives to present only the most intriguing signposts and meticulously wrought prose from one of our most important humorists, to make his fans laugh once again at "fresh" jokes and to shed light on his painstaking craft. Much of the material was considerably reworked at the time – the original draft of Life, The Universe and Everything had been two-thirds written and rewritten before being entirely abandoned and started from scratch in 1981. My researches also turned up unfinished drafts for a second Hitchhiker TV series that never got made, reams of printed notes for an abandoned Dirk Gently book, A Spoon Too Short, and numerous abandoned chapters for Mostly Harmless.

Marvin's discovery of some form of happiness thanks to misery-loving beings known as "the Drubbers", Arthur Dent's befuddled diaries covering the Krikkit invasion, the potted biography of some messianic former editor of the Guide known as "Baggy the Runch", the original adventures of Wowbagger the Infinitely Prolonged, before he became famous for his intergalactic odyssey of abuse… so many extracts provided fresh food for thought for even the most obsessive Hitchhiker fans, but none of the chosen material in any way compromised the sadly absent author.'

Only the best of this material has been deemed worth sharing, but my trump card for anyone doubting the deep respect with which Adams' paper trail has been followed here is the late author's own introduction. He had written this for a second edition of Sunset at Blandings, the unfinished last novel of his hero PG Wodehouse, who was just as much of a perfectionist himself. Properly contextualised, the publication of private material can be seen as the greatest compliment that can be paid to a deceased writer, enabling them to continue to entertain and illuminate long after they are gone.

Richard Usborne wrote one of the finest posthumous appreciations of the humorist's art in Wodehouse at Work to the End, one year after Wodehouse's death. I hope that The Frood, coming 13 years after the loss of Douglas Adams, will stand as a similar tribute – and that the continued interest in the Galaxy he created will serve as a testament to his hard work at the keyboard and wastepaper basket.

'The Frood: The Authorised and Very Official History of Douglas Adams & The Hitchhiker's Guide to the Galaxy' by Jem Roberts is published on 25 September 2014 by Preface, hardback.

Lost laughter: Other precious comedy gems yet to be dug up...

Monty Python's 'Hastily Cobbled Together For a Fast Buck'

This Python bootleg was put together by their regular producer André Jacquemin from album off-cuts in 1981, and was due to be sold with a proud disregard for quality or morality. However, despite the team's dedication to earning lots of lovely currency from every crumb of comedy they ever produced, and although the bootleg is packed with top-grade 'pepperpot' dialogues and alternative song takes, the full album has never officially surfaced.

Peter Cook & Eleanor Bron

Widely held to be the finest wit of the 20th century, the hideously profligate and disorganised Cook left mountains of offhand humour in private corners when he died in 1995. Of particular interest would be recordings made late in life with one of his longest-standing collaborators, Eleanor Bron. While filming small roles as Lord and Lady Wexmire in Black Beauty, they developed the characters of "two extravagant foreigners" and improvised reams of dialogue with a view to creating a two-hander play.

Alexei Sayle's 'Small World'

When asked which project in his career he most regrets never seeing come to fruition, Alexei Sayle never hesitates before nominating Small World, a 1990 BBC2 sitcom pilot recorded for its "Comedy Asides" strand, which gave us Mornin' Sarge, I, Lovett, KYTV and The High Life. Co-starring with Phil Nice and John Fortune, Sayle played dual roles, Sir Roland and Winston Crust, in an apparently inexplicable plot involving nuclear war and gangs of rabbis…Written by John Sullivan! Allegedly this esoteric misturn in Sayles's career was dismissed as a mid-life moment of madness and prevented from ever reaching TV screens.

'On the Margin' and more

The BBC's short-sighted deletion of oceans of comedy gold in the 1960s and beyond have robbed us of whole series of Cook & Moore's Not Only But Also, John Fortune and Eleanor Bron's Where Was Spring? and another programme to feature both unfortunate comedians at the height of their powers, Alan Bennett's A Bit of Fry & Laurie-inspiring intelligent sketch show, On the Margin. Still, with random half-hours of Doctor Who surfacing in different states of disrepair all over the place, every comedy fan still harbours hope that any of the above may turn up somewhere, somehow...