The art of diary writing

January signals not only the arrival of a new year – for many, it will also open another chapter of obsessive diary-writing. What drives these solitary scribblers? Tim Walker delves into their secret world

The 1894 volume of Godfrey Williams' diary is squarebound in green cloth and offers accommodation for two days' events per each A5 side. Williams recorded the year, which he spent posted to India with the British Army, in a neat and consistent hand. Each densely-inscribed page is interleaved with blotting paper, providing space for footnotes, clarifications and obituaries of his former officers and comrades, which were clipped from his daily newspaper later in life. In total there are 76 such volumes, a good 2ft of shelf space rich with the details of Williams' daily life. A veteran of two world wars, a committed cricket fan, stamp collector and rearer of poultry who died aged 85, he composed his very last entry in the summer of 1955: "A very bad night indeed last night ... " He had been keeping a diary since his school days at Harrow, beginning in 1879.

Williams was no Samuel Pepys. But his diary – a distillation of almost an entire life, traversing two centuries – could nonetheless be a goldmine for historians or psychologists. And it might have been split, lost or destroyed altogether, were it not for the intervention of Dr Irving Finkel. Finkel is a curator at the British Museum specialising in ancient Mesopotamian scripts, but it is the preservation of more modern texts that has become his hobby since he stumbled across Williams' diary.

"About 10 years ago," Finkel recalls, "A friend of mine came to me looking very grumpy. He had [Williams'] diaries and he wanted to sell them, but the Imperial War Museum was only interested in those sections that covered the wars, and he didn't want to split them up. So I bought them, and that started me off; I've never stopped. Now I indiscriminately collect every diary I can find, on the basis that even one about changing the tax disc or mending the fence, if it were 400 years old, would be very interesting. Something that's absolutely contemporary looks like rubbish. Something from the Sixties that might have been in the same room as Jimi Hendrix already has a certain romance. Something from the 1860s is precious. And something from 1660 is like gold dust. It's the march of time that makes things interesting."

This Friday, 1 January, it will be 350 years since Samuel Pepys first put quill to parchment and began his famous diaries. It's very easy to imagine not only that journals like Godfrey Williams' are under threat, but that the very idea of the daily diary is imperilled by blogs, PDAs and the teeming distractions of modern life; that Bridget Jones's Biro could soon be replaced by the blogger's beloved keyboard. Yet if diary devotees like Finkel are to be believed, then the blog is incapable of replacing the diary, and diarists are likely as numerous as ever; it's just that most of them are, by their very nature, private about the practice. "Since I bought that first set I've found it difficult not to ask everyone I meet whether they keep a diary," says Finkel – who does not keep one himself. "I've found that even people I've known personally for a long time are diarists and I never knew it. We won't know whether diary writing is in decline until later, because we don't realise that it's going on right now. Diarists have to be left in privacy."

Some people can be persuaded to share the intimate moments recorded in their diaries, but only at a slight remove from those moments. The writer Sarah Brown organises a regular event at the George pub in central London called Cringe, at which volunteers read embarrassing entries from their teenage journals aloud, long after the fact. Brown has selected the very best examples ("Mark said I had big knockers today, which isn't strictly true, but is a lovely compliment anyway ... "; "I hate my family. I want curly hair and to be happy. Suicide is the only way ... "; and so on) and compiled a book, also entitled Cringe (Michael O'Mara, 2008). "I've kept a diary since I was five years old," says Brown, "and I've still got them all. The one I've been writing for the last five years is still too tender to look back through, but the point of Cringe is that it's teenage diaries, with all the hormones and mood swings and drama it entails. Everything seemed a big deal then, which is why it's so funny to look back at now."

The fictional Adrian Mole famously began recording his many youthful humiliations aged 13-and-three-quarters, and their teens are when most people begin keeping diaries, be it as a way to let off steam about a much-fancied classmate, or simply because a generous aunt gave them a leather-bound notebook for Christmas that they feel obliged to fill. A recent Senate House catalogue of published and unpublished diaries about London, says Finkel, proved that diarists come from all walks of life, from cabinet ministers to cabbies. Sharing diaries, however, is "a great equaliser," Brown insists. "If they didn't get up to read, you'd have a different view of most of the people at Cringe. If you were standing next to them at the bar, you'd give them a once-over and form a judgement, but the minute they get up and read those experiences from their teenage diaries you have an immediate respect for them – because they have a sense of humour about themselves."

Many of the diaries read aloud at Cringe would have been written before instant publication was possible thanks to Blogger, Wordpress or LiveJournal but, says Brown, on the few occasions she has received blog submissions from younger writers, she has rejected them. A blogger's instinct is often to fashion a supplementary persona – like call-girl Belle de Jour, or Salam Pax, the Baghdad Blogger – which can prevent them pouring their most intimate feelings onto the page. "I write a blog," Brown explains, "and when I first started I didn't think anyone would ever find it so I wrote in some detail about my private life. But I soon discovered that my friends and family were reading it. No matter what you're comfortable confessing online, you'll never write for an audience the way you write for yourself. I don't think the internet could ever replace a book that you keep under your pillow."

Both Brown and Finkel are distrustful of diaries written with publication in mind, but such records serve a valuable public purpose, too. Chris Mullin, Labour MP for Sunderland South, recently published A View from the Foothills, (Profile, 2008) a diary of his years as a junior minister in Tony Blair's government from 1999 to 2005 – and he intends to publish two more volumes, covering 1994-1999 and 2005-2010. "I kept a diary when I was a teenager," Mullin admits, "but hopefully none of that will ever see the light of day ... I started keeping it again on the night John Smith died, in May 1994. I felt I was a witness to significant historical events and, being a writer [Mullin is also a novelist] felt a compulsive desire to record them. Over the years it became a kind of therapy; I could let off steam in the diary when I couldn't in public."

Many private diarists neglect descriptions of major news events – from World War Two to 9/11 – in favour of a paragraph about vacuuming the sitting room. Not so the political diarist. Alan Clark's diaries turned the junior Conservative minister into the definitive insider chronicler of the Thatcher years; Mullin's ambition is to provide the same service for New Labour. "It's those occupying the lower foothills who write the best diaries," he explains. "People at the very top are far too busy to keep a diary and, if they do, they understandably spend a lot of their time trying to justify their decisions. Lower down the pecking order, you have less inclination to do so." Samuel Pepys himself was an impeccably connected backbench MP, eventually rising to become Chief Secretary to the Admiralty.

"Jock Colville [whose diaries cover the war years in 10 Downing Street] was private secretary to Churchill," says Mullin. "Harold Nicolson [whose published diaries covered 1930 to the 1950s] never rose above junior minister but was a toff so he knew everybody. Chips Channon never rose above the lowly level of PPS to the Deputy Foreign Secretary, but he was married to a Guinness, so he entertained on a gargantuan scale. His diaries had the King coming to dinner during the abdication crisis."

Irving Finkel's diary collection fills two large cupboards, a number of boxes dotted around his artefact-strewn office at the British Museum and, unknown to his wife, a couple more at home. "I'm convinced people tell the truth when they write a diary," he says. "It's very hard to encounter someone so intimately in normal human interaction unless they're a family member. Most people are encapsulated by social devices that keep others at a distance."

Even within some diaries there are such devices. An apparently impenetrable table of numbers and country names in the back of one of Godfrey Williams' early diaries turned out to be a catalogue of his stamp collection. One of Finkel's other acquisitions is the wartime diary of a schoolgirl that was split between teenage reminiscences in English, and lengthy passages in a substitution code. A linguist by trade, Finkel saw it as a point of honour to break the code, and found that "those sections were about her mother being a part-time prostitute and having clients come to the house. It's heartbreaking ... In some respect, the diary provides a rescue corner for the human spirit." Says Brown: "Re-reading an old diary is a good way to remember that if you're having a hard time, you've had hard times before and got through them."

In his search for the diaries of ordinary people, Finkel has found that many potential historical treasures have disappeared, discarded or destroyed by unthinking mourners or removal men. In the hope of preserving more, he has proposed the notion of a National Diary Repository: a single, large space where people could send old diaries, to be salvaged for posterity. "If you find a long run of diaries," he says, "say 40 years, it takes a long time to read, and you can be drawn in by the most apparently mundane things with the same power as a clever literary novel ... Some people are novelists or journalists and get published regularly. But a great proportion of the human race never publishes anything, so if you're a diarist you have an unpublished manuscript of substantial length that sums up your whole life. It seems a tragedy that it could be lost, burned or thrown away."