The Secret Life of Walter Mitty: The return of the little man with the big imagination

In his new film, Ben Stiller takes on the role made famous by Danny Kaye. But behind the final production lie decades of planning and several tantalising near-misses

James Thurber's short story The Secret Life of Walter Mitty is just over 2,000 words long, and was first published in The New Yorker on 18 March 1939. Its protagonist, a henpecked husband living in suburban Connecticut, periodically escapes from his humdrum existence into fantasy, where he re-imagines himself as a daring pilot, a dashing surgeon and a deadly assassin.

Almost 75 years later, Thurber's creation has become a byword for self-delusion, and his simple premise holds enduring appeal for film-makers: since the story's publication, the snap-cut from heroic fantasy to banal real-life has become a commonplace cinematic trope. Walter Mitty himself first made it to the big screen in 1947, and a new film adaptation is due in cinemas this month. Directed by and starring Ben Stiller, this contemporary version introduces Mitty as a milquetoast magazine employee, whose reveries elide with reality when he sets off across the world to track down a missing photographic negative.

However, the project has taken so long to emerge from development hell that at times it must have seemed as fantastical as one of its protagonist's daydreams. The directors who have been tempted to take the helm are among the most celebrated in Hollywood. The list of stars attached is a roster of the defining comic talents of the past 20 years. And all along, one family from the film-industry aristocracy has remained committed to returning Mitty to the movies.

Samuel Goldwyn was one of the original Hollywood moguls, a Polish-Jewish immigrant who arrived in the US with nothing at the end of the 19th century, and rose to head his own major production company, Samuel Goldwyn Pictures. No Walter Mitty he. Eight years after it appeared in The New Yorker, Goldwyn produced the first film adaptation of Thurber's story, which he had altered to accommodate the talents of its star, song-and-dance man Danny Kaye. Though Goldwyn consulted the author repeatedly during the writing of the script, Thurber was reportedly dismayed by the result, which he privately described as "The Public Life of Danny Kaye".

Half a century later, Goldwyn's son, Samuel Goldwyn Jr, himself a successful movie producer, hit on the idea of a remake featuring the biggest comedy star of the moment: Jim Carrey. Carrey was fresh from a string of hits, including Ace Ventura: Pet Detective, The Mask and Dumb and Dumber, all released in 1994. Mitty could be tailored to fit his wacky screen persona. Carrey was offered $20m to take the role, and in 1997 Ron Howard came on board as director. Yet their writers struggled to devise a satisfactory modern storyline, and Howard left to pursue other projects – including How the Grinch Stole Christmas (2000), starring Carrey.

At least one more director had come and gone by the time Goldwyn Jr filed a lawsuit against New Line, the studio that owned the rights to Thurber's story, in 2001. New Line had acquired Mitty six years beforehand on the understanding that Goldwyn would produce the remake; the studio wanted to pass the rights to another company earlier than the deal stipulated, thus removing Goldwyn's creative control of the project. In 2002, he won the suit and the rights, which he took to Paramount Pictures, where his son, John Goldwyn, was an executive.

In Spring 2003, John Goldwyn met with Carrey and Steven Spielberg, who were keen to collaborate, and had at one time considered working together on Meet the Parents (2000) – a film that eventually fell to Stiller. Goldwyn later described the meeting to Variety, recalling, "We're sitting there with Steven… when Jim asks, 'What ever happened to that Walter Mitty project?' Steven says, 'You mean the old Danny Kaye movie? I'd direct that if you starred in it.' Jim says, 'I'd star in it if you directed it.' And [Paramount studio head] Sherry Lansing says, 'I'd finance it if you directed it and you starred in it.' She turns to me and says, 'John, get the rights.' And I was able to say, 'I already have them.'"

Yet again the project stalled, with Spielberg moving on to make War of the Worlds. Samuel Goldwyn Jr hired the Oscar-nominated screenwriter Richard LaGravenese to rethink the script, and it was LaGravenese who introduced the love story that survived into the Stiller version, which features Kristen Wiig as Walter's co-worker and the object of his affection. That wasn't enough to avert another disaster: Carrey, who had been with the project since its inception, pulled out in 2005.

Owen Wilson, supposedly a fan of Thurber's story, took Carrey's place, but changed his mind months later due to so-called creative differences. Paramount's hold on the remake expired at the end of 2005, John Goldwyn had already left the studio, and two years later his father's long-gestating project surfaced again at 20th Century Fox, with Mike Myers in the title role.

A new script was commissioned to match the star of such broad parodies as Wayne's World and Austin Powers, who was still bankable thanks to the Shrek franchise. But then, in 2008, Myers wrote and starred in The Love Guru, which made back just $41m of its $62m budget at the box office. He has not starred in a movie since.

In April 2010, it was reported that Sacha Baron Cohen would play Mitty, from yet another script, this time by Peter Morgan, the British screenwriter of The Queen and Frost/Nixon. In Morgan's version, Mitty would be a "timid mega-store owner" who puts fantasy aside to deal with a real-life drama. But when Gore Verbinski, director of Pirates of the Caribbean, agreed to direct the film, Cohen's name dropped off the cast list.

Verbinksi, too, departed the film, but a screenplay by his collaborator Steve Conrad ended up in the hands of Stiller, who in April 2011 signed on to star, and several months later to direct. The Secret Life of Walter Mitty will finally appear in US cinemas on Christmas Day. Among its producers are the Goldwyns: John and Samuel Jr, who is now 87. Mitty marks the latter's first credit for 10 years, since 2003's Master and Commander – though it took him almost 20 years to earn it.