Carnival of culture: Our guide to the pick of the Edinburgh Festival

Despite the recession, the Edinburgh Festival and its anarchic Fringe, which start today, are more popular and inventive than ever. Jonathan Brown celebrates this year's fantastic line-up

Ask Jonathan Mills why he thinks it is that so many people make the annual pilgrimage to the Scottish capital each summer to feast on the extravaganza of performance, art and culture that has given the city the world's biggest and most famous festival and he provides an answer of quite breathtaking lucidity. "They come for the wonderful and wild juxtaposition of the highly programmed to the chaotically construed," he explains. "From the precision of a military band to the speculative nature of an author rambling on about what they were thinking when they started their first novel – it is these contrasts that give Edinburgh its special flavour."

The Australian-born artistic director of the Edinburgh International Festival, the ninth person to occupy the illustrious post in the event's 63-year history, is under little illusion that there will be any single star in this year's constellation of offerings, though he is sure many will shine brightly. Since its inception in the austere days following the Second World War, the city fathers' unique vision of a platform to provide for the "flowering of the human spirit" has taken on a life of its own far beyond the control of a single organising committee or individual curator. Today a dozen festivals – from science to film, jazz to Asian culture – stud the calendar. But it is at the height of summer, when thousands of performers vie for attention over an intense three weeks that, according to Mills, a spirit of "exuberant chaos" reigns free, and the city and the people who have travelled from all over the world to join in become as much a part of the festival proceedings as the various goings-on at the venues themselves. Such is the appeal of Edinburgh in August that only the Olympic Games and the Football World Cup are capable of selling more tickets, he suggests.

Much of the speculation in the run-up to this year's festival has focused on whether or not the festivals can continue to exert such extraordinary pulling power. At the centre of this speculation is the Fringe Festival. Having grown from humble origins as an alternative to the International programme at its 1947 outset, when eight theatre troupes gatecrashed the proceedings, the Fringe, with more than 30,000 performances spread over 250 locations, now dwarfs the other festivals. But last year, following decades of apparently unstoppable growth, ticket sales fell by 10 per cent. The main cause of the slump was the repeated failures of the Fringe's computerised box office. The crashes spawned a slew of negative headlines. Add to this a sopping-wet August and the breakaway of some of the festival's most high-profile venues to create their own comedy offering, and the Fringe found itself dogged with the prefix "troubled" whenever it was written about.

Jon Morgan, the director, stunned the arts world when he announced he was standing down in August after just over a year in the job. His replacement, Kath Mainland, was a popular choice, however, having an association with the city arts scene dating back 20 years to an early job looking after a brace of Shetland ponies deployed in the King's Theatre pantomime – a position that required regular and smelly cleaning-up duties as the horses suffered from stage fright. Mainland was also known as the woman who brought Christian Slater to heel at the Assembly Rooms bar after the Hollywood A-lister had been enthusiastically celebrating his birthday during his appearances in a run of One Flew Over the Cuckoo's Nest.

Ensconced as the Fringe's first chief executive, she declares herself "cautiously optimistic" as the first previews get underway. While there have been concerns over a tram project in Princes Street – which, according to one venue boss, has turned the city's most important thoroughfare into a "war zone" – and angst that a strike by council refuse collectors could deter festival goers, things are good. The box office is currently working without a glitch and sales are well ahead of the bumper year of 2007 – defying the gloom-mongers who believed the festival would fall foul of the recession. "There are things to do with the economic climate that are not bad for us," explains Mainland. "The pound is not as strong as it has been and if people are staying at home or holidaying in this country we are benefiting from that. There are pluses and minuses and we have a long way to go."

What will matter most, though, is the quality of the shows. For a long time, critics have accused the Fringe of growing too big and too commercial, and providing too many opportunities for a certain type of show-off with a questionable degree of talent. Meanwhile, in the last decade the number of festivals in Britain and around the world has soared, with powerful alternatives emerging such as the biennial Manchester International or Latitude. Fixtures such as Glastonbury have also widened their remit beyond music, marijuana and mud to include literary and performance-based stages.

But Mainland, whose career has taken in senior roles such as running the Assembly Rooms, the Hogmanay celebrations and the International Book Festival, believes that "you can never have too much artistic endeavour". She says: "The appeal of the Festival is so many people congregated together in one place to experience a convergence of stuff you would never be able to see if it was not part of a single festival."

The founding principle that has driven the Fringe to gigantic proportions (even in these recession-stricken times, there are 10 more acts than last year) is that of open access. This, believes Mainland, has tapped into a "huge artistic entrepreneurial spirit", which sees venues locked in healthy battle to attract audiences. The punters vote with their feet.

Of course, what everyone looks for at Edinburgh, alongside the long, light nights and the plentiful whisky – be they agents, promoters, producers or arts lovers looking for dinner-party bragging rights – is the next big thing. This Edinburgh tradition is normally dated to the Fringe debut of Tom Stoppard's Rosencrantz and Guildenstern Are Dead in 1966, which went on to become a West End and Broadway hit. Forty years later, Gregory Burke's Iraq drama Black Watch was similarly acclaimed. But it is in the field of comedy that most growth has occurred in the past 20 years, and where some of the most glittering reputations have been forged. In four years in the early 1990s, Frank Skinner, Lee Evans, Steve Coogan, Harry Hill and Sean Hughes all picked up Perrier Awards before going on to enjoy major success.

The funny men and women again dominate this year, accounting for 35 per cent of the acts on show, compared to 28 per cent for theatre and 16 per cent music. The banking crisis, however, has claimed the sponsorship of Perrier's successors, the internet saving house Intelligent Finance, and this year's gong will be known simply as the Edinburgh Comedy Award as a new corporate partner is sought. But big-name acts want to be here, most notably Ricky Gervais, who is using the festival to launch his new stand-up tour, Science. Julian Clary celebrates his 50th birthday (with or without special mention of Norman Lamont), while Paul Merton and top US acts such as Janeane Garofalo also appear. Stewart Lee will return to the festival with a new show, while the surprise 2005 Perrier winner Laura Solon brings her Radio 4 sketch show to the Assembly Rooms.

Alongside the comics will be an array of retro, camp musical theatre, including A Team: The Musical and an unlikely offering based on the life of the Radio 1 DJ Scott Mills. The theatre schedule also shows that no event is too recent to be satirised, with the premier of Michael Jackson at the Gates of Heaven and Hell expected to generate headlines. Throw in a powerful line-up of musical performances, including the post-punk pioneers Magazine and Edinburgh's own Edwyn Collins, alongside the usual extensive children's shows, and Mainland is convinced the eclecticism will be matched by quality and a new generation of Edinburgh legends. "I love the way that word of mouth operates here. Things that no one was talking about yesterday, everybody is talking abut today," she says.

In contrast to the anarchy of the Fringe line-up, the International Festival director Jonathan Mills has opted to "shape a journey" for his audiences that will explore some of the themes of the Scottish Enlightenment. The festival coincides with Homecoming 2009, a tourism-inspired extravaganza to mark the 250th anniversary of the birth of the poet Robbie Burns. But Mills, who has commissioned 22 new works to mark his third year in charge of the EIF, believes many of the issues arising from that era – the quest for certainty, faith and what happens when the utopias go wrong – will play well with modern audiences, not least here in the city of Adam Smith and David Hume.

Festival goers can expect major new productions, including Rona Munro's The Last Witch, which tells the story of a woman on a remote Scottish community who is suspected of sorcery and ends up being boiled alive in a vat of tar. There will also be a production of Goethe's Faust with a cast of 120 enacting scenes of sex and violence, which carries adult-only warnings. The Royal Ballet of Flanders will be performing The Return of Ulysses – combining unlikely musical contributions from Purcell and Doris Day– and there is the much-anticipated Actus Tragicus, the late German director-designer Herbert Wernicke's danse macabre based on six works by Bach.

Mills says he is unfazed by the growth in the number of competing festivals around the world. "I am never worried about a successful festival – only an unsuccessful one because the bad ones turn people off. There has to be a limit but I do not know that anyone knows what it is. Who would have thought back in 1947 that in any given year the Edinburgh Festival would have issued four million tickets?"

The other great attraction of the coming weeks, the Edinburgh International Book Festival, is now ensconced as the world's number one literary event. Some 200,000 people are expected to pass through Charlotte Square Gardens over the duration to come face to face with the 750 authors booked to appear. This year's highlight will be Margaret Atwood, introducing the world to her latest book, Year of the Flood, and teaming up with the Festival of Spirituality and Peace at St John's Church at Princes Street. The event has been sold out for months but organisers say tickets are still available for big-name writers including Douglas Coupland, the Red Riding trilogy creator, David Peace, and the collaborators on the cult TV series The Wire, David Simon and Richard Price.

The former Bishop of Edinburgh Richard Holloway was drafted in at short notice after the departure of the Book Festival's previous artistic director, Catherine Lockerbie, due to ill health. As guest director, the former Bishop said he inherited a Rolls-Royce organisation and an almost fully formed programme. Holloway, a popular broadcaster and writer as well as a well-known liberal theologian, has devoted his energies to developing the festival debate strand.

Among the subjects under discussion will be Islam and women, poverty in Britain and the future of childhood. "There is a new thirst for real, deep, content-full debate everywhere. There is a new hunger for it which is really exciting," he says. He expects there to be an "ecstatically wonderful" atmosphere in the book tents, with "the whole authorial world dipping in and out for a couple of weeks".

Holloway reckons that "under those tents, the very air is invigorating". It is a noble and inspiring image, and one which the festival idealistic founders might recognize – whatever the doom-mongers say.

The pick of the crop: 20 unmissable events

Calvin Harris

Dancing shoes on for the Scottish-born DJ best known for his work with Dizzee Rascal and Kylie Minogue. Picture House, 10 August

Jack Bruce/Robin Trower/Gary Husband

After Cream's reunion, the band's original songwriter, Jack Bruce, teams up with the virtuoso guitarist Robin Trower and the drummer Gary Husband. Beware of musos. Queens Hall, 5 August

The Last Witch

Rona Munro's play tells the story of an 18th-century woman able to talk the fish out of the sea. Royal Lyceum Theatre, 23-29 August

Janeane Garofalo

Left the stage after six minutes at Latitude, but this has done little to quell the excitement over the US stand-up's Edinburgh debut. Gilded Balloon Debating Hall, 6-15 August

Magazine

Howard Devoto's post-punks enjoyed more influence than success but they have been thrilling audiences since reforming this year. Picture House, 30 August

A-Team: The Musical

The return of Hannibal, BA Baracus and friends from the hit 1980s action show, with the added allure of music and songs. Gilded Balloon Wine Bar, 5-31 August

Margaret Atwood

The Grande Dame of Commonwealth fiction launches a new novel, 'The Year of the Flood' with actors and a choir. St John's Church, 30 August

Ricky Gervais

Tickets for the opening night of the 'Office' star's stand-up tour have been changing hands for vast sums. Playhouse, 25 August

Malcolm McLaren Live!

'History Is for Pissing On'. Never one to fight shy of controversy, the former Sex Pistol boss reminisces. Pleasance Grand, 23 August

The Discovery of Spain

Masterpieces by El Greco, Goya and Picasso as well as works from British artists seduced by the people and landscapes of Iberia. National Gallery Complex, until 11 October

Women and Islam

Join what promises to be an explosive debate on this most controversial of issues with the Islamic scholar Carole Hillenbrand and Yasmin Hai. Charlotte Square Gardens, 18 August

Cornelia Funke

Germany's answer to J K Rowling will be discussing her 'Inkworld' series as well as her forthcoming book 'Reckless'. Charlotte Square Gardens, 17 August

Edwyn Collins

Back on the road to recovery after he suffered a near-fatal stroke four years ago, the former Orange Juice frontman and Edinburgh homeboy revels in one of his best years to date. Assembly Hall, 20-22 August

Michael Jackson at the Gates of Heaven and Hell

The King of Pop has scarcely hung up his spangled glove for the last time... when he finds himself the subject of this satirical Fringe premiere. Underbelly, 17-30 August

Actus Tragicus

Six Bach cantatas brought together by the late German director Herbert Wernicke in an ambitious danse macabre of the lives of the lonely, set inside a cross-section of a four-storey house. Festival Theatre, 4-5 September

Pythonesque

The real lives of the creators of 'Life of Brian', dead parrots, Whicker Island and Hell's Grannies, by the playwright Roy Smiles. Udderbelly Pasture, 8-31 August

Michael Clark

The dance-choreographer returns to Edinburgh for the first time in 20 years as he celebrates the work of rock's "holy trinity", David Bowie, Iggy Pop and Lou Reed. Playhouse, 28-31 August

Comedy Club 4 Kids

A real live comedy club with top comics performing specially for children and even giving their hard-pressed parents the occasional (clean) laugh, too. The Bongo Club, 6-31 August

Blondes

Essex superstar Denise Van Outen sings and dances the night away as she reminds us that those of a certain hair colour like to enjoy themselves. Underbelly, 6-31 August

Edinburgh Military Tattoo

Bagpipes, sporrans, kilts, men in big hats marching in circles by torchlight: the annual tartan-fest is in its 60th year. Edinburgh Castle, 7-29 August