This year's Berlin Film Festival gives us a leaden race for the Golden Bear

It's one of cinema's big events – but it is having a bad year. Surveying the contest, Geoffrey Macnab gives thanks for Juliette Binoche

Shia vagary: LaBeouf and Evan Rachel Wood in the incoherent ‘The Necessary Death of Charlie Countryman’

The 63rd Berlin Film Festival has been a curiously muted affair. Critics have been grumbling about a competition that hasn't yet delivered much excitement. Well over midway into the two-week event, the only contender for the festival's Golden Bear prize that had really excited the reviewers and the marauding packs of distributors was Gloria, a Chilean feature about a 58-year-old divorcee looking for love.

Wong Kar-Wai's long-gestating martial arts movie The Grandmaster (which opened the festival) was well liked but was showing out of competition and wasn't a world premiere. Richard Linklater's Before Midnight was likewise out of competition.

Jaws dropped at some of the films that were contending for the Golden Bear. The Necessary Death of Charlie Countryman perplexed many critics. The debut feature from Swedish advertising wunderkind Fredrik Bond was as slickly shot as any Benetton ad but was undermined by a screenplay that verged on the deranged. Shia LaBeouf stars as a young American who heads to Romania on a whim after the death of his mother. In short order, he falls in love with a musician (Evan Rachel Wood, adopting an absurd Eastern European accent), provokes the wrath of her gangster boyfriend (the excellent Mads Mikkelsen) and takes drugs with Rupert Grint while staying in a hippie youth hostel.

What was most startling about Charlie Countryman was its utterly random quality. Bond (working from a screenplay by Matt Drake) seemed to be aiming for the freewheeling energy that characterises old Nouvelle Vague films from Godard and Truffaut. There is lots of handheld camera as we see LaBeouf running wildly through the streets of Bucharest or having phantasmagoric experiences in nightclubs. The film's view of Romania is patronising in the extreme. The jokes about Eastern European Viagra don't help and the voice-over from John Hurt does nothing to make matters more coherent.

There were several films in competition about incarcerated women. The most austere was Bruno Dumont's Camille Claudel 1915, starring Juliette Binoche as the artist and former lover of Rodin. She has been confined by her family to an asylum. Dumont depicts her daily life in grim detail. She refuses to wash so the asylum nurses force her to bathe. Her only freedom is that she is allowed to cook her own food. Camille seems far more sane than many of her fellow inmates but when her brother, the writer Paul Claudel (Jean-Luc Vincent) comes to visit, we begin to realise why she has been incarcerated. The pride and independence that made her into an artist have eaten away at her sanity.

Dumont tells Camille's story in a deadpan and detached way. This is more like an anthropological study than a conventional drama. Binoche portrays her with great subtlety, capturing both her feral quality (the sense that she's an imprisoned animal) and her sensitivity. We also gradually begin to realise that Camille is a monstrous egotist. This is a forbidding and difficult film to watch but its formal rigour and seriousness come as a relief after the flippancy of Charlie Countryman.

Guillaume Nicloux's The Nun, adapted from Denis Diderot's 18th-century novel, was also a study of a woman imprisoned against her will. The film unfolds like a very dark version of Cinderella. Pauline Etienne plays Suzanne, a young woman from an aristocratic family fallen on hard times. She has been placed in a nunnery because her parents can't afford to pay her dowry. Her married sisters have no desire to help her. Shot handsomely by Yves Cape, the film recounts Suzanne's attempts to stay afloat in a hostile, humiliating world. The film was respectfully received although distributors grumbled that the storytelling faltered with the introduction of an older nun (Isabelle Huppert) with Sapphic desires in the second half.

Alongside the “official” films screening in competition, there have been hundreds of films showing in the Berlin European Film Market. Ron Howard was in town to show distributors a near completed version of his Formula One movie Rush, which tells the story of the rivalry between drivers Niki Lauda (Daniel Brühl) and British playboy James Hunt (Chris Hemsworth.) The press was kept out of the screening but the buzz from the distributors was very positive.

One of the stranger titles available in the market was Paul Verhoeven's latest effort, Tricked. Made for the small screen, based on a plot that the public helped write as part of a reality TV show, this played like a very overwrought soap opera. Its main character is a sleazy middle-class businessman (Peter Blok), who is being blackmailed by a former mistress and is carrying on an affair with his daughter's best friend. It didn't have the visual sweep and flamboyance that you find in Verhoeven films from Soldier of Orange to Basic Instinct but there are a few of the schlockmeister's familiar touches.

Many of the films screening in competition and in the market this week are destined never to receive mainstream distribution and will never be seen by big audiences. That is not the case, though, for young Swedish director Simon Klose's TPB AFK: The Pirate Bay away from Keyboard, a feature doc about the founders of the notorious file-sharing website. Klose made the film (whose backers include the BBC) available online for free at exactly the moment (5pm last Friday) that it began its official screening in the festival. Since then, the film has received more than a million hits. This isn't going to make Klose rich. (He told me the doc had earned around $30,000 from online donations.) Nonetheless, at a Berlin Festival full of esoteric and marginal fare, this is at least one movie that already has a life beyond the festival.