The Imaginarium of Doctor Panassus, Terry Gilliam, 122 mins (12A)

There are moments in the new Terry Gilliam movie when his florid animations burst out of nowhere, reminding you of those giant trompe l'oeils he famously conjured for Monty Python.

It is a shame that Gilliam didn't settle for being an animator, because these squiggles of invention are so much more his forte than film directing. Whatever cult status he's been living off since Brazil (1985) has dwindled drastically in the last 10 years, partly due to misfortune (his cherished Don Quixote film collapsed in bad weather and money trouble) and partly due to a string of duds (Fear and Loathing in Las Vegas, The Brothers Grimm, Tideland) whose only surprise was that somebody was willing to finance them in the first place. Those films had plenty of Gilliam's trademark visual invention; what they absolutely lacked was the smallest notion of how to tell a story. That's quite an important skill for a film-maker to have.

The title of his latest does not encourage us to hope that Gilliam has learnt it. As soon as you see the word "imaginarium" you think: this will not be short of anything zany, wacky or loopy, but it might be short on anything that makes sense. And so it proves. It opens in nighttime London, where a group of travelling players have set up their tatty vaudeville show. Two eager young souls, Anton (Andrew Garfield) and Valentina (Lily Cole), prance about on stage while a gaggle of yobs hurl insults at them. In the background presides an old chalky-faced tippler and father of Valentina, one Dr Parnassus (Christopher Plummer), whose imaginarium – entered via a tinfoil curtain – leads to an alternate looking-glass world in which initiates choose their own heaven or hell. "Don't worry if you don't understand it all immediately," confides the good Doctor. Thanks for the warning.

The vaudevillian caravan staggers on, in the course of which a vague plot begins to emerge. Tom Waits, bowler-hatted and bow-tied, plays Mr Nick – the Devil, probably – with whom Parnassus struck a deal a long time back. In exchange for his immortality he agreed to hand over Valentina when she reaches her 16th birthday, now a matter of days hence. But no sooner has this pact been established than Gilliam changes it; Mr Nick agrees to let Parnassus keep his daughter so long as he can claim five souls in two days. What this could mean, or how it might happen, is never made clear. No time to ponder, either, because the film has lurched on to its next situation, one which, in the light of events, carries a tragic undertow.

The players spot a man swinging from his neck beneath Blackfriars Bridge, and rescue him from an attempted suicide. The man, a shady chancer named Tony, is played by Heath Ledger in his final role. An uncompleted role, at that, for the actor died before filming was through.

The odd thing is that the trio of stars Gilliam recruited to fill in for the unfortunate Ledger – Johnny Depp, Jude Law, Colin Farrell – don't look out of place, because nothing about the movie feels in place. The rubber ball of invention requires the stout wooden floor of realism to bounce off, but Gilliam's screenplay hasn't provided any floors. Or walls. Or ceilings. What he does is to devise dreamlike set-pieces, hopping from one to another like a frog to a lily pad.

Under these conditions the actors have to take their chances. Gilliam has admitted that he "allowed more ad-libbing on this film than on anything I have ever done," principally because Heath Ledger was "so full of ideas and fresh dialogue". That is the sort of commendation to make a film critic's blood run cold. The actor had just come from playing the Joker, and actually looks worn out, so Gilliam was either being nice or else is a hopeless judge of fresh dialogue. When Ledger has to do a spiel in front of a bunch of shoppers in Leadenhall Market the effort of it – his being suave, their being charmed – is horribly stilted and false. It's nearly always a terrible idea to let actors improvise in any case; they are there to interpret, not to invent. Andrew Garfield, a bright star in the making (Kid A, Lions for Lambs), struggles gamely with the extra burden imposed on him. You keep noticing dead spots in the script where the actors don't seem to have been properly directed.

Perhaps kids will respond to it more readily than adults, what with the shape-shifting (Tom Waits's head twists from a river in the form of a giant cobra) and the jokey interplay between innocence and experience. Lily Cole does creditably in her role as the coquettish Valentina; Christopher Plummer, enjoying a great "late" period, has the look of a soup-kitchen Dumbledore. Younger audiences might also be more tolerant of the Potterish swoops into fantasy which the Imaginarium enables, and they probably won't want to miss Ledger in his final bow (I'd much prefer to remember him for Brokeback Mountain and Monster's Ball).

It says something for the director – I'm not sure what precisely – that stars of Ledger's calibre want to work with him. But then Terry Gilliam's whole career seems mysterious to me. Nobody would doubt that he's possessed of an extravagant imagination. It's just unfortunate that what he imagines is very, very boring.