Once upon a time in the marketing department...

The global success of the Harry Potter books has been helped enormously by adult fans. This unexpected audience may be caught up in nostalgia, but now cynical publishers are pushing 'kid lit' as a crossover phenomenon. Will this new breed of novel captivate us all? Boyd Tonkin reads between the lines

In 1979, the late playwright Dennis Potter wrote his unsettling TV drama Blue Remembered Hills. A group of adult actors play pre-teen children, sporting bare knees and dishevelled uniforms, sudden tantrums and broody sulks. The cast do not impersonate kids, in the manner of past-it Hollywood eye candy desperate to defy the calendar. Rather, they revert to childhood while remaining fully adult, superimposing one stage of life on another. Nostalgia shades uncannily into psychopathology.

Dennis Potter grew up in the Forest of Dean, that lovely but lonely stretch of Gloucestershire that rolls down to the river Wye. Only one other leading post-war British writer shares those local roots. Unlike the challenging dramatist, Forest born and bred, she moved to the area at the age of nine. Later, she would create her own Potter, one who charmed a global audience instead of upsetting a national one.

Yet Joanne Rowling, a former pupil of Wyedean Comprehensive, has done more than enchant the planet's children on the way to her estimated £220m fortune and the 140 million-odd sales of her four children's novels. She has also instigated a trend in publishing that seems able to thrust countless adults back into the imaginative equivalent of those short trousers, knee socks, gingham frocks and Alice bands. Had he not succumbed in 1994 to the cancer he christened "Rupert" (after Murdoch), now would be the perfect time to quiz Dennis Potter about Harry Potter.

First, her apprentice wizard's escapades took off as cult reading for grown-ups. When it grew clear that JK Rowling's books had won legions of fans above the voting age, her publisher Bloomsbury re-issued them with tasteful "adult" covers. In retrospect, the firm need not have bothered. Such was Harry's heft that the smart executive lost in a gaudy-jacketed children's edition became as common a sight on commuter trains as brollies and brief-cases. Professor David Harvey, the classicist who helped secure Rowling an honorary doctorate from the University of Exeter (where she studied), spoke for an army of unapologetic followers when he said: "I am 61 and I enjoy the books as much as children."

Not long after, the more strenuous theological myths spun by Philip Pullman in the trilogy His Dark Materials made a passion for current children's writing intellectually watertight – a process crowned by Pullman's victory in the Whitbread book awards. Elsewhere, the relentless, movie-swollen sales of JRR Tolkien's The Lord of the Rings are matched by the solid biannual success of Terry Pratchett's comic Discworld fantasies, adored by middle-aged mothers as much as by their spotty kids. All this demographic promiscuity has helped to unleash a new wave of "kid lit" novels that aim to ensnare young and adult readers alike.

Is this vogue for "crossover" fiction the sign of a culture full of jaded Peter Pans, eager to blot out the heartache of post-adolescent life in an orgy of infantile regression? Or should we welcome it as a renewed recognition that fine children's fiction can blend peerless storytelling skills with grand ideas and a timeless wisdom that scorns limits of age?

Several publishers have now decided to make a commercial stand at the crossroads where (they hope) younger and older readers meet. No novel this autumn arrived with a louder fanfare than Across the Nightingale Floor. This medieval Japanese quest-tale was written by the Australia-based children's author Gillian Rubinstein under the pseudonym of "Lian Hearn". Sold to Macmillan for about £300,000, with the inevitable megabucks movie deal (above £2.5m), her opening story in a trilogy breaks the "kid lit" mould with its outbreaks of bloody violence and sexual frankness.

"Crossover" titles often appeal to adults because they promise a holiday from the sex-saturated pop culture which, these days, extends down the age scale to the pubescent provocations of groups such as S Club Juniors. Here, though, Hearn/Rubinstein's language may be euphemistic ("our bodies rushed towards each other with the urgency of madness of youth"), but her erotic matter-of-factness keeps faith with the Japanese culture that inspired her book. Prior to falling in love, the young hero Takeo reveals casually that "I knew about desire, had satisfied my own with the other boys of my village, or with girls in the brothel". Somehow, the chaste dorms of Hogwarts seem a million miles away.

Across the Nightingale Floor also marks a new direction by shifting the scene of crossover fiction out of Europe, where it traditionally flourishes. The Lord of the Rings, forefather of the genre, plays mix-and-match with a dozen strands of north European folklore, while CS Lewis's Narnia chronicles stretch out (literally) behind the furniture of the English bourgeois nursery. Meanwhile, musty boarding-school yarns underpin the whole Potter saga, to the dismay of its more puritanical critics.

This Brit hegemony has piqued the patriotic pride of the Pulitzer prize-winning US novelist Michael Chabon. Published this week, his first children's novel, Summerland, relocates the standard formula of magic, monsters and myths to an all-American landscape of baseball lore and Indian legend. It looks as if today's isolationist current in the US has swept even into the fairylands of "kid lit" fiction – a sign of how much grown-up money is at stake. Less chippily, the Chilean bestseller Isabel Allende has plunged into the waters of crossover with a rich Amazonian adventure, City of the Beasts. It fuses ecological fable with faint memories of Conan Doyle's fictional search for a jungle-girt "lost world" and its prehistoric animals.

In bookstores, then, the Potter industry faces new and stiff competition, spawned by its very success. And the newly married author has (to the chagrin of the trade) chosen to savour her own life for a spell rather than gallop to publish the fifth novel, Harry Potter and the Order of the Phoenix, which is still a few weeks away from completion. But Pottermania is about to hit top gear again with the release of Chris Columbus's second film, Harry Potter and the Chamber of Secrets. Its predecessor earned receipts in excess of $600m: a feat (surpassed only by Titanic) that family outings alone cannot explain. Last year, many cinemas bowed to the demographic free-for-all of the Potter fan-base by arranging almost as many late-night screenings as matinées.

Early reports suggest that this second show will be stolen by Kenneth Branagh as the conceited teacher – and fêted author of magical exploits – Gilderoy Lockhart. In the novel, Lockhart warns Harry that literary success is "not all book-signing and publicity photos, you know. You want fame, you have to be prepared for a long, hard slog". Remarkably, JK Rowling was writing those lines when Harry had netted little more than a meagre £1,500 advance for The Philosopher's Stone. Yet they still hold true for her. Pictured at the new movie's premiere, she seemed, as ever, to enjoy the public burdens of celebrity about as much as dental work without an anaesthetic. In her impeccably madeover blonde-bombshell guise, she also brings to mind Lockhart's advice to Harry that "nobody wants to read about some ugly old Armenian warlock, even if he did save a village from werewolves. He'd look dreadful on the front cover".

Such canny gags hint at the playful self-consciousness that endears the Potter series to many (chronologically) mature fans. Just as Lewis Carroll did, Rowling's teasing intelligence seizes on every opportunity for parody, pastiche and satire that comes her way. These devices don't soar over the heads of kids (as Carroll's logical conundrums can) so much as connect adults with children as media-savvy consumers exposed to the same range of voices and values.

Rowling's satire reached its zenith in her blistering caricature, in The Goblet of Fire, of the unctuous tabloid hackette Rita Skeeter, all crocodile skin and crocodile tears. That was an act of revenge, of course, but also proof that, through Harry, these books cleverly comment on their author's inexorable rise – and its drawbacks. This sophisticated strategy can tickle the subtlest of grown-up readers while never forgetting that children, too, have a keen ear for the language of tabloids, television and adverts. Publishers who imagine that the key to crossover triumph lies in far-fetched slabs of escapist fantasy couched in solemn "literary" prose and garnished with Christmas-cracker philosophy haven't been paying enough attention during lessons at Hogwarts.

Look at the most enduring titles in the children's canon, and you find that none sets out to speak with a forked tongue; one part aimed at youngsters, while another whispers over their heads at some knowing adult behind. On the contrary: they seek a comprehensive truth within the emotional world of their child characters. The 19th-century American South in Mark Twain's Huckleberry Finn, and the expansionist British Raj in Rudyard Kipling's Kim, stay unforgettably alive because, and not in spite of, the peculiar perspective of their youthful protagonists. Philip Pullman's Lyra and Will, their guiding daemons by their sides, achieve the same empathy among readers for deeply ambivalent states of faith, doubt and disbelief. The crossover moment arrives, if it does, not because grown-ups want to shed their sober mental suits in favour of shorts and gymslips – like some literary answer to the ghastly School Disco craze that now fills the nation's nightclubs. Rather, it's a matter of adult readers (and adult authors) recovering within themselves the special intensity, and the special focus, of the pre-adolescent vision. The aim is not to resurrect a vanished "innocence", but to enrich a fragmented present with a shared past. Every reader in the world, after all, either has been – or remains – a child, with access at some level to the child's capacity for rapture, terror, boredom and excitement. This is, perhaps, the only sort of universality that literature can claim.

It follows that publishers can never reach this point of convergence cynically, as a slick response to market trends. If the alchemy – the magic, if you like – doesn't happen in the writing, then no amount of hype will procure it. So some of those books now optimistically targeted at "all age groups" may end up attracting none at all. For authors, the only safe advice must be to locate what the shrinks would call an inner child, and then lavish on him or her every cunning adult art of invention and narration. Writers can seldom serve, or satisfy, two masters at once.

Coincidentally, an autumn packed with crossover ventures has also seen the publication of a bewitching adult novel that centres on the delights – and dangers – of young people's literature. The bookish 12-year-old heroine of Donna Tartt's The Little Friend views and judges her surroundings through the prism of children's classics, from Treasure Island and The Jungle Book to The Wizard of Oz. Inevitably, these paper tigers prove misleading guides to the crazy world of adults that she must, eventually, join. Aghast at the hormonal turbulence ahead of her, young Harriet fears that "growing up" will mean "the swift and inexplicable dwindling of character" she detects in dull stories about older children.

There, of course, lies the painful transition that looms for Harry Potter and his creator. Perhaps the sternest challenge for crossover fiction comes when its heroes have at last to step across the boundary that marks the exit from childhood. Harry is still teetering on that line, and will do until the next novel appears early next year. In front of him, and all fans of whatever age, stands what Peter Pan would call an awfully big adventure.

KID-LIT FICTION: THE CLASS OF AUTUMN 2002

Sabriel, by Garth Nix (Collins, £12.99)

A young wizard must keep faith with her late father's art when she leaves the cosy, Hobbit-y calm of Ancelstierre for the Old Kingdom. There, the dead roam and dark forces practise out-of-control "Free Magic". First in a trilogy that ticks all the fantasy boxes (and scores extra for its strong heroine) from an Australian author who is already a bestseller in the States.

City of the Beasts, by Isabel Allende (Flamingo, £12.99)

Conan Doyle and the legend of El Dorado meet Greenpeace as a young Californian, Alex, with his intrepid granny and a local girl, Nadia, sets off into the heart of the Amazon rain forest. Real-life wonders among the jungle fauna and flora deftly blend into elements of magic and mystery, as the children help to protect Indian culture and wild nature against its would-be exploiters.

Summerland, by Michael Chabon (Collins, £12.99)

Tolkien does baseball, as the celebrated novelist puts a wholesome US spin on a cast of shape-shifting monsters and eerie little people. In Washington state, Ethan and Jennifer, junior ball stars, join wise old black, Hispanic and Indian elders to defeat the wiles of our mythical friend Coyote, win the crucial game and rescue their family life.

Across the Nightingale Floor, by Lian Hearn (Macmillan, £12.99)

In flight from a family tragedy, young Takeo, of the gifted Hidden tribe, teams up with Lord Otori to seek revenge. On the way, he falls in love, romps through Crouching Tiger-style scrapes and meets his destiny in the castle of the wicked warlord Iida. The atmosphere of medieval Japan suffuses a tough teenage tale filled with betrayal, brutality and desire.