Photography: One big dysfunctional family: Andrew Palmer on 'Who's Looking at the Family?', a show giving star billing to the controversial pictures of Sally Mann

Who's Looking at the Family, at the Barbican Centre, London, is not so much a 'look' at the family as a sidelong glance. From whatever angle you take it, the selection betrays that inevitable urge to swim against the stereotypical tide - you'll pass images of humans in cages, drunken parties, victims of Alzheimer's, people dying, still lifes of a dinner table, even clouds. So when you do come across a small section containing a family album it is positively shocking.

Tommy Thompson's family snaps are as conventional as Sunday afternoon tea. Tommy, his wife and his twin daughters, June and Hilda, plod happily through the usual rites of passage. On holiday in Blackpool. June and Hilda with their pet rabbit. First day at school, and so on. It is doubly shocking, then, when it turns out that behind all these smiling faces lies a history of abuse and murder.

Hilda and June were continually subjected to violent and sexual abuse by their father until the spring of 1988, when they shot Tommy dead. The chances are that the viewer will not discover this story until after having seen the pictures (the text which relates the events is hung on a different wall). The arrangement is designed to provoke, and it works: not so much because it pronounces the family to be in a bad way, but because it warns the viewer not to read too much into the photographs they see. The Thompson album is there to stuff the old adage that the camera never lies. When it comes to domestic harmony, the camera can be at worst a deceptive, at best an ineffective medium of communication.

It is a lesson that some others in the show would do well to ponder. For some reason, the works of those who have tried to access the family through the piling up of seemingly incontrovertible physical evidence tend to be the least illuminating. We gain little insight into their respective territories, for instance, from Thomas Ruff's interiors (details of bedrooms and living-rooms); or Anna and Bernhard Blume's Kitchen Frenzy (a series about potatoes being chucked around a room, see illustration); or Robert F Hammerstiel's Midday Portraits (pedestrian shots of place settings in Germany).

Place settings on paper could be very eloquent, of course. There may be any number of interesting cultural inferences to be drawn about diet and eating rituals with respect to class, caste and religious belief. But in the same way that the biological classification of human groups in itself tells us little or nothing about the cultures and background that those groups inhabit, a picture of two burgenwurst on a plate tells us nothing more than that German food can be unappetising.

There is much else in the show which does not flirt so coarsely with the pseudo-sociological. It has nice touches - pictures of humans taken by apes, Martin Parr's jigsaw puzzle of couples, Nick Waplington's documentation of riotous tenants on a Nottingham council estate. And then just some nice ideas, like Linda Duvall's Babies That Look Alike (see illustration), a series of giant enlargements of entries in the birth notices column of the Canadian Windsor Star.

Duvall's work humorously probes the emotional mine-field of parenthood - suggesting that, while all parents think their children are unique, they are in fact the same. At a first look, they do all seem the same (the poor quality of the original newsprint is certainly a great leveller). It is because of their similarities, though, that we are drawn to the notices beneath - each family's proud declarations of their own little miracles. 'A little sprout for Vincent and Carol Pelkey', 'Ecstatic 1st time grand parents are Ken and Marge Brown', 'Waiting to spoil their first niece are Aunt Kristin, Helen, Marla . . .' and so on.

What really unites the images is the inadequacy of words to express the momentousness of birth. These pictures convey the simultaneous portentousness and mundanity of new life. Each image is a declaration of hope - a hope which every family experiences, and every family is apt to betray.

Duvall has perhaps inadvertently struck at the heart of what family photography is all about. Just as babies look alike, family portraits should look the same. By definition, photographed families are happy families. We do not tend to set out to expose the negative aspects of an institution riddled with tensions and instability. Each family album is, in its own way, an exercise in conformity. A point well executed by Alexander Honory's magnificent montage of a collection of christening portraits, all taken by the same high-street studio, which he picked up at a Cologne flea market.

No wonder, then, that Sally Mann's family album has ended up dominating this exhibition. Mann's pictures are presented as a diary of her young children growing up in rural Virginia. This may be the case, but they are far removed from the conventional logging of a child's development. Her children inhabit a dark world laced with sexual violence and masochism. In Dirty Jesse, Mann's daughter, aged three, provocatively pinches her nipple, wearing an expression of challenging allure. The Wet Bed, of her other daughter lying asleep naked on a bed, is an image which calls to mind the aftermath of a rape rather than a slumbering dreamland. And what to make of The Terrible Picture, a mud-caked girl apparently hanging, a motionless victim, from a tree?

The photographs are beautifully printed - the black-and-white tones can be so overwhelming that they distract the eye from the ongoing scenarios. Nevertheless Mann is playing with fire. In America she has been branded a pornographer and worse. And given the experience of some photographers in this country, where the laws governing the rights and wrongs of photographing children naked are so ambiguous (witness the fate of the child portraitist Ron Oliver, currently living in Holland with a prosecution pending from the Obscene Publications Squad), Mann should count herself lucky for being given star billing at a top London gallery.

One suspects that, were she taking pictures of other people's children, and were she a British man rather than an American woman, she would have been given a very rough ride. (Oliver's pictures are frankly tame in comparison.)

Still, star billing is what she's got - you almost gain the impression that the organisers, who hail her as the show's 'centrepiece', decided Mann's work was so piquant that they needed to pad it out to take the edge off her bite.

Wherever there is taboo, of course, there will always be an 'artist' waving a flag for subversion. Mann has staked her small claim in this venerable tradition. Some will argue that it is a legitimate pursuit to explore child sexuality - why pretend that children are not sexually aware? Others will condemn her for turning her children into sexual objects.

Either way, the route she has taken is not very dissimilar to the one which ended in Tommy Thompson's death. Like Mann, Thompson compiled his family album for propaganda purposes. His was a statement of conservatism, not radicalism - the family album replete with happy faces was there to prove to the outside world that everything was all right, that domestic harmony was intact. But his children, like Mann's, have been manipulated for political ends. And this is the crux: while one would not wish to dispute Mann's claim that her pictures were 'gifts' to her from her children, we know from Thompson's album that the photographic evidence counts for nothing.

At the Barbican Art Gallery, London EC2 to 4 Sept (071-638 4141)

(Photograph omitted)