Not even Jim can fix it now

With ambitious wannabes managing its lacklustre campaign, Sir James Goldsmith's Referendum Party now has more to do with personal gain than political idealism

The Referendum Party has a new secret weapon. Pagers. Malcolm Glenn, the party's managing director, who also has a mobile phone clipped to his trouser braces, points at the black box at his waist: "Any time Sir James is about to be on TV or the radio

Shrimsley smiles a little at the admission. "I've only come on board recently," he says. His office seems barely unpacked: just a plain desk, a trail of telephone wires, and an open copy of the Daily Telegraph. The rest of the Referendum Party headquarters is much the same. Mid-morning, the dim corridor of rooms mutters rather than bustles. Envelopes are stacked, not scattered. Small party posters hang in dark corners. All four dozen staff are at their desks: young, well-bred heads over their keyboards, the women thinner than the men, typing for "JG" as if his party was a medium-sized charity. They do not offer handshakes.

Yet Shrimsley conducts the tour apparently unperturbed. "It's appropriate that we've got such a small set-up," he says, "because we're a narrow- issue party." And there is method in his meander through the worn hallways. Shrimsley is an old Fleet Street hand; he leads the visiting eye to the modest, the discreetly enthusiastic. "You don't want to bore everyone to death," he says. The Referendum Party, pounds 20m strong, the creation of the cartoon tycoon to end them all, is quietly adjusting its ambitions downwards.

Listening to Sir James Goldsmith last week, pouring out his answers on Breakfast With Frost, pounding callers on a phone-in on Radio Five, you might not have thought so. Behind this bluster, though, election day expectations are lowering. "Judging it on seats is not going to be very helpful," says one party manager. "We've got candidates who really believe they can do it. We'll give them all the support we can. But history is going to judge by the total vote. If we get a million votes [1,500 in each constituency], it will be respectable."

On Radio Five, the inevitable question about polling day quietened even Goldsmith. Then he answered, with what might have been a nervous quaver: "I've not the slightest idea." This may prove to be wise. It is very difficult to find a political analyst currently pondering the Referendum Party at all, let alone any possibility of their success. "As their impact is likely to be limited and temporary," says Ian Budge, a professor at Essex University's department of government, "no one who studies politics is terribly interested in them." David Denver, an election specialist at Lancaster University, agrees: "They've got nobody on the ground to do the street-fighting, and they can't buy up television airtime like Perot or Berlusconi."

Instead, more dispiriting precedents suggest themselves: Lord Beaverbrook's Empire Crusade, blowing itself out in the St George's Westminster by-election of 1931, or Oswald Mosley's swaggering crowd dwindling down, later that decade, to a huddle of taxi drivers and ex-colonels. Earlier this month, Goldsmith's son-in-law, Imran Khan, led a populist surge of his own to annihilation in the Pakistani elections. And for all the impressive-sounding poll ratings among "might-dos" and "maybes" that the Referendum Party presses on journalists, the figure that newspapers have given most space to - MORI's estimate for committed Goldsmith voters - is an embarrassing one: 0.5 per cent.

Some of this is not his party's fault. It is a potent and enduring truism of British politics that our system of disproportional representation crushes small parties. The process is most thorough at general elections.

Over the last six months, however, the party has blundered with sufficient regularity to risk collapse in any system. It has included in its membership list reporters who expressed a merely professional interest; failed to gag a defecting campaign manager who denounced "a nothing party ... run by political amateurs"; changed press officers three times; even attempted to smother bad poll figures by ordering the agency it hired to angle the questions. While Goldsmith has been striding through television studios, and billboards have been bellowing, and pro-Referendum newspapers floating down on to every doorstep, his party has simultaneously seemed shifty, defensive, fond of legal heavy breathing. It has seemed to be falling apart.

A contrast has been widely drawn between the confident whirl of the Referendum Party conference, in Brighton last October, and these more recent confusions. Then, the Sunday Times could gush, giddy with the activists' glitz, that "the whole party is part of the Goldsmith family". Now, candidates withdraw to the small sound of gloating.

But the party's weaknesses run deep. From birth, it made more noise than its actual strength or attractiveness to voters merited.

Last spring, for all the Conservatives' anxieties about the Referendum Party snatching away seats, Goldsmith had virtually no one to stand in them. With the election at most a year away, necessity made careful scrutiny of potential candidates and party organisers impossible. In May, the party recruited Andy Carmichael as its campaign manager for the West Midlands. "In the interview I was quite surprised," he says. "They said, 'We have to watch out for right-wing extremists', but they never asked me what I had been doing between 1991 and 1996." He had been standing in by-elections for the National Front.

Carmichael was given a mobile phone and mileage allowance. He started hiring colleagues from the National Front (now the National Democratic Party) as Goldsmith's local agents. "The Referendum Party knew nothing about electioneering, like the logistics of sending in nomination papers," he says. The potential candidates that came forward were no more experienced: "You'd get people calling up who wanted to make names for themselves and knew Goldsmith had the money."

At headquarters, a different kind of amateurism reigned through the summer. It was the holidays, and "all these Henry and Camilla types were going on about organising parties", says one manager they irritated. Conservative Central Office was just round the corner, but Referendum Party volunteers bragged in the pub regardless. "Every time there was a leak," says the manager, "we'd say, 'Who?' It could have been anyone."

Some of this bumbling was curtailed by October. Half the volunteers were sacked. The rave they had planned for Brighton was cancelled. Carmichael was uncovered and dismissed, following a call from a Birmingham Post reporter. Nevertheless, by October the Referendum Party looked significant. It had its list of candidates: some cerebral, like Alan Walters, Margaret Thatcher's economics guru; some almost famous, like John Aspinall, the zoo owner; many eccentric, to judge by all their connections to far-right causes in Africa, but just as many appealingly ordinary, with jobs in social services and estate agents. And then there was the conference itself, attended by 4,000 members; Goldsmith's "rabble" was marching.

The trouble was, not all of his generals were quite as enthusiastic. "The majority of headquarters staff are disaffected Tories," says one former employee. "Everybody knows that the Referendum Party is going to fold in May; they want to damage the Tory party, but not so much that there's no way back."

At first, this closeness to the Conservatives gave Goldsmith's party much of its leverage. Officially, it was and is neutral on the question of Europe, other than to ask for a popular vote on the character of the European Union. In reality, the Referendum Party has always been, in its personnel and targeting of voters, a particularly rebellious Eurosceptic faction of the Conservative Party - hence its flirtations with MPs like John Redwood and Bill Cash.

But if it has been given life by Conservative intrigue, it can die by the same turmoil. The leaks, the infiltrators, the paranoia and fear of putting things on paper - many of the practices that stifle headquarters derive from the dominating presence of Central Office. The lure of advancement in Goldsmith's business empire - seen as dependent on his personal favour - is another force for instability. As one former staffer puts it: "The Referendum Party is not the main consideration for anyone significant in the party."

Then again, this temporary vehicle still has to be rolled to the election. Bernard Shrimsley proffers a video that is being sent to party workers to spread the word "on a tupperware basis". Regional offices are still being opened. Sir James will become more visible: "It's like Hamlet without the prince when he's not here," says Shrimsley.

Yet Goldsmith is unlikely to be enough on his own. His Boeing 757 may be ready for a national tour, but on Radio Five last week, his answers were hectoring, unattractive. In reply to a critical caller, Goldsmith began: "You're talking complete nonsense, of course." Election day is likely to turn another of his blurted assertions back on him: "People have greater wisdom than the political elites."