£375 for Jeremy Clarkson, £655 for JK Rowling: the private eye's lucrative trade

Vast range of activities revealed in second part of Independent investigation

The tabloid press hired private detectives to investigate Catherine, the Duchess of Cambridge, when she was still a student in Scotland, and Chelsea Clinton, the daughter of the former US President, when she was studying at Oxford. The same detective agency was hired to investigate Philippa Middleton, the Duchess's younger sister, when she was a teenager, and to carry out a "blag" for information on the hen and stag parties of Prince Edward and the Countess of Wessex.

The Independent has examined files seized as part of Operation Motorman in 2003 and been told by the lead investigator on that inquiry that his team were forbidden from interviewing journalists from a wide range of media organisations who hired a private detective agency to track down personal information. More than 17,000 searches were carried out, many of them in breach of data protection laws.

In a signed witness statement given to this newspaper, the former police detective inspector who led Operation Motorman, accused the authorities of serious failings. "We weren't allowed to talk to journalists," said the investigator, who was working for the Information Commissioner's Office (ICO). "It was fear – they were frightened."

The files show that JK Rowling, who was named yesterday as one of the core participants in the Leveson Inquiry into media standards, was targeted by the private investigator Steve Whittamore in the summer of 2000, at the time that Harry Potter and the Goblet of Fire, the record-breaking fourth book in the Harry Potter series, was published. The private detective charged £655 for his unspecified inquiries.

Whittamore was asked to obtain confidential information on the Duchess of Cambridge in April 2002, when she was known simply as Kate Middleton, a 20-year-old student at St Andrews University who had been linked with her fellow student Prince William. The "yellow book" file, one of four colour-coded A4 books kept by Whittamore detailing his business dealings with journalists, reveals that the private eye was given her mobile phone number and asked to locate her family home. He was then required to provide the "Family & Friends" numbers for the Middleton home address in Berkshire and returned with 10 numbers. He charged £500 per number to obtain, using those details, the names and addresses of the family's close circle. The book then records separate inquiries for "Catherine Middleton" and her 18-year-old sister Philippa, known as Pippa. Both jobs are listed as separate Family & Friends searches, presumably made on their mobile numbers. Some of the work was recorded in the files as being carried out by one of Whittamore's associates, a former Hells Angel based on the South Coast.

At the time, Kate Middleton had just come on to the media's radar after appearing at a university charity fashion show in a sheer dress that revealed her underwear. Prince William, who attended the fashion show, had previously played tennis with his fellow student and speculation focused on his future living arrangements. In total, the operation to target the Middleton sisters would have involved more than 50 breaches of the Data Protection Act 1998, Section 55, which covers the unlawful disclosure, procurement and selling of personal information contained within a database. Each time a number is passed on or converted, another breach of the law is committed. To justify its actions, a newspaper would have to demonstrate that its inquiries were in the public interest.

Whittamore was also hired to conduct something he listed as the "Hen & Stag Blag" ahead of the marriage of Prince Edward to Sophie Rhys-Jones in 1999. The luxury Lanesborough Hotel near Hyde Park, which hosted both the hen and the stag party, was targeted by the private investigator to obtain details of the events. A report later emerged comparing the two parties and giving details of guests, cocktails and canapés.

Chelsea Clinton was subject to a Whittamore investigation while studying for a Master's degree in international relations at University College, Oxford. In October 2002, the investigator was hired to locate Ms Clinton and her boyfriend Ian Klaus, a Rhodes scholar at Jesus College. Searches focused on Jericho, a historic neighbourhood of the city. The private investigator was then hired to conduct something he recorded in his files as the "Peak Fitness Blag", which is understood to have been an attempt to procure information from the Oxford gym where the couple were members.

Jonathan Ross was investigated in 1999 when Whittamore's company, JJ Services, demanded a fee of £480 for inquiries which were simply headed "wife". At that time, the tabloids were reporting that the couple had briefly split up and that the television presenter's wife had been suffering from depression. Jeremy Clarkson, now a tabloid and broadsheet columnist himself, was targeted in 1997 for £375-worth of private information as he became a rising star on the BBC's Top Gear and details were emerging of his wild times as a public schoolboy.

Papers asked the detective agency to access the Police National Computer in order to satisfy their suspicions about the pasts of popular entertainment stars. The singer Ms Dynamite, at the height of her fame after winning the Mercury Prize in 2002, was promptly made the subject of a criminal record check, as was her manager, Desmond George. Also checked for a possible criminal record was Sada Walkington, a housemate in the Channel 4 reality show Big Brother who was characterised as a hippie with a taste for yoga and eating tofu.

Peter Salmon, the head of BBC North and currently being tipped as a possible future director-general of the BBC, was the subject of a Whittamore operation dubbed the "Belair Beverly Hills Blag", after the former Granada television executive's marriage break-up and subsequent relationship with Coronation Street actress Sarah Lancashire (who was herself the subject of a £130 Whittamore invoice). Other Whittamore blags on various targets included the "Montego Bay Blag", the "Hillgrove Cat Farm Blag", the "Equity Blag", the "Amanda Barrie Blag" and the "Dawn's Massage Parlour Blag". The heart surgeon Sir Magdi Yacoub was subject to an operation described by Whittamore as "2 blags and find about". The jockey Kieren Fallon, whose private life was under investigation by the tabloids in 1999, was the subject of the "GPO Blag Fallon" and a series of ex-directory searches.

The former Mayor of London Ken Livingstone was the subject of an invoice from JJ Services, first under the heading "love child", when rumours of his earlier children surfaced in 1999, and then in 2002 under the heading "Com Cab blag". At the time, in 2002, the mayor was being criticised for his use of taxis and the Greater London Authority had an account with a firm called Computer Cab. The BBC Match of the Day presenter Mark Lawrenson was the subject of at least three vehicle checks and a bizarre attempt to discover if the former Liverpool defender had any business dealings with the television presenter Lionel Blair. Whittamore was asked to conduct ex-directory searches for various papers on such well-known figures as Jeremy Paxman, John Cleese, Elizabeth Hurley and Linford Christie. The award-winning charity worker Camila Batmanghelidjh, founder of Kids Company, was similarly targeted.

One paper spent around £2,000 investigating the actress Tamzin Outhwaite, obtaining her ex-directory number and the names and addresses of the six numbers listed as her Friends & Family. Whittamore's company received £250 for its inquiries into Sir Paul McCartney, £300 for an investigation into the "ex" of actress Anna Friel, £250 for a probe into a "hotel" associated with Mick Jagger, £120 for a job linked to the "daughter" of John Major and £250 on the "exes" of David Beckham. Some "research" on the comedian Frank Skinner generated a further £67.50.

Wayne Rooney was a money-spinner even before Whittamore was raided in 2003. Whittamore was asked to trawl addresses in the young footballer's Liverpool neighbourhood of Croxteth for numbers of relatives and friends.

The vast amount of Whittamore's work came from the tabloids but broadsheets also used his services. One title hired him to make inquiries into bank payments as part of an investigation into renovations at the home of a government minister. The private investigator came back with details of a private company's bank balance and overdraft facility. Television companies were also among Whittamore's clients. He lists business transactions with two television programmes and with a Wales-based production company.

The Information Commissioner Christopher Graham defended the ICO's Motorman investigation. "The ICO has always been clear that our decision not to pursue legal action against any of the journalists linked to the Operation Motorman investigation was based on a lack of evidence that the journalists who had received information from Mr Whittamore had directly asked him to obtain the information illegally," he said. "Without this evidence the ICO could not justify chasing every possible prosecution as this would have taken a disproportionate amount of time and resource and was unlikely to lead to any meaningful results."

The Whittamore investigation

Steve Whittamore was grilling sausages at his family bungalow near the New Forest when the knock on the door came in 2003. The stocky private detective in his mid-Fifties was spending a rare moment outside of the small office he had set up between his home and garage, a room crowded with paper work.

On the top of a cabinet were a series of hardback A4 note books, colour-coded blue, red, green and yellow. They contained details of his work for journalists working in national newspapers, magazines and television.

When investigators arrived from the Information Commissioner's Office, in the company of a couple of local police officers whose services were not required, Whittamore was amiable but quite adamant from the outset he would not talk about the journalists who were his clients.

But the files that were seized from his home offered unprecedented insight into the media's secret dealings with private detectives, revealing for the first time a vast, lucrative trade in illicit personal data.

The ICO investigation had begun into two DVLA employees who were selling the private details of registered car owners to private detectives and others, hence the name: Operation Motorman. The evidence led back to Whittamore. The private eye was shown a warrant and an ICO search team accompanied by an independent forensic computer analyst entered his home in New Milton, Hampshire.

The investigators later began to uncover how Whittamore's operation ran, unravelling his network of associates. Whittamore himself was a seasoned 'blagger': adept at conning information out of the unwitting. But if he wasn't able to get something himself, he had sources who could.

One of the most prolific was an ex-soldier and Hells Angel in Sussex who specialised in posing as a phone engineer to prise confidential records out of phone companies.

For criminal records, Whittamore would contact a third party who in turn would contact Paul Marshall, a civilian police worker in south London who trawled police databases for payment. If there was a 'blag' he couldn't do himself, he might turn to another private investigator, John Boyall. Whatever it was - vehicle registration numbers, bank statements, tax records – somewhere along the chain someone could get them.

The ICO investigators started pulling together 30 bundles of evidence – linking Whittamore's invoices to named journalists and their victims, whose privacy had been breached.

They wanted to bring a conspiracy charge against both the journalists and the private detectives. Three reporters were spoken to by the police investigating the illegal accessing of criminal record databases but the ICO officers never spoke to a single journalist.

In April 2005 Marshall, Whittamore, Boyall and a retired policeman called Alan King were given conditional discharges. Whittamore and others still faced charges from the Information Commissioner, but within two months they had been dropped.

Mark Olden and Ian Burrell