Joe McGinniss, a master of reportage, whatever the cost

For the American journalist and author, writing a book required total immersion in the subject – even if that meant becoming part of the story himself

Undercover: Joe McGinnis bought a home in Alaska, next door to Sarah Palin, so that he could be close to her while writing his biography

Americans are supposed not to "get" soccer, the game the rest of the world calls football. That is one reason why the passing of the American journalist and author Joe McGinniss demands notice. For, amazingly, he gave us one of the best books about football ever written. But it's not the only reason.

He also produced arguably the most prescient piece of reportage on American presidential politics in the past half-century, not to mention a true-crime classic that many would place in the company of In Cold Blood – and which proved a dozen times more controversial than Truman Capote's masterpiece.

Yes, the coda of McGinniss's career may have generated not glory but near ridicule, with The Rogue, a 2011 biography of Sarah Palin memorable less for the acuity of its observations about the former Republican vice-presidential candidate than for the fact that McGinniss ostentatiously moved next door to the Palins in Wasilla, Alaska, while he was gathering material for the book.

For Todd Palin, Sarah's husband, the whole thing was no more than stalking, an exercise "to satisfy his own creepy obsession with my wife". But that was the McGinniss way. Writing a book, he claimed, required total immersion, physical as well as intellectual, in the subject's world. And if that led to extra publicity, he wasn't going to object. More to the point, the formula worked.

McGinniss was a 26-year-old columnist on a Philadelphia newspaper when he wrote the book that made him a sensation: The Selling of the President 1968, on the great makeover of Richard Nixon from the charmless and rather sinister figure who had lost to JFK eight years earlier. Somehow he talked his way into the Nixon inner sanctum, to produce a first-hand account of how manipulators and spin doctors were taking over presidential campaigns.

It was an uncanny foretaste of the future, of today's vastly expensive exercises that turn candidates from normal humans into performers who never vary from a script, marketed like a cereal or detergent. The book was a smash; McGinniss became the youngest living author ever to top the US non-fiction bestseller list. Throughout he never hid his distaste for Nixon.

In his next blockbuster, McGinniss was truly part of the story. The Jeffrey MacDonald case remains one of America's most lurid 20th-century crime mysteries, revolving around a former US Army doctor whose wife and two young daughters were knifed and beaten to death in 1970, in the family home at Fort Bragg, North Carolina.

MacDonald, who was injured in the episode, claimed the crimes were a Charles-Manson type rampage carried out by a group of drug-crazed hippies who had burst into the house. But first military, then civil, investigators came to suspect him, and in 1979 he finally went on trial for murder. By then he had become friends with McGinniss, whom he persuaded of his innocence.

The two struck a deal for McGinniss to write a book about the case, that would presumably exonerate him. But as the trial progressed, and the evidence piled up, McGinniss gradually changed his mind, concluding that the doctor was indeed guilty as charged. But he kept those doubts from MacDonald, maintaining their close relationship even after the latter's conviction. Then Fatal Vision appeared in 1983, in which the relationship between author and his subject was a central strand in a complex, riveting narrative that depicted MacDonald not as victim, but as a psychopath and multiple murderer.

MacDonald sued for breach of contract and eventually the two sides settled out of court. But McGinniss steadfastly defended his conduct, claiming that even if he had dissimulated, he was entitled to do so because the evidence showed – and the jury's verdict confirmed – that the doctor had lied to him throughout. But that was only the start of the controversy.

In 1989, the writer Janet Malcolm published a blistering piece in the The New Yorker, entitled The Journalist and the Murderer, claiming that reporters were no more than cynical manipulators who wormed their way into the trust of their victims, only to betray them. "Every journalist who is not too stupid or too full of himself to notice what is going on, knows that what he does is morally indefensible," the essay began. Exhibit A for this thesis was Joe McGinniss.

The argument rages to this day. MacDonald, aged 70, is still in prison, still insisting on his innocence, as have two other subsequent books on the case. As late as 2012, DNA appeals were wending their way through the courts. And for some, McGinniss, wounded by the Malcolm piece and the firestorm that ensued, was afterwards never quite the same writer. If you believe that however, you cannot have read The Miracle of Castel di Sangro.

It's about a town of 5,000 people in the poor and mountainous Abruzzi region, and its makeshift football team that somehow wins promotion to Italy's second highest professional league, just one rung below such titans as Juventus and AC Milan. The book recounts Castel di Sangro's first season in Serie B and the moving, sometimes hilarious and always human struggle to stay there and avoid relegation. It is a tale peopled with rogues and innocents, lions and lambs. The ending is sweet, then suddenly and stunningly bitter.

McGinniss was apparently won over to football by the 1990 World Cup in Italy. In a distant past he apparently wanted to be a sports reporter. Quite how he came across the Castel di Sangro story, he never really makes clear. But the story is all the better for his being a central part of it: "l'Americano", the illustrious journalist who, to the delight of his newfound Italian friends, had intended to write a book about the world-famous trial of OJ Simpson, but instead crossed the ocean to chronicle the adventures of a football team no one had ever heard of. The result was a classic – and not the least reason Joe McGinniss will be missed.