Heroes of Zeroes: Nate Silver, his rivals and the big electoral data revolution

Statisticians' predictions have revolutionised political punditry and basketball alike. How did the big data revolution come about?

Nate Silver shocked and annoyed US newspaper pundits by correctly predicting the outcome of the 2012 US election (Getty Images)

If you listened to the pundits with the loudest mouths in the lead-up to last year’s US Presidential election, you probably thought it would be a close run thing.

If, on the other hand, you read Nate Silver’s blog, then you knew that President Obama had a healthy chance – a 90 per cent chance, in fact – of staying in the White House. Silver, a data journalist who at the 2008 election accurately forecasted the results in 49 states, called all 50 correctly in 2012.

This week, the value of Silver’s predictions became even clearer, when it was announced that the 35-year-old was leaving his berth at The New York Times and taking his blog, FiveThirtyEight, to the ESPN network, where he will oversee a new site devoted to statistical analyses of sports, politics and other fields.

Professor Marie Davidian, president of the American Statistical Association, says she noticed an “uptick” in attention in her subject in recent years, as the so-called “big data revolution” began. “But Nate Silver’s election predictions put a much broader spotlight on statistics. It resonated with people that someone could predict the election perfectly with dispassionate use of data.”

This is the International Year of Statistics, and as Silver’s ESPN deal demonstrates, there’s a growing market for people who can turn complex numbers into nuggets that the public can understand, and professionals can put to use – and not just for politics.

Following his exit, the New York Times public editor Margaret Sullivan revealed in a blog that Silver’s statistical approach to poll analysis had unsettled the pundit class. “A number of traditional and well-respected Times journalists disliked his work,” she wrote. Silver began his career as a baseball statistician, and the scepticism of his elders closely resembled that of the Oakland A’s scouts to the new-fangled strategy of general manager Billy Beane, in the movie Moneyball.

Beane’s method of picking players, which brought the As a record 20-game winning streak during the 2002 Major League Baseball season, was based on the “sabermetrics” system devised by statistician Bill James, whose work inspired people across the sporting world. In the early 2000s, for example, the basketball statistician John Hollinger, now vice president of the Memphis Grizzlies, created the player efficiency rating (PER), a formula to quantify the skills of any National Basketball Association player.

Ben Alamar, professor of sports management at California’s Menlo College, has consulted for several teams. If there’s a superstar analyst in American sports today, he says, it’s Dean Oliver, director of production analytics at, yes, ESPN. “Statistical analysis in basketball and football is a very new field,” Alamar says. “When I started around nine years ago, there was just one analyst working in the NBA. Now most teams have one.”

Oliver, who has worked with both the Seattle Supersonics and the Denver Nuggets, wrote the definitive book on basketball analysis, Basketball on Paper. “He was the first significant analyst employed by a team,” Alamar explains. “His role now is to develop metrics for every sport that ESPN is interested in.”

Dr Jeff Leek, who teaches at Johns Hopkins University’s school of public health in Baltimore, is the co-editor of the blog SimplyStatistics.org. “Everyone has become a statistician on some level,” he says. “Every article you read on ESPN has some statistical argument in it. The price of data has dramatically dropped, as has the price of software that makes it easy to analyse them. So there’s a bigger group of people with access to useful data.”

In Leek’s field of public health, Hans Rosling captured the imagination of millions with a pair of inspirational talks in 2006 and 2007. The 64-year-old Swedish doctor and his Gapminder foundation developed software which turns complicated figures for, say, global health or economics, into interactive graphics. “He’s like Nate Silver, in that he can distil down relatively complicated ideas into understandable, digestible bits,” says Leek. “I consider him a rock star in the field.”

Many trace the rise of statistics to a 2009 quote by Hal Varian, Google’s chief economist. “I keep saying that the sexy job in the next 10 years will be statisticians,” Varian told The New York Times. The subject is seeing a boom in college applications, while a 2011 study by the McKinsey Global Institute suggested the demand for workers with data analytic skills would rise to four million worldwide in the next five years.

Much of that demand is driven by tech companies like Varian’s own, which are both the creators and consumers of vast amounts of data. In 2009, Netflix awarded a $1m prize to a team of researchers, Bellkor’s Pragmatic Chaos, who won the competition to create an algorithm to predict user ratings for films. The team’s lead statistician was Chris Volinsky, director of the statistics department at AT&T, another stats rock star. Says Davidian, “There are rock stars, and then there are rock bands: statisticians frequently work in teams.”

One such rock band was the Obama 2012 campaign’s data analytics team, who not only predicted the President’s electoral success, but helped to make it happen. Its frontman was 30-year-old Dan Wagner. In the closing months of the 2012 race, Obama’s top advisers were troubled by the President’s fluctuating poll numbers in the key swing state of Ohio. Wagner’s statistical model, however, suggested Obama was comfortably, consistently ahead. Like Silver, he had seen the future.