Beware rushing to conclusions about which policies are best for defeating coronavirus

Despite recent remarks from Sweden's chief epidemiologist, attempts to identify a dominant policy explanation for the variation in excess deaths between countries are not (yet) convincing, writes Ben Chu

'If we would encounter the same disease, with exactly what we know about it today, I think we would land midway between what Sweden did and what the rest of the world did,' Dr Tegnell told a local radio station (Getty)

Natural scientists run experiments in a laboratory to test their hypotheses. Imagine you’re a biologist and you want to see the impact of rising temperature on the growth of a strain of bacteria? Take two samples, heat one but not the other, and observe the results. But social scientists lack that ability to test their ideas about the effectiveness of various social policies in a lab. You can’t put two countries on petri dishes and inject them with different policies.

That’s why social scientist are always on the hunt for so-called natural experiments. Think of two similar countries. One imposes a ban on smoking indoors while the other doesn’t. What subsequently happens to overall cigarette sales in both?

Consider two adjacent US states. One hikes the local minimum wage significantly while the other keeps its own steady. What then happens to the rate of unemployment in each? Such natural experiments are immensely valuable to researchers as a source of empirical (as opposed to theoretical) evidence for what works and what doesn’t when it comes to policy.

The coronavirus pandemic has created an abundance of natural experiments as governments have responded to the disease outbreak in sometimes very different ways. In Europe, Sweden has been a major outlier, controversially refraining from locking down its economy and society like most of the rest of the continent.

And this week Sweden’s chief epidemiologist, Anders Tegnell, seemed to draw some policy conclusions from this historic natural experiment. “If we would encounter the same disease, with exactly what we know about it today, I think we would land midway between what Sweden did and what the rest of the world did,” he told a local radio station.

On the face of it this as an example of progressive social science in action: evaluate evidence, improve policy. Yet, as all good social scientists recognise, the evidence supplied by natural experiments also needs to be interpreted with immense care and caution.

Other things can also be affecting the results which are unrelated to the policy change under examination. It’s been suggested one of the reasons Japan has not been hit hard by the coronavirus pandemic is due to a strong pre-existing culture of public cleanliness rather than policy decisions by its government. There are also differences in populations that might need to be taken into account. A country with an older population would be likely to see higher mortality numbers, all else equal, given the much higher vulnerability of older people to the disease.

Countries with bigger and denser cities might also expect to see the disease spread more rapidly. Drawing conclusions while an outbreak is still taking place is also potentially hazardous. The statistical data produced by different countries may be collected differently, making comparisons problematic.

Think of the high number of cases identified early in the pandemic in Germany, which had an extensive testing infrastructure, with the low numbers reported in many African countries, which often had very little testing capability.

Excess deaths – deaths above the country’s own average for the time of year – have since become the main dependent variable for comparing public policy effectiveness between countries. Yet even here there are problems with drawing conclusions in the heat of a crisis, not least the fact that different countries don’t report deaths on the same timetable.

Despite Dr Tegnell’s remarks, attempts to identify a dominant policy explanation for the variation in excess deaths between countries in this pandemic are not (yet) convincing. Many make the case for earlier lockdowns. Yet several countries in Asia that never locked down at all seem to have been hit less hard.

There seems to be an element of randomness in how the disease takes hold in different countries. One explanation for this is that the disease’s transmission is very uneven between different people and the role of “super-spreaders” and super-spreading events is central.

That supports the case for an assiduous and comprehensive policy of track, trace and isolate. So where does all this leave the social scientist trying to evaluate policy?

Natural experiments will remain important but also potentially misleading. Comparisons are necessary and inevitable, but sometimes, as the old saying goes, odious.