Bang goes the theory: How physicists lost touch with reality

From Einstein's formulations to the Hadron Collider, experiments stretch the limits of science. So how did physics finds itself overwhelmed by theory?

The day I reached the summit of Mauna Kea – Hawaii's highest and most sacred mountain – the sun had just set. But even the dying sun seemed capable of setting alight everything that stretched before us.

The bank of clouds spread out below the summit looked as if fires raged beneath them. The bottom layers of high, storm-driven cirrus clouds burned bright. Between these two layers were the domes of the Keck I and Keck II – twin 10-metre-class telescopes – silhouetted against a sky replete with yellows, oranges, and reds.

The Keck telescopes are helping us understand one of the biggest mysteries in physics and cosmology today: why is the universe being blown apart? Astronomers discovered in the late Nineties – to considerable astonishment, even alarm – that the expansion of the universe is accelerating, and not slowing down as many had predicted. It's as if some mysterious energy is creating a repulsive force to counter gravity. Clueless as to the exact nature of this force, cosmologists call it dark energy. More important, it seems to constitute nearly three-quarters of the total matter and energy in the universe.

Dark energy is the latest and most daunting puzzle to confront cosmologists, adding to another mystery that has haunted them for decades: dark matter. Nearly 90 per cent of the mass of galaxies seems to be made of matter that is unknown and unseen. We know it must be there, for without its gravitational pull the galaxies would have disintegrated. Cosmologists in particular and physicists in general, are now faced with the stark reality that roughly 96 per cent of the universe cannot be explained with the theories at hand. All our efforts to understand the material world have illuminated only a tiny fraction of the cosmos.

And there are other mysteries. What is the origin of mass? What happened to the anti matter that should have been produced along with matter during the big bang? After almost a century of success at explaining our world using the twin pillars of modern physics – quantum mechanics and Einstein's general theory of relativity – physicists have reached a plateau.

The way forward will involve reconciling quantum mechanics with general relativity into a theory of quantum gravity. In situations where the two domains collide – where overwhelming gravity meets microscopic volumes, such as in black holes or in a big bang – the theories don't work well together. In fact, they fail miserably. One of the most ambitious attempts to bring them together is string theory, an edifice of incredible mathematical complexity. Its most ardent proponents hope that it will lead us not just to quantum gravity but to a theory of everything, allowing us to describe every aspect of the universe with a few simple equations.

But the theory's hoped-for denouement is nowhere in sight. Far from explaining our universe, string theory seems to predict the existence of 10 to the power of 500 universes or more. Crucially, the theory is so far from being verified experimentally that it has become the poster child of what is wrong with physics today. Theory has lost touch with experiments – and, so, with reality.

The greatest advances in physics have come when theory has moved in near-lockstep with experiment. Sometimes the theory has come first and sometimes it's the other way around. It was an experiment in 1887 by Albert Michelson and Edward Morley – showing that the speed of light is not dependant on the motion of the observer – that influenced Einstein's 1905 formulation of the special theory of relativity. A decade later, Einstein produced the general theory of relativity, but it was only after experiments in 1919 verified a fascinating implication of general relativity – the bending of starlight by the sun's gravity – that the theory gained widespread acceptance. And throughout the early to mid-1900s, theorists and experimentalists jostled and outdid each other as they shaped quantum mechanics. An equally fruitful collaboration occurred in the 1960s and 1970s, when particle physicists theorised about the fundamental particles and forces that make up the material world, and experiments confirmed their startlingly accurate predictions. But this energetic interplay is now deadlocked. The discovery of dark energy and dark matter, along with the failure, so far, of experiments to find the Higgs boson (thought to give elementary particles their mass), has allowed theorists free rein. Ideas abound, adrift in a sea of speculation. Can the next generation of experiments help anchor the theories to reality?

It was to answer to this question that I travelled to the distant reaches of the Earth, from desolate deserts to the depths of derelict mines, from mountain tops to the bottom of the world, looking for experiments that promise to drag physics out of its theoretical morass.

Many I visited are tackling the twin mysteries of dark matter and dark energy. But I also saw the telescopes and detectors searching for antimatter, the Higgs boson, and neutrinos; elusive subatomic particles pervading the universe. All experiments at the cutting edge of physics and cosmology. Modern observational cosmology began at Mount Wilson in California, where Edwin Hubble discovered that our universe is expanding, thus laying the foundation for the big bang theory. The 100-inch telescope atop Mount Wilson pushed the technological boundaries of its time. Every evening, they open their giant domes to peer across the universe, gathering light. The instruments that analyse this light are equally powerful. Consider the 8.6-ton spectrograph on Mauna Kea helping astronomers to study the universe with incredible accuracy. In contrast are the small silicon and germanium detectors, so exquisitely engineered they are handled like works of art, as they wait patiently, week after week, for the merest hint of dark matter.

These experiments are dwarfed by gigantic balloons that soar into the stratosphere bearing experiments that search for primordial antimatter and study the cosmic microwave background (a radiation left over from the big bang). Experimental physics reaches its pinnacle at the Large Hadron Collider, the world's largest particle smasher. Machines weighing thousands of tons monitor the paths of subatomic particles, which then spew forth from collisions of proton beams – each carrying as much energy as a 400-ton train going 150km per hour. Superconducting magnets strain to keep these beams confined to their paths around a 27km-long underground tunnel. New particles that emerge from the cauldron of proton smash-ups may contain anything from the Higgs boson, to dark matter, to the first hints of extra dimensions.

This magnificent machinery can work only in the most extreme settings, their surreal environments, the unsung characters in this story. The cold, dry air above the Atacama Desert high in the Chilean Andes allows starlight that has travelled for billions of years to enter a telescope without being smudged in its final approach by something as mundane as water vapour. The crystalline clarity of Lake Baikal in Siberia is crucial to a pioneering underwater neutrino telescope.

Descending into the Earth's crust provides similar benefits. Deep within an abandoned iron mine in Minnesota, physicists hunt for dark matter, their detectors shielded from the chaos of cosmic rays by a half-mile of rock. Meanwhile, a vast and arid land in the interior of South Africa – a desolate expanse devoid of pollution – has been proposed as the site of the world's biggest radio telescope, its 3,000 antennas capable of sweeping across vast swathes of the universe faster than any instrument ever built.

As extreme destinations go, there are few that compare with Antarctica, a land so frigid that a sharp intake of breath can sear one's lungs, and mortal danger, in the form of snow-covered crevasses, is only a moment of distraction away. Still, cosmologists cherish the Antarctic Plateau for its thin, dry, stable and unpolluted air, and they are building gigantic telescopes to probe the cosmic microwave background with a precision that's impossible to emulate almost anywhere else on Earth. But it's not just the air above Antarctica that attracts the scientists. They are also turning the kilometres-thick ice at the South Pole into a neutrino detector. Nowhere else exists a block of material so massive, clear and solid that it can be used to study the slipperiest particle in the universe. A frozen wasteland could lead us toward the correct theory of quantum gravity.

Atop Mauna Kea, by the time I entered the Keck II dome it was dark. There were no lights. The telescope wasn't being used yet for observing, but the dome was open. Despite the high cirrus clouds, a dozen or so bright stars were visible through the open dome. It was quiet inside, except for the cooling system of one of the telescope's instruments, its high-pitched pistons pumping refrigerants in long, bird-like chirps. Once my eyes adjusted to the almost total darkness of the dome, the opening seemed to fill up with stars, perhaps a hundred of them, that hadn't been visible moments earlier. I looked down at the primary mirror and caught a glimpse of it through the tangle of trusses that held it in place, a flash of silver, a bright star reflecting off its shiny, polished surface. For a brief instant, Hawaii and its overwhelming assault on the senses dis- appeared. A quiet descended. It was just the dome, the telescope, and starlight that had travelled for thousands of years.

'The Edge of Physics' by Anil Ananthaswamy is published in hardback by Gerald Duckworth & Co (£16.99), www.edgeofphysics.com. To order a copy for the special price of £15.29 (free P&P) call Independent Books Direct on 08430 600 030, or visit www.independentbooksdirect.co.uk