The cosmological constant is a problem. Actually, that is an understatement, the cosmological constant was a problem, is a problem, and may always be a problem. To understand why this little constant has caused so stress for cosmologists, it is necessary to divide its history into two very distinct eras, and perhaps then, consider its future. Thus this introduction is to the cosmological constant what Marley’s ghost was to Scrooge. A warning of a trip through its history, its future, and perhaps, offering it a hint of redemption. 

The current theoretical estimations of the cosmological constant differ from the experimental measurements by such a shocking and huge magnitude, that it has often been referred to as ‘the worst prediction in the history of science.’ And as these values are provided by the non-unitable fields of physics quantum field theory and general relativity, respectively, finding an agreeable value for the constant, or even the reason why the values diverge so greatly could be the key to finding a quantum theory of gravity. 

Before embarking on that trip, let’s first meet our Scrooge — the central character of this potential redemption arc. The cosmological constant represents now, something different than it did when it was first introduced.

The easiest way to understand what it means now is by considering dark energy — the hypothetical force driving the Universe apart — to be the physical manifestation of the cosmological constant. As such, again solving the mystery of the cosmological constant could be the key to discovering exactly what dark energy is, and, in turn, discovering what the final fate of the Universe will look like. 

In many ways, the cosmological constant can be considered as a ‘counterpoint’ to gravity, a value for a force that repels as gravity attracts. This is something that links its present with its past. 

Einstein’s biggest blunder?

It’s sometimes mind-boggling to consider that the biggest problem in modern physics is a hangover from 1917. With all the advancements we have made in terms of understanding our Universe, how can this one little element, provide such a challenge?

The key to understanding why the cosmological constant has been such a thorn in the side of physics is understanding how it confounded the greatest physicist who ever lived — Albert Einstein. 

The cosmological constant, often represented by the Greek letter Lambda (λ), was added to Einstein’s field equations to balance the force of gravity. This is explainable by the fact that if only gravity acts in the Universe — an attractive force — then how can it not be shrinking? How did it form at all, if all matter is naturally drawn together?

Einstein felt that his field equations needed a repulsive factor to counter-balance the attractive force of gravity, and if this sounds like an ad-hoc solution — a fudge factor — that is because it was. Not only was the cosmological constant something of an arbitrary ‘fix’ it also made the field equations unstable. A slight variation should, according to these revised equations, cause the Universe to fall out of its static state. For example, if separation increases, gravitational attraction decreases and repulsion increases — resulting in further deviation from the initial state.  

Einstein was influenced to introduce the cosmological constant by the fact that the scientific consensus in 1917 was that the Universe was static — neither expanding nor contracting — and Einstein agreed with the consensus. Unfortunately, his field equations disagreed.

The field equations of general relativity did not allow for a static universe, predicting that the Universe should either be contracting or expanding. Thus, the first role that the cosmological constant was to provide negative pressure to counterbalance gravity. The argument underpinning this addition was that even empty space-time has a gravitational influence, a so-called vacuum energy. 

For twelve years, the cosmological constant remained in the field equations, fulfilling this role. But, trouble was brewing on the horizon, and by ‘the horizon,’ we mean the most distant horizon imaginable — the very edge of the Universe. 

Our understanding of the cosmos was about to change forever…

Hubble trouble — an expanding problem

It may be slightly difficult to believe today, but just 90 years ago we understood far less about the cosmos and the Universe around us. The idea of billions of galaxies outside of our Milky Way was almost undreamt of, as was the idea that these galaxies could be receding from each other as space expands. Likewise, the idea that the Universe could have inflated from an infinitely small point — the concept of the ‘Big Bang’ was pure fantasy. 

In 1929, Edwin Hubble’s seminal paper “A relation between distance and radial velocity among extra-galactic nebulae” would change this thinking forever. Hubble showed that the Universe was not infinite in either its reach or its age. In this relatively short paper, the astronomer presented the first observational evidence that distant galaxies are moving away from us, and further to this, the more distant they are, the more rapidly they recede. 

What Hubble was unaware about when he published his 1929 paper, was that other physicists had already provided solutions to Einstein’s field equations that his results confirmed. Both Alexander Friedmann, a Russian cosmologist, and Georges Lemaitre, a Catholic priest, mathematician, astronomer, and professor of physics, had provided solutions to the field equations that showed an expanding Universe. Even with this theoretical basis, Einstein wanted to see this evidence for himself, not being quite ready to scrap his cosmological constant and accept a non-static universe.

On January 29th 1931, Edwin Hubble met Einstein at Mount Wilson, taking him to see the famous 100-inch telescope where the astronomer had made the observation that doomed the first iteration of the cosmological constant. Shortly after Einstein published his first paper with revised field equations omitting lambda. He deemed the constant ‘redundant’ as relativity could explain the expansion of the Universe without it.

George Gamow, physicist and cosmologist, remarked in a 1956 Scientific American article and then later in his autobiography, that Einstein had confided in him that the introduction of the cosmological constant was his ‘biggest blunder.’ The remark has now passed into the lore surrounding the great scientist, and even though we can’t be certain that he actually said it, he very likely believed it. 

Yet, despite Einstein’s dismissal of the cosmological constant, many physicists were not quite ready to give up on this element of general relativity just yet. They argued without a cosmological constant term, models of the evolution of the cosmos would predict a universe with an age younger than the oldest stars within it. 

And though Einstein was unmoved by this argument, in 1998, 43 years after his death, the cosmological constant and the symbol that represents it would be rescued from obscurity and drafted to explain a new, but related conundrum.

The modern cosmological constant and dark energy

There is something of a pleasing irony about the fact that it was the discovery of the Universe expanding that consigned the cosmological constant to the dustbin and it was the equally important discovery that this same expansion is accelerating that saved it. 

During the cosmological constant’s ‘downtime’ our understanding of the origins of the Universe underwent a revolution. Cosmologists were able to deduce that regions of the Universe now separated by unimaginable distances were once in close proximity. The idea that the Universe expanded in a period of rapid inflation from an infinitely small point — though this point would become progressively smaller — to the vast entity we see today was accepted and termed the ‘Big Bang.’ 

Yet the commonly held idea that this period of rapid inflation had given way to a more steady rate was challenged in 1998.

In the mid-nineties, cosmologists had used solutions to the field equations of general relativity to assess the geometry of the Universe, determining that it is flat. This left some problems to be addressed. In a flat universe, we should have a matter/energy density which matches a value known as the critical density. Yet all the matter and energy that we can observe accounts for only a third of this value. Further to this missing energy problem, the flat universe suffers from a cosmic age problem, why do the oldest stars appear older than the predicted age of a flat universe?

One solution to these problems could arise if the Universe is filled with a fluid of negative pressure, a ‘dark energy’ that accounts for the energy deficit and provides accelerated expansion that would neatly explain the Universe taking a longer time to reach its current state. To measure this changing rate of expansion researchers would need a tool that could measure extraordinarily large cosmic distances — as large as 5 billion light-years, in fact. 

In 1998, astronomers found evidence of such a theoretical fluid from observations of the redshifts of distant yet incredibly, bright Type Ia supernovae — often referred to as ‘standard candles’ due to their reliability in measuring cosmic distances. And of course, scientists would need a symbol to represent dark energy within their equations. As cosmology already had such a representation of negative pressure, why not simply resurrect it and place it back in the equations of general relativity?

But, they should not have been surprised, given its history, that re-employing the cosmological constant would bring new problems.

Still crazy after all these years…

The new issues with the cosmological constant very much reflect the major hurdles within physics as it currently stands. Whilst revolutions were being made on incredibly large scales thanks to cosmology, our understanding of the incredibly small was burgeoning thanks to the success of quantum physics. 

The problem arises from the fact that quantum physics — and in particular quantum field theory — and general relativity can not be reconciled, there is no theory of quantum gravity. 

If we are creative and slightly whimsical, we could perhaps give this struggle to unify these disciplines a value — 10¹²¹ — the magnitude between which the quantum field theory’s theoretical prediction of the cosmological constant and the observed value provided by cosmology. This massive disparity–often described as ‘the worst theoretical prediction in the history of science’– arises from the fact that quantum field theory predicts that virtual particles are popping in and out of existence at all times — an idea that may sound ridiculous but has been experimentally verified — even in the vacuum of space. Thus particles should have a measurable effect on the vacuum energy driving the expansion of the Universe but such an effect isn’t measured by cosmologists observing the redshifts of Type Ia supernovae.

There are, of course, solutions. Dark energy could be associated with some, as yet undiscovered field, which fills space in a way similar to that of the Higgs field from which the recently discovered Higgs boson emerges. Or perhaps other constants that occupy an unchallenged place in our equations of gravity aren’t constants at all but vary with time, as University of Geneva cosmologists Lucas Lombriser suggests. More extreme solutions lie in the suggestion that Einstein’s theory of gravity must be modified to account for dark energy — all though this family of theories, MONDS, are steadily moving out of favour within the physics community.

Whatever the solution to this problem is, it has a remarkable impact on the future of the Universe. Determining the true value of the cosmological constant and the strength of dark energy driving this accelerating expansion will ultimately tell us if the Universe’s final fate is to rip apart or violently crush together. 

Whether by ‘Big Rip’ or ‘Big Crunch’ the Universe’s end will be determined by the value of the cosmological constant. A value that still continues to evade us and confuse us as much as it did Einstein.

Sources

Ta-Pei Cheng, Relativity, Gravitation and Cosmology, Oxford Press, (2010).

Robert Lambourne, Stephen Serjeant, Mark Jones, An Introduction to Galaxies and Cosmology, Cambridge Press, (2015).

Frank Close, The New Cosmic Onion, Taylor &Francis, (2007).

Cormac O’Raifeartaigh, Investigating the legend of Einstein’s “biggest blunder”, Physics Today, (2018)

Matts Roos, Introduction to Cosmology, Wiley, (2003).