During the 20th century, the idea that the Universe existed in a steady state was seriously challenged and eventually dismissed by the discovery that the Universe is not only expanding but is also doing so at an accelerating rate. The reason for this accelerating expansion is thus far unknown, but scientists have given this force a name–albeit one that is a placeholder–dark energy.

Explaining this accelerating rate of expansion has become one of the most challenging problems in cosmology. The fact that the value for this acceleration varies wildly between theory and practical observations has created a raft of problems in itself. This means that the net result arising from any attempt to explain dark energy creates more questions than it answers.

Dark Energy: The Basics

Dark energy is whatever is causing the expansion of the Universe to accelerate. One of the most striking things about this mysterious energy is just how much ‘stuff’ in the Universe it accounts for. If you consider the contents of the cosmos to be matter and energy –more formally known as the Universe’s mass-energy density–then dark energy accounts for between 68% to 72% of all cosmic ‘stuff.

Dark matter is the second-largest contributor, with just a tiny proportion of the Universe’s ‘stuff’ made up of the baryonic matter consisting of atoms that we see around us on a day-to-day basis.

This is even more staggering when you consider that all the stars, planets, dust, gas and cosmic bodies that make up the visible Universe are contained in this tiny fraction of cosmic stuff that doesn’t even amount to 5% of the Universe’s contents.

Dark energy could very roughly be described as a force acting in opposition to gravity. Whereas the more familiar everyday force of gravity holds objects like planets and stars together in orbits, dark energy acts as a repulsive force, driving galaxies themselves apart. But, whereas gravity acts upon objects themselves, dark energy acts on the very fabric of spacetime between objects. Because dark matter is the largest contributor to gravity, this means this mysterious substance is locked in what has been termed a ‘cosmic tug of war.’ And it’s clear during our current epoch, dark energy is winning!

A popular analogy for this is the description of the Universe as the surface of a balloon. Galaxies are represented by two marker ink dots on this surface. As the balloon is inflated the points move apart with the very space between them expanding.

Repeat the experiment with three unevenly placed dots and it is clear that the dots that are initially further apart recede away from each other more rapidly. This extremely rough analogy carries over to galaxies. The further apart the galaxies are, the more quickly they recede.

An Expanding Surprise

The discovery that the Universe is expanding came as a considerable shock to the scientific community when it was confirmed by Edwin Hubble in 1929. Hubble had built upon theory provided by George Lemaitre and Alexander Friedman who had used the equations of general relativity to predict that the Universe was non-static, something that very much contradicted the scientific consensus of the time.

Albert Einstein who devised general relativity had also found that his geometric theory of gravity predicted a non-static Universe, something he wasn’t exactly comfortable with. Despite having already killed many of the sacred cows of physics, Einstein was unwilling to do away with the concept of a static Universe. To recover a static Universe that was neither expanding nor contracting, the world’s most famous scientist introduced to his equations a ‘fudge factor’ called the cosmological constant–commonly represented with the Greek letter lambda.

The cosmological constant was in danger from the start. Once Hubble managed to persuade Einstein that the Universe was indeed expanding, the physicist abandoned the cosmological constant, allegedly describing it as his ‘greatest blunder’ in his later years. The cosmological constant wouldn’t stay in the cosmological dustbin for very long, however.

If physicists had been surprised by the discovery at the beginning of the 20th Century that the Universe is expanding, they would be blown away when at the end of that same century when the observations of distant supernovae made by two separate teams of astronomers revealed that not only is the Universe expanding, it is doing so at an accelerating rate.

To understand why this is shocking and how it leads to the conclusion that some repulsive force is driving this expansion, it is necessary to journey to the very beginning of time… Let’s take some balloons too…

Escaping the Big Crunch

When thinking about the initial expansion of the Universe it makes sense to conclude that the introduction of an attractive force within that Universe would slow and eventually halt this expansion. That is exactly what gravity should do, and it seems did do during the early stages of expansion that had nothing to do with dark energy (we think).

Some cosmologists are willing to go a step further. If there is no outward pressure but an inward attractive force, shouldn’t the Universe actually start to contract?

This leads to the theory that the Universe will end in what physicists term the ‘Big Crunch’–an idea that dark energy could make obsolete. Think about how counter-intuitive this was to scientists when the idea was first suggested and evidenced. Let’s return to the balloon analogy; imagine you stop blowing into the balloon, and instead start sucking the air out of it.

How shocked would you be to find that the balloon isn’t contracting, it’s continuing to expand? And not just that, it’s actually expanding faster than it was when you were blowing into it!

With that in mind, consider the initial moments of the Universe. Beginning in an indescribably dense and hot state, squeezed into a quantum speck, the Universe undergoes a period of rapid expansion. This period of expansion wasn’t driven by dark energy. As it expands, the Universe cools allowing electrons to form atoms with protons and neutrons, which in turn frees photons to travel the cosmos.

Soon there is enough matter in the Universe to allow the attractive force of gravity to slow its expansion. And this does seem to be what happened in the early cosmos. The rapid inflation of the infant Universe is believed to have halted at around 10 ^-32 seconds after the Big Bang, with the Universe still expanding, albeit at a much slower rate.

This period of expansion continued to slow as a result of the growing dominance of matter during what cosmologists call the ‘matter-dominated epoch.’ But, at around 9.8 billion years into the Universe’s 13.8 billion year history, something strange begins to happen. The Universe begins to expand again, this time at an accelerating rate.

This is the dawn of the dark energy dominated epoch.

The Cosmological Constant is Back and Still Causing Trouble

That explains why the accelerating expansion of the Universe is so troubling and the need to introduce the placeholder concept of dark energy to explain it. Yet this accelerating expansion would still need a mathematical representation in equations used to describe the Universe. To do this cosmologists would return to the cosmological constant and its symbol, lambda.

This new iteration of the cosmological constant would be used in a different way to Einstein’s version. Whereas the earlier cosmological constant was used to balance gravity and hold the Universe steady and static, this new version would be used to overwhelm gravity and account for the acceleration of its expansion. But, this revised use of the cosmological constant does not mean it is any less troublesome than Einstein had found its predecessor.

In fact, the difference between the cosmological constant’s measured value, found by measuring the redshift of distant Type Ia supernovae, diverges from the value predicted by quantum field theory and particle physics by a value as large as 10¹²¹ (that’s 1 followed by 121 zeroes). Thus, it should come as no surprise that this value has been described as the worst prediction in the history of physics.

And as it represents the action of dark energy, that makes dark energy itself cosmology’s biggest conundrum.

OK… But What is Dark Energy?

So by now, you might well be thinking all of this is all fine and good, but this article specifically asks ‘what is dark energy?’ Isn’t it time to get to answering this question? It should come as no surprise that the answer is no one knows. But, that doesn’t mean that cosmologists don’t have some very good ideas.

One of the explanations for dark energy says that it could be vacuum energy, an underlying background energy that permeates that Universe and is represented by the cosmological constant. The most commonly cited evidence for vacuum energy–the energy of ’empty’ space which manifests as the Casimir effect.

Without delving too deeply into this, as relativity states that energy and mass are equivalent and mass has gravitational effects, then it stands to reason if empty space has vacuum energy, this too should contribute to the effect of gravity across the cosmos. That contribution has been factored in as a negative repulsive influence acting against the attractive influence of gravity.

The big problem with this is that quantum field theory suggests that this negative pressure contribution from vacuum energy should arise from all particles and thus, should give lambda a value that is tremendously larger than that obtain when our astronomers measure the redshift of Type Ia supernovae in distant galaxies.

This problem could be solved by dark energy’s effects being the result of something other than vacuum energy, of course. The Universe’s accelerating expansion could be due to some, as of yet undiscovered fundamental force of nature. Alternatively, it could indicate that our best current theory of gravity–general relativity– is incorrect.

A new generation of cosmologists is currently actively tackling the dark energy puzzle with new and revolutionary ideas. These include the idea that dark energy could have started work in the early Universe, an idea proposed by Early Dark Energy (EDE) models of the Universe. Another alternative is that dark energy does not influence the curvature of the Universe, or perhaps does so weakly–a theory referred to as the ‘well-behaved cosmological constant.’

As unsatisfying an answer as it is, the only honest way of addressing the question ‘what is dark energy?’ right now is by saying; we just don’t know. But, science wouldn’t be anywhere near as fascinating without mysteries to solve, and revolutionary ideas to be uncovered.