The recent film The Theory of Everything tells the story of Stephen Hawking, who managed to become a world-famous physicist despite being confined to a wheelchair by a degenerative disease. It's mostly about his relationship with his ex-wife Jane, but it does find a bit of time to explain what Hawking has spent his career doing.

He certainly didn't lack ambition. Hawking has been one of many physicists trying to come up with a "theory of everything", a single theory that will explain everything about our universe. He was following in the footsteps of Albert Einstein, who tried and failed to devise such a theory.

Finding a theory of everything would be a staggering achievement, finally making sense of all the weird and wonderful things in our universe. For decades, confident physicists have said that one is just around the corner. So are we really on the verge of understanding everything?

On the face of it, a theory of everything sounds like a tall order. It would have to explain everything from the works of Shakespeare to the human brain and the forests and valleys of our natural world, says John Barrow of the University of Cambridge in the UK. "That's the question of the universe."

Nevertheless, Barrow thinks finding a theory of everything "is quite conceivable". That's because "the laws of nature are rather few, they're simple and symmetrical and there are only four fundamental forces."

In a way we have to put aside the complexity of the world we live in. "The outcomes of the laws - the things that we see around us - are infinitely more complicated," says Barrow. But the rules underlying it all may be simple.

In 1687, it seemed to many scientists that a theory of everything had been found.

Newton was walking in a garden when he saw an apple fall from a tree

The English physicist Isaac Newton published a book in which he explained how objects move, and set out how gravity works. The Philosophiæ Naturalis Principia Mathematica – that's "Mathematical Principles of Natural Philosophy" to you and me – presented the world as a beautiful, ordered place.

The story goes that, at the age of 23, Newton was walking in a garden when he saw an apple fall from a tree. At the time, physicists knew that the Earth somehow pulled objects down by the force of gravity. Newton would take this idea further.

According to John Conduitt, his assistant in later years, seeing the apple fall led Newton to the idea that the gravitational force "was not limited to a certain distance from earth, but that this power must extend much further than was usually thought". According to Conduitt's account, Newton then asked: "Why not as high as the Moon?"

Inspired, Newton developed a law of gravity, which worked equally well for apples on Earth and planets orbiting the Sun.  All these objects, which seemed so different, turned out to obey the same laws.

In the same book, Newton set out three laws governing how objects move. Combined with the law of gravity, these laws explained how a ball moves when you throw it and why the Moon orbits the Earth.

"People thought that he had explained everything there was to explain," says Barrow. "His achievement was immense."

The problem was, Newton knew his work had holes.

For instance, gravity doesn't explain how small objects hold themselves together, as the force isn't strong enough.  Also, while Newton could describe what was happening, he couldn't explain how it worked. The theory was incomplete.

Mercury wasn't playing ball

But there was a bigger problem. While Newton's laws explained most of the common phenomena in the universe, in some cases objects broke his laws. These situations were rare, and generally involved extreme speeds or powerful gravity, but they were there.

One such circumstance was the orbit of Mercury, the closest planet to the Sun. As each planet orbits the Sun it also rotates. Newton's laws could be used to calculate how they should rotate, but Mercury wasn't playing ball. Equally strangely, its orbit was off-centre.

The evidence was clear. Newton's universal law of gravitation wasn't universal, and wasn't a law.

Over two centuries later, Albert Einstein came to the rescue with his theory of general relativity.  Einstein's idea, which in 2015 celebrates its 100th anniversary, offered a much deeper understanding of gravity.

Really heavy objects like planets, or really fast-moving ones, can distort space-time

The core idea is that space and time, which seem like different things, are actually interwoven. Space has its three dimensions: length, breadth and height. Then there is a fourth dimension, which we call time. All four are linked in a kind of giant cosmic sheet. If you've ever heard a character in a science fiction movie mention "the space-time continuum", this is what they're talking about.

Einstein's big idea was that really heavy objects like planets, or really fast-moving ones, can distort space-time. It's a bit like the taut fabric of a trampoline: if you put a heavy weight on it, the fabric bows and curves. Any other objects will then roll down the sheet towards the object. This, according to Einstein, is why gravity pulls objects towards each other.

This is a deeply weird idea. But physicists are convinced that it is true. For one thing, it explains the strange orbit of Mercury.

According to general relativity, the Sun's huge mass warps space and time around it.

As the closest planet to the sun, Mercury experiences much bigger distortions than any of the other planets.  The equations of general relativity describe how this warped space-time should affect Mercury's orbit, and predict the planet's position down to a tee.

But despite this success, general relativity isn't a theory of everything, any more than Newton's theories were. Just as Newton's theory didn't work for really massive objects, Einstein's didn't work on the very small.

Once you start looking at tiny things like atoms, matter starts to behave very oddly indeed.

Up until the late 19th century, the atom was thought to be the smallest unit of matter. Coming from the Greek atomos meaning "indivisible", the atom by its very definition was not supposed to be able to be divided into smaller particles.

But in the 1870s, scientists found particles that were almost 2000 times lighter than atoms.

Scientists have found ways to divide matter smaller and smaller

By weighing light rays in a vacuum tube, they found extraordinarily light, negatively-charged particles. This was the first discovery of a subatomic particle: the electron.

In the next half-century scientists discovered that the atom had a nucleus hub, which the electrons buzzed around. This hub – which was by far the heaviest part of the atom – was made up of two types of subatomic particles: neutrons, which are neutrally charged and protons, which are positively charged.

But it didn't stop there. Since this time, scientists have found ways to divide matter smaller and smaller, continuing to redefine our notion of fundamental particles. By the 1960s, scientists had found dozens of elementary particles, drawing up a long list known as the particle zoo.

As we understand it today, of the three components of an atom, electrons are the only fundamental particles. Neutrons and protons can be divided further into teeny, tiny particles called "quarks".

Einstein never really believed in quantum theory

These subatomic particles were governed by an entirely different set of laws than those governing big objects like trees or planets.  And these new laws – which were far less predictable - threw a spanner in the works.

In quantum physics, particles don't have defined locations: their whereabouts is a bit fuzzy.  All we can say is that each particle has a certain probability of being in each location. This means the world is a fundamentally uncertain place.

This may all seem very unfathomable and far-out. All we can say is, it's not just you that feels that way. The physicist Richard Feynman, an expert on the quantum, once said: "I think I can safely say that no one understands quantum mechanics."

Einstein was also disturbed by the fuzziness of quantum mechanics. "Despite having instigated it, Einstein never really believed in quantum theory," says Barrow.

All the same, for their respective domains – the big and the small – both general relativity and quantum mechanics have proven, time and time again, to be tremendously accurate.  

Quantum physics has explained the structure and behaviour of atoms, including why some of them are radioactive. It also underlies all modern electronics. You could not read this article without it.

Meanwhile general relativity was used to predict the existence of black holes. These are stars so massive that they have collapsed in on themselves. Their gravitational attraction is so powerful that nothing – not even light – can escape from it.

   

But the issue is, the two theories are not compatible, so they can't both be right. General relativity says that objects' behaviours can be predicted exactly, whereas quantum mechanics says all you can know is the probability that they will do something.

That means there are some things physicists still can't describe. Black holes are a particular problem. They are massive so general relativity applies, but they are also small so quantum mechanics applies too.

Unless you're close to a black hole, this incompatibility doesn't affect your day-to-day life. But it has perplexed physicists for most of the last century. It's this incompatibility that has driven the quest for a theory of everything.

Einstein spent much of his life trying to find such a theory. Never a fan of the randomness of quantum mechanics, he wanted to create a theory that would bring together gravity and the rest of physics, with all the quantum weirdness as a secondary consequence.

Einstein spent 30 years on a fruitless quest

His major challenge was to make gravity work with electromagnetism. In the 1800s, physicists had worked out that electrically-charged particles could be attracted or repelled by each other. That's why some metals are attracted to magnets. This meant there were two kinds of force that objects could exert on each other: they could attract each other with their gravity, and either attract or repel with their electromagnetism.

Einstein wanted to bring the two forces together into a "unified field theory". To do this, he extended his space-time to five dimensions. As well as the three of space and one of time, he added a fifth dimension that was so small and curled up we couldn't see it.

This didn't work out, and Einstein spent 30 years on a fruitless quest. He died in 1955, his unified field theory still undiscovered. But in the following decade, the strongest contender for a theory of everything emerged: string theory.

The idea behind string theory is oddly simple. The basic ingredients of the world, such as electrons, are not actually particles at all. Instead they are little loops or "strings". It's just that these strings are so small, they seem to be mere points.

All the different particles discovered in the 20th century are really the same kinds of strings

Just like the strings on a guitar, these loops are under tension. That means they vibrate at different frequencies, depending on their size.

In turn, these oscillations determine what sort of "particle" each string appears to be. Vibrate a string one way and you get an electron. Vibrate it another way, and you get something else. All the different particles discovered in the 20th century are really the same kinds of strings, just vibrating in different ways.

It may not be immediately obvious why this is a good idea. But it seems to make sense of all the forces acting in nature: gravity and electromagnetism, plus two that were only discovered in the 20th century.

The strong and weak nuclear forces are only active within the tiny nuclei of atoms, which is why it took so long for anyone to notice them. The strong force holds the nucleus together. The weak force normally does nothing, but if it gets strong enough it breaks the nucleus apart: this is why some atoms are radioactive.

For the first time, general relativity and quantum mechanics had found common ground

Any theory of everything would have to explain all four. Fortunately, the two nuclear forces and electromagnetism are all covered by quantum mechanics. Each is carried by a specialized particle. But there's no particle to carry the force of gravity.

Some physicists think there is. They call this particle the "graviton". Gravitons would have to have no mass, spin in a particular way, and travel at the speed of light. Unfortunately, nobody has ever managed to find one.

This is where string theory comes in. It describes a string that looks exactly like a graviton: it spins in the right way, is massless and travels at the speed of light. For the first time, general relativity and quantum mechanics had found common ground.

As a result, in the mid-1980s physicists became hugely excited about string theory. "In 1985 we realised string theory solved a lot of the problems people had struggled with for the last 50 years," says Barrow. But it also has a host of problems.

For starters, "we don't really understand what string theory is in full detail," according to Philip Candelas of the University of Oxford in the UK. "We don't have a good way to describe it."

It also makes some predictions that seem outright bizarre. While Einstein's unified field theory relied on a single hidden extra dimension, the earliest forms of string theory called for a total of 26 dimensions. These had to be there to make the mathematics consistent with what we already know about the universe.

More advanced versions, known as "superstring theories", get by with just 10 dimensions. But even that is a far cry from the three dimensions we see on Earth.

"The way we reconcile this is by saying that only three expanded in our world and became large," says Barrow. "The others are there but remain fantastically small."

Because of these and other problems, many physicists are unconvinced by string theory. Some have instead studied another theory: loop quantum gravity.

Loop quantum gravity proposes that space-time is actually divided into small chunks

This isn't an attempt at an overarching theory that incorporates particle physics. Instead, loop quantum gravity just sets out to find a quantum theory of gravity. It's more limited than string theory – but it's also not as unwieldy.

Loop quantum gravity proposes that space-time is actually divided into small chunks. When you zoom out it appears to be a smooth sheet, but when you zoom in, it is a bunch of dots connected by lines or loops. These small fibres, which are woven together, offer an explanation for gravity.

This idea is just as boggling as string theory, and it has the same problem: there's no hard experimental evidence. 

Why do these theories keep stumbling? One possibility is that we simply don't know enough yet. If there are major phenomena that we've never even seen, we are trying to understand the big picture while missing half the pieces.

"It's very tempting to think we've discovered everything," says Barrow. "But it would be very suspicious if in the year 2015 we could make all the observations necessary to have a theory of everything.  Why should it be us?"

For all its problems, string theory still looks promising

There's also a more immediate problem. The theories are really difficult to test, largely because the maths is so fiendish. Candelas has struggled for years to find a way to test string theory, so far without success.

"The main obstacle to the advancement of string theory is there's not enough maths known to advance the study of physics," says Barrow. "It's such an early stage and there's so much to explore."

For all its problems, string theory still looks promising. "For many years people have been trying to unify gravity with the rest of physics," says Candelas. "We had theories that explained electromagnetism and the other forces well, but not gravity. With string theory we put them together."

The real problem is that a theory of everything may simply be impossible to identify.

When string theory became popular in the 1980s, there were actually five different versions of it. "People began to worry," says Barrow. "If there's a theory of everything, why are there five of them?"

Over the next decade, physicists discovered that these theories could be transformed into each other. They were different ways of looking at the same thing.

M-theory doesn't offer a single theory of everything

The end result was M-theory, put forward in 1995. This is a deeper version of string theory, incorporating all the earlier versions. That looks good: at least we're back to a single theory. M-theory also only needs 11 dimensions, which is at least better than 26.

But M-theory doesn't offer a single theory of everything. It offers billions upon billions of them. In total, M-theory gives us 10 to the power of 500 theories, all of them logically consistent and capable of describing a universe.

That looks worse than useless, but many physicists now think it points to a deeper truth.

The simplest conclusion is that our universe is one of many, each of them described by one of the trillions of versions of M-theory. This huge collection of universes is called the "multiverse".

At the beginning of time, the multiverse was like "a great foam of bubbles, all slightly different shapes and sizes," says Barrow. Each bubble then expanded into its own universe.

"We're in just one of those bubbles," says Barrow. As the bubbles expand, other bubbles can arise inside them, each one a new universe. "It's making the geography of the universe really complicated."

Within each bubble universe, the same physical laws will apply. That's why everything in our universe seems to behave the same.

There are trillions of other universes, each one unique

But the rules will be different in other universes. "The laws we see in our universe are just like bylaws," says Barrow. "They govern our bit, but not all of the universes."

This leads us to a strange conclusion. If string theory really is the best way to combine general relativity and quantum mechanics, then it both is and isn't a theory of everything.

On the one hand, string theory may give us a perfect description of our own universe. But it also seems to lead, inescapably, to the idea that there are trillions of other universes, each one unique.

"The big change in thinking is we don't expect there to be a unique theory of everything," says Barrow. "There are so many possible theories they're almost filling every possibility of thinking."