The science of serendipity

You could call it luck. Maybe coincidence. Scientists like the word “serendipity”, so let’s go with that. This particular moment of serendipity happened in a WeWork shared office space in central London in 2019. James Nightingale, a young cosmologist intrigued by dark matter, had come to London from Durham University for a conference with his supervisor, James Murray, and was meeting a friend – a computer-coding whiz – to talk data.

They sat for a coffee and chatted. How, Nightingale wondered, could they stop the Euclid space telescope being damaged by stellar radiation? His friend mentioned an obscure statistical tool called nested sampling.

There was, at that point, an interruption from a man at the next table. “Nested sampling? What do you know about nested sampling?”

The interruptor was Matthew Griffiths, who had recently left his research position at Cambridge to found a cancer research startup, Concr. He had used nested sampling for a recent paper he had written on atomic chemistry. The three of them got talking. The Durham researchers realised the tool they had in mind to model radiation damage to Euclid might also model cell damage in humans.

Seven years later, that conversation has paid off. Concr has a tool that will make cancer treatment much less toxic for patients. At the moment, doctors usually prescribe a cocktail of chemotherapy drugs – four, five, maybe more – and if one doesn’t work, another will. But chemotherapy is a kind of controlled poison, so it would be better to leave out the ones that don’t work. By analysing thousands of data points in patients’ medical histories and in drug trials, Concr can say with about 85%-90% confidence which of these treatments will not work for an individual patient.

The unlikely collaboration that started in that chance encounter has also paid off for cosmologists. The Durham team is using the tool it developed to understand dark matter, a mysterious substance that makes up about 85% of the universe. It’s mysterious because it’s so hard to detect; one method is by spotting when light from a star goes past dark matter and its gravity warps the path of the light – a phenomenon known as gravitational lensing.

The Euclid space telescope is scanning the cosmos, and data from its images can be crunched just like cancer histories, as visitors to the Royal Society’s summer science exhibition discovered this month.

Two significant advances, born from a random encounter at a shared work space. What were the chances?

I asked David Spiegelhalter that question. He is the eminent Cambridge statistician and author of The Art of Uncertainty, who once compiled more than 3,000 coincidences from members of the public to demonstrate that they were maybe a bit more common than people imagine. “This story reflects our experience that a major driver of coincidences and serendipity is when people who don’t know each other start talking,” Spiegelhalter said. “That’s why coincidences never happen to me: I don’t talk to anyone.”

Unlikely tales aside, stories such as dark matter-meets-cancer are all over science. In the course of writing this piece, I heard from Sarab Sethi, an ecologist at Imperial College London, who monitors biodiversity by tracking birdsong. It’s difficult to plant microphones and retrieve them in remote forests, so these areas of high biodiversity are understudied.

By chance, Sethi read about an electronics engineer making biodegradable sensors – “It sounded like science fiction,” he told me – but when he got in touch with Clementine Boutry, it turned into a deep collaboration that is almost at the point where biodegradable microphones can be dropped into the jungle then beam their results back to Sethi via a drone.

Another Imperial scientist, James MacDonald, had been working in the emerging field of designing artificial proteins using computers, an idea that made American biochemist David Baker a winner of the Nobel prize in Chemistry in 2024. His discovery opens up the possibility of entirely new types of material, which inspired MacDonald. By chance, he met a materials scientist at a networking event and, during the conversation, realised that their ideas overlapped.

Their collaboration led to the design of an entirely biodegradable material similar to silk but stronger, and their London-based company, Solena Materials, is now working with a US sportswear brand.

These stories invite a simple equation: two people + conversation = breakthrough. Arthur Koestler, the novelist and philosopher, gave it a name: “bisociation”. His 1964 book, The Act of Creation, holds that if you take two clashing concepts and rub them together, then you have the foundations of every joke ever told, every metaphor ever imagined and every … well, probably at least, a substantial number of scientific breakthroughs.

It’s a simple idea, but is it practical? Can we matchmake more physicists with cancer researchers, more ecologists with electronics engineers? Can we engineer serendipity?

“High-quality science is always looking for the new and the unexpected, and serendipity is a consequence of looking at things that are not always obviously related,” said Paul Nurse, president of the Royal Society, who in 2001 shared the Nobel prize in physiology or medicine for his work on genetics.

Scientists can get stuck in one area. “But you can exploit serendipity – in a more interdisciplinary environment.”

Getting mathematicians to rub alongside chemists and philosophers and computer engineers and oncologists every day seems more efficient than plonking people in a shared office space. When in 2010 Nurse was made the inaugural director and chief executive of the Francis Crick Institute, the UK’s leading biomedical research centre, he decided that it would be designed with serendipity in mind.

“We decided to simply abolish departments,” he said. The Crick’s 120-plus research groups are not under any structure. There are no divisions to separate people. Labs are shared. Communal spaces abound. “And the offices for group leaders are tiny – you can just about get two people in them. So when you have conversations, you’re forced out of the office.”

The result was the Crick’s gleaming tower of science, opposite London’s St Pancras International station (occasionally described as “Sir Paul’s Cathedral”). When the architect wanted to save money by removing a centrepiece, a four-storey spiral staircase, Nurse insisted it stayed. People won’t talk in a lift, but they will on the stairs, he told me. Watching people walk up and down, one half of a giant double helix, it’s hard to escape the thought that serendipity is literally part of the Crick’s DNA.

The Crick is not the only institution to try this – and by no means the first. Interdisciplinary working has been happening at least since the Second World War in some form. The Manhattan Project was a synthesis of maths, physics, chemistry and engineering. Building 20, an open-plan warehouse constructed during wartime at the Massachusetts Institute of Technology, was flexible enough for collaborations to happen by accident.

It probably inspired the owner of Bell Labs when it wanted to expand its engineering powerhouse in 1962. The labs, originally co-founded in the late 1800s by Alexander Graham Bell, had already invented the solar cell, transistors and the first communications satellite in the 1940s and 1950s. Bell Labs commissioned modernist architect Eero Saarinen to design a new institution at its Holmdel campus in New Jersey and asked him to create a gargantuan building, full of rooms for serendipitous encounters.

Since then, creative hubs and startup incubators have sprung up in any city looking for a slice of the knowledge economy: Station F in Paris, the Brooklyn Navy Yard in New York, the Factory Berlin in the German capital. Apple co-founder Steve Jobs wanted serendipity to be integral to its UFO-shaped headquarters in Cupertino, California, after whetting his appetite on a similar idea at his animation company Pixar. All are out-weirded by Amazon’s Spheres, three bubble-shaped greenhouses with 40,000 plants in which the retailer’s Seattle staff can brainstorm (alongside hundreds of visitors).

Imperial’s new White City campuses in west London are Crick-like in their rejection of divisions. Space is shared with businesses, which can use the lab equipment. Serendipity was “front of mind”, according to Richard Craster, dean of Imperial’s faculty of natural Sciences and a maths professor, who has spent more than 25 years collaborating with Omar Matar, a professor of chemical engineering, after they met during a break at a workshop in the millennial year.

It was, Craster points out, in an Imperial building that one of the most famously serendipitous discoveries was made. In 1928, Alexander Fleming noticed that mould growing on a petri dish at St Mary’s hospital had stopped bacteria from spreading, thus transforming medicine with the discovery of the first antibiotic: penicillin.

Less well known than Fleming’s eye for mould is that the world of science ignored his discovery for 10 years, until biochemist Ernst Chain and pathologist Howard Florey attempted to concentrate penicillin into something that might work as a medicine. They were not inspired by Fleming, however, but by René Dubos, a microbiologist who managed to isolate a different anti-bacterial agent in soil. The fact that soldiers were dying from infections in the Second World War made the matter all the more urgent.

Chain, Florey and Fleming were awarded the Nobel prize in physiology or medicine in 1945, and Fleming – by accounts a modest man– was asked during a speech afterwards whether he and previous winners had succeeded because of “deep thought or … Dame Fortune?”

Robert Hooke, who discovered and named the cell in 1665, had a similar thought. “The greatest part of invention being but a lucky bit of chance,” he wrote in 1679, having also discovered the law of elasticity, Jupiter’s great red spot, developed a theory of human memory, and observed that ammonites and other fossils might be evidence of extinct species.

Who needs serendipitous encounters when you have everything in your head? There used to be only one discipline – natural philosophy – and Francis Bacon had complained in 1620 that by “partitioning out” astronomy, optics, music, medicine and politics, they lacked depth. “We can little wonder that the sciences grow not when separated from their roots,” wrote the lawyer, spymaster and statesman, sometimes known as the father of empiricism.

Since disciplines such as physics, chemistry and biology were formalised under the university system in the 19th century, science has sped up. Now there are about nine million scientists around the world, marking out ever narrower territories.

“It is an issue,” Paul Nurse said. “It’s made even worse because of the amount of literature.”

Those nine million scientists published more than three million papers last year. How can anyone even stay on top of their own field, let alone another?

“I have six PhD students and a couple of postdocs,” Nurse told me. “My lab members know more about the recent advances in our area than I do.” The geneticist’s contribution is to read widely. “I pick up things – it could be astrophysics – and I suddenly think, ‘There’s something that could be relevant’, and we trade.

“If I want to know something about the enzymes we work on, I go and ask them, but I will come up with something weird; I look for the oddities. It leads to greater mutual respect in the laboratory. I’m not ‘Herr Doktor’ who knows everything.”

There’s an awkward question here, one that Bacon the empiricist might have asked: do attempts to engineer serendipity actually work?

Alexander Krauss, a research associate at the London School of Economics, is fascinated by breakthroughs. For his book The Engine of Scientific Discovery, Krauss analysed more than 750 key discoveries worthy of a Nobel prize or a place in the history books and looked at issues from serendipity to the type of university the scientist was linked to.

His analysis found that successful researchers tended to be younger and not at a top-tier university. Funding did not seem to matter much: there was no spike in discoveries after the Second World War, when money poured into science.

But Krauss did find that most Nobel prize discoveries went to a single researcher rather than a team, and a standout factor was that about half of all discoveries were made by scientists with at least two degrees in different fields. Polymaths still rule.

What about serendipity? This is not good news for interdisciplinarians: Krauss found that discoveries he classified as serendipitous had fallen from just over a quarter in the 17th century to only 11% from 2000 to 2022.

“Just placing people from different disciplines together is often insufficient,” Krauss said. Researchers needed a common problem, not just a common space.

Are all these cathedrals of collaboration pointless then? The Bell Labs modernist behemoth at Holmdel was not a nailed-on tale of success. The open spaces designed by Saarinen were mostly empty, which meant fewer serendipitous encounters, according to a New York Times report from 1972. Instead, employees used smaller side corridors, and Holmdel became a different kind of inspiration, as the home of Lumon Industries in the dystopian TV series Severance.

Perhaps we haven’t really had time yet for true interdisciplinary research to pay off. Most of the institutions remodelling themselves and their buildings to avoid silos have been at it for less than a decade. And Krauss’s research suggest these approaches are valuable. “They enable researchers to recognise opportunities,” he said.

He distinguishes between “encounters” between scientists and “capabilities”. “Many institutional initiatives focus on encounters,” he said. “But, historically, major advances often follow the use of new methodological capabilities.”

Tools, in other words. Krauss found that an overwhelming number of Nobel-level breakthroughs were driven by access to a new scientific tool.

Breakthroughs come, Krauss said, when “researchers gain access to novel tools and methods that allow them to ask questions that were previously impossible to investigate”.

Hooke discovered and named the cell because he had a much more powerful microscope. Galileo discovered Jupiter’s moons in 1610 because the telescope was invented in the Netherlands two years earlier.

Crick and James Watson discovered that DNA was a double helix because x-ray crystallography had been refined enough for King’s College London chemist Rosalind Franklin and her PhD student Raymond Gosling to create the image they used – known as Photo 51.

Modern gene-editing techniques and genomic sequencing have generated an enormous amount of research thanks to Jennifer Doudna and Emmanuelle Charpentier’s discovery of the Crispr tool. And, by extension, the absence of such tools to purify penicillin is why Fleming’s discovery was mostly ignored.

This offers an explanation for our “anecdata” as well. Sethi’s collaboration is all about inventing a new tool to study biodiversity. MacDonald’s new material derives from new tools that can design artificial proteins. And Nightingale and Griffiths were united by a desire to work out the best way to use a new statistical tool.

The optimum time for discoveries is about five to 10 years after a new tool is invented. “Institutions matter most when they enable researchers to build … and exploit new tool capabilities,” Krauss said.

So where does that leave the tool of the moment: the AI chatbot?

The history of science is built on people browsing through journals and books, coming across things by chance on a different shelf. Since digital searching became the primary method of discovery, the opportunities for serendipity are more limited.

“[Serendipity] is the complete opposite of large language models,” Nurse said. “They will give you a sensible account of the present state of research – mostly rather dull, but it saves you two weeks in the library. But you need the creativity of juxtapositioning different things, which they are not designed to do.”

It’s hard to let go of serendipity. There is something appealing about imagining ourselves in Fleming’s lab and wondering if we also would have noticed the mould killing the bacteria. Genius is unknowable; anyone can get lucky.

And if you can’t make your own luck, get an engineer to do it for you.

Photograph by Daily Herald Archive, Bettmann Archive