I have worked in two fields that share a problem: the quick wins are gone.

The first is particle physics.

I did my PhD at Imperial College London and at CERN.

I was lucky.

I was there when the Large Hadron Collider, the machine built to find the Higgs boson, was switched on for the first time. There was electricity in the air. But there was also an accepted truth in the room: the golden age was already behind us.

One generation earlier, the field had been buzzing with discoveries. In the 1970s, having two or three discoveries in your thesis was normal. Me, when I finished my thesis in 2011, it contained no new discoveries. The same was true for hundreds of people in my cohort. The Higgs boson was confirmed experimentally in 2012. One year after my thesis, and roughly fifty years after it was first predicted.

For my generation, particle physics had left the era of quick wins.

The second field is data.

In 2015 I left physics to enter the world of enterprise data as a consultant. This time I was in the right place at the right moment. Tableau had gone public two years earlier. Self-service analytics and BI tools were everywhere, and everyone believed that data was going to transform every company on the planet.

I watched analytics become a function in its own right, separate from IT. Head of Analytics and Chief Data Officer became formal titles. I knew people who were hired into those roles. The bet behind them was simple: data would grow revenue, improve profitability, and pay back the investment many times over. For a good while, it worked. The low-hanging fruit was everywhere and easy to reach.

These past months I spoke with three data leaders, and their day-to-day has changed. It used to be about insights, about what you could build with the data. Today, more and more, it is about costs, headcount, and platform budgets. And as if that were not hard enough, CEOs are now asking their CDOs to "do AI," without anyone being sure how to do it or how to measure its value.

So if you are a data leader today, I would bet you feel this pressure.

Particle physics has felt it forever.

Let's take a leaf or two, from it's playbook.

1. The hamster wheel.

Particle physics is a very data-hungry field. The LHC generates so much data that we physically cannot store it all. Our answer was to become selective. In 2008, my first task was to write an algorithm that rejected more than 95 percent of our data, in real time, as it came off the detector. We could not keep everything, so we had to use what we already knew with the existing research, to estimate what an interesting collision looks like.

We kept only what was interesting and rare, plus a small data sample of ordinary collisions to calibrate the analysis.

Contrast that with my consulting career.

In 2016, my main project was a pipeline that took eight hours to run and produced millions of rows. I brought it down to forty-five minutes after four weeks of work, still producing millions of rows. In 2024, a client wanted to process twenty billion rows, but the process never reached the end. I optimised the pipeline so that it ran in under two hours.

2. Data, Data Everywhere, and Not a Join in Sight.

There is another problem shared by both fields, and paradoxically it is a lack of data.

Let me explain.

In physics, we have known for a long time that a large part of the universe is not directly observable. We call it dark matter. We know it exists because we see its imprint on the phenomena we can observe. What do we do about it? We create research programs and projects to fill that gap in our understanding.

Organisations have dark matter too.

I worked with a client who wanted to understand the flow of work across their business. Their business ran on seven or eight siloed applications. I spent six weeks building all the pipelines. When we connected everything and analysed it, we found that the identifiers that would have linked the work from one system to the next were simply never recorded. Work was clearly passing from one application to another. The business kept running. But that link was never captured.

But wait. Physics has first principles.

Looking at these two examples, you might object that physics has something the enterprise does not: first principles, the universal laws that guide which data to keep and which data to go looking for.

Yes. And in the enterprise, you also have first principles. They are your subject matter experts.

You frown.

But is it enough to put your subject matter experts in a room with a data engineering team, add resources and infrastructure, and expect insights that create value?

I agree. That alone will not do the trick.

Nobody can guarantee the ROI, and nobody wants to fail. The result? No one takes the risk.

The biggest lesson from physics: the scientific method.

This is where the biggest lesson from my experience in particle physics comes in. The field, while being expansive and competitive, kept making progress long after the easy discoveries were gone. What makes that possible is the scientific method and the experimental mindset.

In simplified form, here are the five steps to use the scientific method in a business setting:

1. State clearly what you believe, i.e. your hypothesis.
2. Decide in advance what would prove you right, or wrong.
3. Decide how you will pull the plug if the success conditions are not met.
4. Run the experiment.
5. Treat a null or negative result as progress, not as failure.

This is how physicists all over the world stay motivated to keep pushing forward.

I can already hear the objection: in a corporate setting, experimentation is hard to get accepted.

I agree.

Experimentation is only possible with ownership: owning the hypothesis, and owning the conditions of its success or failure. Led with ownership, experimentation becomes responsible, because we decide from the very start when to pull the plug, without needing to convene four steering committees.

For the skeptics among you, a real example:

I worked with a client on optimising their processes.

First, we generated the hypothesis. Subject matter experts believed that part of the process was unnecessary, and that removing it would cut costs for the business and eliminate redundant steps for their customers.

Next, the criterion. They decided what would prove them wrong: any significant drift in customer behaviour.

Then, the pull-the-plug process. If drift appeared, the old process would be restored.

Then, the experiment. They cut that part of the process. To watch for drift, they kept a random sample of customer journeys.

Finally, the result. On the business side, costs went down, as expected. On the customer side, there was no drift: a neutral result, and exactly the one they were hoping for.

Why experimentation matters even more with AI.

The experimental mindset becomes even more important with the explosion of AI.

With AI, building costs almost nothing. Anyone can create a new model, a new process, or an automation in a single afternoon.

That is wonderful.

And also terrifying.

If everyone rushes into new projects, the organisation will drown in half-useful tools bolted onto every corner of the business... in complexity that adds no value.

TLDR;

The quick wins are gone.

The scientific method gives you a framework to resolve the tension between practicality and exploration.

Concrete to-dos for data leaders:

1. Be selective. Do not let yourself drown in data. Choose what you analyse.
2. Be proactive. Go and find your organization's dark matter. It hides a few skeletons, but it also hides real value worth uncovering.
3. Take the lead on AI. Shape how AI projects get framed. Encourage creativity and experimentation. Manage risk with ownership and up-front criteria.