Experimentation teams are not short of ideas. Whether that is ideas raised in meetings, from the wider organisation or even AI tools spitting out hundreds of different ideas for teams to test. What they are really missing is the answer to this question: Is this idea worth testing?
Tests are running. A dashboard exists. Results get shared in a monthly email. Someone presents a win at the quarterly review. If you glance at it, you would conclude the organisation is experimenting and learning. Then a new CFO arrives, or a cost review lands, or a new product leader asks a simple question: "What has this changed?" And the programme cannot answer. Within two budget cycles, it is gone.
You hired capable people. You invested in testing tools. You wrote a playbook. You ran workshops. You created Slack channels, hosted demo days, built onboarding materials, and maybe even flew everyone to the same city for a two-day alignment session. And then you waited for it to work. But after some time, you notice a drift.
Vanity metrics create a dangerous illusion of progress while masking the fundamental question: Is your experimentation program actually driving better business decisions, or is it just generating activity?
A year ago, we changed our name from Effective Experiments to Efestra. At the time, I wrote that it was not just a name change. It was the beginning of a fundamental rethink of what we were building, who we were building it for, and how we believed experimentation governance should work. Today, that rethink is complete. This post is about what changed, what we held onto, and what Efestra looks like now.
The experimentation industry advocates democratisation as the path to maturity. But giving more teams access to testing tools without decision intelligence infrastructure does not create distributed learning. It creates distributed chaos.
Most organisations run their experimentation programmes on hope. Hope that teams will follow the playbook. Hope that insights will be shared. Hope that someone will connect experiments to strategy. A company invests in testing tools, hires specialists, maybe even writes a playbook. Then they hand it all over and wait.
How practitioner-level thinking turns strategic governance challenges into tool selection exercises and how this impacts the organisations that hire them
Recent posts
-
Introducing Brainstorm – the missing layer between ideas and tests -
Seven Signs Your Experimentation Practice Is Surviving, Not Thriving -
Why Workshops, Playbooks, and Good Intentions Will Never Scale Your Experimentation Programme -
Introducing Mosaic Companion: Because Your Best Experiment Ideas Happen in Meetings, Not in Backlogs -
Beyond Vanity Metrics: Measuring Experimentation Governance for Strategic Impact
