How practitioner-level thinking turns strategic governance challenges into tool selection exercises and how this impacts the organisations that hire them
The Pattern
Over the past month, I’ve had nearly identical conversations with experimentation specialists at three very different enterprise companies: a fast-growing gaming platform, a global entertainment conglomerate, and a major financial services firm.
Different industries. Different scales. Different maturity levels.
Yet every conversation followed the same script. Each specialist clearly articulated the chaos in their experimentation programme. Each had diagnosed the problem. Each had identified what they believed was the solution.
And each was completely, fundamentally wrong about what they actually needed.
Not because they weren’t smart or experienced. But because they were solving the problem they could see from their organisational level—whilst the real problem existed several layers above them.
Here’s what they said:
Conversation 1: Gaming Company “We need better tracking. Our experiments are scattered across Jira, Notion, and spreadsheets. If we could just consolidate everything into one place, we’d solve our governance problem.”
Conversation 2: Entertainment Company
“It’s a documentation issue. We have 40 individuals running experiments with no coordination. We need software to track what everyone’s testing so we can prevent conflicts.”
Conversation 3: Financial Services Company “Our data is fragmented across multiple tools. We need to centralise our experiment information so leadership can see what we’re doing.”
What They All Missed
They diagnosed fragmentation when the real problem is lack of authority.
They prescribed consolidation when what’s needed is governance.
They thought they needed better tracking when what executives actually need is strategic confidence.
The Practitioner Blind Spot
Here’s what experimentation specialists see:
The Symptoms:
- Information scattered across tools
- No single source of truth
- Difficulty finding past experiments
- Teams running conflicting tests
- Leadership asking “what are we testing?”
The Diagnosis: “We have a data fragmentation problem.”
The Prescription: “We need software that consolidates everything into one place.”
The Expected Outcome: “Once everything is in one system, our governance problems will solve themselves.”
What Leadership Executives Actually See
Meanwhile, leadership is looking at the same organisation and seeing something completely different:
Their Symptoms:
- Significant experimentation investment with unclear ROI
- Test results that don’t translate to business decisions
- Low confidence in experimentation outcomes
- No connection between testing activity and strategic objectives
- Knowledge disappearing when team members leave
Their Diagnosis: “We lack strategic oversight of our experimentation programme.”
What They Need: “Confidence that experimentation is driving strategic value, not just activity.”
The Fundamental Misunderstanding
Practitioners think governance = organisation “If we just organise our experiments better, we’ll have governance.”
Reality: Governance = authority + accountability + decision frameworks Organisation is table stakes. Governance is what happens when you enforce quality standards, align testing with strategy, and track whether experiments actually drive decisions.
The Tell-Tale Signs You’re Thinking Like a Practitioner
You say: “We need to consolidate our experiment data”
Translation: You think visibility solves governance
You say: “We need better documentation”
Translation: You think recording activity equals strategic value
You say: “We need to prevent conflicting tests”
Translation: You think coordination is the problem, not strategic alignment
You say: “Leadership needs to see what we’re testing”
Translation: You think transparency equals trust
Real Examples: Where Practitioner Thinking Fails
Case Study 1: The Gaming Company (120 Experiments, Zero Governance)
What the specialist saw: “We have experiments in Amplitude, Jira, Notion, and FigJam. Our problem is fragmentation. We need one system.”
What was actually happening:
- 50+ people running experiments with no quality standards
- No strategic alignment framework
- Director/VP level pressure for ROI visibility
- Teams testing random ideas without proper hypotheses
- No decision criteria or implementation tracking
The practitioner prescription: “Let’s consolidate everything into one tracking tool.”
What they actually needed:
- Executive mandate for experiment quality standards
- Strategic alignment framework connecting tests to business goals
- Decision protocols requiring proper hypothesis and success criteria
- Implementation accountability (did successful tests get rolled out?)
- Knowledge management system that survives team changes
Tracking tool solves: Finding past experiments
Governance solves: Ensuring experiments are run to the highest quality and drive strategic decisions
Case Study 2: The Entertainment Company (40 product managers running tests , Chaos)
What the specialists saw: “We have 40 PMs running experiments with no coordination. Regional teams are running conflicting tests. We need visibility.”
What was actually happening:
- No authority structure for experimentation
- Regional vs global tension with unclear decision rights
- Senior stakeholders demanding changes based on opinion
- Frequent leadership changes creating instability
- Teams fighting over limited testing traffic
The practitioner prescription: “We need tracking software so teams can see what others are testing.”
What they actually needed:
- Executive-level authority defining who can test what and when
- Decision rights framework (regional autonomy vs global standards)
- Quality gates that stakeholders can’t bypass with opinions
- Strategic prioritisation replacing political prioritisation
- Change management to establish new operating model
Tracking tool solves: Preventing duplicate tests
Governance solves: Establishing authority and decision rights
Case Study 3: The Financial Services Company (Fragmented Data)
What the specialist saw: “Our experiment data lives in multiple systems. Leadership can’t see what we’re doing. We need consolidation.”
What was actually happening:
- Experiments running without strategic justification
- No connection between tests and business objectives
- Results not influencing actual business decisions
- Compliance concerns about experiment documentation
- Board asking “what’s the ROI of our testing programme?”
The practitioner prescription: “Centralise experiment data so executives can see our testing activity.”
What they actually needed:
- Strategic alignment framework showing how tests connect to business goals
- Decision tracking showing which experiments drove actual changes
- ROI measurement showing business impact, not just test results
- Compliance documentation for regulatory requirements
- Executive dashboard showing strategic value, not testing volume
Tracking tool solves: Making activity visible
Governance solves: Making activity strategically valuable
Why This Keeps Happening
Experimentation specialists keep making this mistake because they’re solving the problems they can see from their vantage point.
From practitioner level, you see:
- Operational inefficiency
- Coordination challenges
- Information scattered
- Workflow friction
From executive level, they see:
- Strategic misalignment
- Unclear ROI
- Low decision confidence
- Organisational capability gaps
The trap: You can’t solve executive-level problems with practitioner-level solutions.
The Three Questions That Reveal the Real Problem
Next time you think “we just need better tracking,” ask these questions:
Question 1: “If we consolidated everything into one system tomorrow, would executives have more confidence in experimentation results?”
If answer is NO: You don’t have a tracking problem. You have a quality and decision-making problem.
Question 2: “Can we currently prevent poor-quality experiments from running?”
If answer is NO: You don’t have a documentation problem. You have an authority problem.
Question 3: “Do we know which experiments actually changed what the business does?”
If answer is NO: You don’t have a fragmentation problem. You have a strategic impact problem.
What Governance Actually Requires
Real experimentation governance isn’t about consolidating data. It’s about:
1. Strategic Alignment Every experiment must connect to business objectives. Not “testing button colours” but “validating hypothesis that simplified checkout reduces cart abandonment, supporting Q4 revenue goal of £X.”
2. Quality Enforcement Standards that experiments must meet before they run:
- Proper hypothesis structure
- Statistical rigour requirements
- Clear success criteria and decision thresholds
- Implementation plan if successful
3. Decision Frameworks Pre-defined criteria for what happens with results:
- When do we implement?
- When do we iterate?
- When do we abandon?
- Who has authority to decide?
4. Implementation Accountability Tracking whether successful experiments actually get rolled out:
- Did the winning variation go to production?
- Did it deliver the projected business impact?
- What did we learn for future experiments?
5. Knowledge Management Institutional learning that survives team changes:
- Why we tested this hypothesis
- What we learned beyond just “winner” or “loser”
- How insights apply to future decisions
- Strategic implications for the business
“consolidating data into one place.” will help but it requires executive authority and systematic enforcement.
The Budget Trap: When Investing In Governance Seems “Too Expensive”
Here’s a revealing pattern from recent conversations:
Scenario 1: An SVP at a major financial institution reviews experimentation governance proposal. His response: “This seems expensive.”
Context: The bank spends £3M+ annually on experimentation tools, teams, and infrastructure. But paying for a governance system “seems expensive.”
Scenario 2: The Gaming Company
Practitioner builds business case for governance platform. Finance asks: “Which department budget does this come from?” Nobody knows, because experimentation spans 6 product tribes.
Scenario 3: The Entertainment Conglomerate Specialist proposes governance solution. Leadership response: “Get three quotes and we’ll evaluate.” They’re treating strategic transformation like purchasing office supplies.
What This Reveals
When someone thinks investing is “too much” for governance whilst spending millions on experimentation, they’re revealing two things:
1. They don’t understand what they’re buying
They think they’re buying “experiment tracking software” (departmental tool, operational expense).
They’re actually being offered “strategic oversight of multi-million-pound experimentation investment” (organisational capability, strategic investment).
2. They don’t have the right budget authority
Department-level budgets are for tools and software licenses. Organisation-level budgets are for strategic capabilities and transformation programmes.
Governance isn’t a department purchase. It’s an organisational investment.
The Budget Authority Test
Question: “Where would the budget for this come from?”
Practitioner answer: “I’d need to find budget in my team’s allocation…”
Translation: Wrong buyer. Department-level authority for organisation-level problem.
Executive answer: “This would come from our strategic initiatives budget…”
Translation: Right buyer. Organisational authority for organisational transformation.
Why Governance Requires Organisational Budget
Department-level budget is for:
- Tools used by specific teams
- Software licenses for team members
- Operational improvements within department
- Tactical efficiency gains
Organisation-level budget is for:
- Strategic capabilities spanning multiple teams
- Transformation programmes requiring change management
- Cross-functional initiatives with executive sponsorship
- Investments that affect how the entire organisation operates
Experimentation governance affects:
- Product teams running experiments
- Engineering teams implementing results
- Analytics teams measuring outcomes
- Leadership teams making strategic decisions
- Knowledge management across the organisation
You cannot fund organisation-level transformation with department-level budget.
The Real Cost Comparison
When investing in a governance system seems “too expensive,” here’s what they’re missing:
Annual Experimentation Investment:
- Testing tools and platforms: £200K+
- Engineering time on experiments: £500K+
- Analytics and data science: £300K+
- Product team time: £400K+
- Opportunity cost of poor decisions: £1M+
Total annual investment: £2M+
Governance cost: £25K (1.25% of total investment)
Would you invest £2M without spending 1.25% to ensure that investment delivers strategic value?
It’s like spending £800K on a house but refusing to pay £10K for a structural survey because “it seems expensive.”
The Uncomfortable Truth
You cannot implement experimentation governance from practitioner level because governance requires organisational authority you don’t have.
You can:
- Build excellent tracking systems
- Create comprehensive documentation
- Demonstrate best practices within your team
- Advocate for better processes
You cannot:
- Mandate quality standards across the organisation
- Enforce strategic alignment requirements
- Establish decision rights and authority
- Allocate resources to governance vs execution
- Require executive visibility and accountability
- Access organisation-level budget for strategic initiatives
This isn’t a criticism. It’s organisational reality.
Governance is a top-down organisational capability, not a bottom-up tool selection decision.
And it requires organisation-level budget authority, not department-level procurement.
What To Do Instead
If you recognise yourself in these patterns, here’s what actually works:
Step 1: Reframe the Problem
Stop saying: “We need better experiment tracking”
Start saying: “We lack strategic oversight of our experimentation investment”
Stop asking for: “Software to consolidate our data”
Start asking for: “Executive sponsorship for experimentation governance”
Step 2: Quantify the Strategic Problem
Calculate what poor governance is costing:
- Wasted experiment investment (£___ annually)
- Lost knowledge when people leave (£___ annually)
- Delayed decisions due to low confidence (£___ in opportunity cost)
- Duplicate work across teams (£___ annually)
Present this to executives in their language: ROI, risk, strategic capability.
Step 3: Get Executive Sponsorship
Governance requires executive mandate. Find a VP or C-level sponsor who:
- Has authority to mandate standards across teams
- Cares about experimentation ROI and strategic alignment
- Can allocate resources to governance implementation
- Will champion the transformation organisationally
Without executive sponsorship, you’re building solutions that people can ignore.
Step 4: Bring in External Expertise
Experimentation governance is a specialised discipline. Partner with firms like Efestra who:
- Have proven governance frameworks
- Can present to executives in strategic terms
- Bring implementation methodology and best practices
- Take accountability for transformation outcomes
This isn’t admitting defeat—it’s recognising that organisational transformation requires external credibility and authority.
The Real Difference
Practitioner solution: “Let’s buy software that tracks all our experiments in one place.”
Governance solution: “Let’s establish executive-mandated frameworks that ensure every experiment drives strategic value, with systematic enforcement and decision accountability.”
Practitioner outcome: Better organised chaos. You can now find past experiments easily, but experiments still don’t drive strategic decisions.
Governance outcome: Strategic capability transformation. Fewer experiments, higher quality, clear business impact, institutional learning, executive confidence.
Stop Solving Tracking Problems. Start Solving Governance Problems.
If you’re an experimentation specialist who recognises these patterns, you have two choices:
Choice 1: Keep pushing for better tracking tools Build excellent systems that organise experiments beautifully whilst the fundamental governance problems remain unsolved. Stay frustrated that leadership doesn’t value your work.
Choice 2: Reframe as governance transformation Help executives recognise the strategic opportunity. Get executive sponsorship. Implement real governance with authority and accountability. Transform experimentation from tactical activity to strategic capability.
The tracking tools are easy. The governance transformation is hard. But only one creates lasting strategic value.
The question is whether you’re ready to stop solving practitioner problems and start solving executive problems.
About Efestra: We provide the experimentation governance platform that transform testing activity into strategic business advantage. Unlike tracking tools that organise experiments, we provide governance frameworks that ensure experiments are run to the highest quality standards and actually drive business decisions.
