Let’s face a harsh truth your CFO likely knows. Most of your marketing “success” is just a mirage. It looks real but isn’t.
You spend a lot on marketing channels that take credit for sales that would happen without them. It’s like saying rain dances cause rain.
The Uber lawsuit showed a big problem. It exposed a widespread belief that’s not true. Performance marketing often just catches traffic that was coming to you.
Last-touch attribution is like thinking the last person you talked to made you alive. Our economy loves quick, easy-to-track results. But these results often show correlation, not cause and effect.
True incrementality is the real change you make. It’s the difference between what happened with your campaign and what would have happened without it. It’s about making new fans, not just catching existing ones.
To break free from this trap, we need to use control groups in real-world marketing labs. Geo split experiments and lift studies are key tools here.
Think of them as the voice of reason at a party. They cut through the noise to show what your money really bought.
Test Design for Seasons and Playoffs
Running a marketing test in a quiet time is like practicing golf on a range. But doing it during big events like the NBA Finals is like trying to sink a putt with everyone watching. The situation changes a lot.
Things like buzz, competitor actions, and weather can mess up your test. You need to learn to handle these challenges.
Traditional control groups don’t work well in these situations. Finding a similar market is hard, as everyone is caught up in the excitement. The synthetic control offers a solution.
It’s like creating a digital copy of your test market. You mix data from different markets to match your test market’s history. This way, you can see how your campaign would have done without the playoffs.
When the playoffs start, you compare your real market with its digital twin. The difference shows how well your campaign did, apart from the playoffs. It’s like having a marketing time machine.
Creating this setup needs careful planning. You must decide how to run your test, whether it’s a big push during one game or a steady effort over several games. Each choice affects how much impact you can have.
You also need to protect your test from outside influences. News stories or tweets can affect your results. Your test design must handle these surprises well.
This method isn’t about predicting the future. It’s about doing a solid test in a chaotic time. With synthetic control, you can really understand your ad’s impact, even when it’s hard.
Sample Size & Power in Event-Driven Markets
Let’s talk about your data honestly. Getting a “lift” from your campaign test is exciting. But was it real, or just a lucky streak?
In the world of sports, a lift study needs strong power. Without it, the noise is too loud. Your insights might just be echoes of fan feelings.
Statistical power is key. It’s the chance to find a real effect if it exists. A weak test is like a blurry camera.
An underpowered lift study is risky. It might show a false positive. You might think a 5% bump is real, when it’s just a momentary high.
Analysts warn about the daily report pressure. It can lead to chasing false signals. This wastes budget and misses the real story.
A strong test is like a clinical trial. It’s about precision, not guesswork.
You wouldn’t approve a drug with just five patients. Marketing spend needs the same careful planning. Decide the smallest effect you want to see before starting.
With your Minimum Detectable Effect (MDE) set, calculate your sample size. This number is often bigger than you think, due to the messy nature of event-driven markets.
A small sample might catch big lifts but miss small, important ones. It’s all about the size of the effect you’re looking for.
An underpowered test is bad news. It leads to:
- False Positives: Mistaking noise for signal.
- False Negatives: Missing a good campaign because you can’t measure it.
- Strategic Paralysis: A cycle of tests that go nowhere.
This forces a choice: duration versus purity. You need a longer test for a bigger sample. But longer tests are more likely to be affected by outside factors.
Finding the right balance is key. A short test is pure but weak. A long test is strong but might be tainted by external factors.
The solution is math, not magic. You find the shortest test duration that’s powerful enough to detect your MDE, while considering future events.
This approach is honest and effective. It means sometimes saying, “We can’t know that yet.” It leads to real, lasting business impact.
Execution on Paid Social/CTV/OOH
Marketing theory is like a clean lab. But execution is a messy battlefield where algorithms fight for survival, not your profit. Here, your geo split design meets the real world of platform incentives. Meta, TikTok, Google focus on their own profits, not your success. You must make them show their worth.
Paid Social is like a digital battlefield. Running a geo split test here is like trying to herd cats. Algorithms look for converters everywhere, ignoring your geographic lines.
The key isn’t to fight the AI. It’s to hacking it. Kevin Goodwin’s playbook shows how: change your optimization window. Use a longer conversion window instead of last-click. This makes the algorithm value brand building over quick sales. Your test cells start to act differently.
Your social media geo split checklist should include:
- Broad over narrow: Avoid hyper-retargeting. Use broad interest targeting. Let the algorithm find your audience within the test region, not your pixel.
- Creative built for the platform: Your “brand” video can’t be a TV spot. It needs to be built for the scroll, not the couch.
- Beware the AI tools: Platforms keep rolling out new tools like Pmax or ASC. These tools are great at finding patterns but bad at proving what works. Sometimes, you need to turn them off to see the truth.
Connected TV (CTV) is a unique challenge. You’re running a digital ad on a streaming service, but the impact often happens offline. How do you measure that?
Geo-experimentation is your answer. CTV is perfect for a geo split test. Run your campaign in specific DMAs. Measure lift in app installs and overall brand health in those regions. The platform can’t claim credit for organic search spikes in your treatment group. That’s pure incrementality.
Out-of-Home (OOH) is all about physical ads. Billboards, stadium ads, transit wraps. Here, geographic design is key. You treat different DMA regions as your test subjects.
One region gets the new creative. Another is the control. OOH exposure is naturally geographic, making your geo split clean. The challenge? Attribution. You’ll need to use mobile location data or brand survey lifts to connect the billboard to the behavior.
The common thread across all channels is honesty. You must design every campaign to answer one question: “What happened in the test regions that wouldn’t have happened anywhere else?”
Paid Social needs its optimization windows tweaked. CTV needs to capture its offline impact. OOH needs to use geographic isolation. Each channel’s geo split execution is different, but the goal is the same: make the channel prove its value, not just its reach.
Execution is messy. Algorithms are selfish. But with the right tactics, you can get the truth to show up. Even on a battlefield.
Readouts and Decision Rules
You’ve made it through the tough part of testing. Now, you face the hardest part: making sense of the data. The numbers are waiting to tell a story that fits the company’s goals. This is where marketing science meets corporate politics.
Most teams make a big mistake here. They only look at the treatment group’s results. Did sales go up? That’s all they care about. But that’s like watching only the hero’s scenes in a movie. You miss the whole story.
The real story is often in the control group. Did it stay the same, just like your synthetic control model said? Or did it drop, showing your ads affected it? That’s not creating new demand—it’s just moving it around.
Let’s get real. Imagine your graph shows Vendor 1 had a 12% lift. That’s great! But look closer. Vendor 3, in your control group, dropped by 5%. That’s not good. You’re taking from one to give to another, and calling it growth. Our guide helps you see this clearly.
This is where your synthetic control shines. It shows what would have happened without your campaign. If your results are way above this, you’re onto something. If they’re close, it’s just noise.
The biggest challenge isn’t just stats. It’s keeping everyone calm. Teams want to change everything every day. Your job is to keep them steady.
How? Set your decision rules before you see the results. This makes your findings clear and fair.
Create a simple matrix. Define your rules:
- Confidence Level: Will you accept 90% confidence, or do you need 95%? Set this in stone.
- Minimum Incremental ROAS: What return justifies continued investment? Is it 1.5:1? 2:1? Know your number.
- Action Triggers: “If result X with confidence Y, then action Z.”
Here’s what that matrix might look like in practice:
| Result Scenario | Confidence Level | Incremental ROAS | Prescribed Action |
|---|---|---|---|
| Strong Positive Lift | >95% | >2.0 | Double down on budget; expand test. |
| Moderate Positive Lift | 90-95% | 1.5 – 2.0 | Continue with measured optimization. |
| Flat or Negative | Any | Pause campaign; investigate design. | |
| Positive but Spillage Detected | >90% | Any | Re-evaluate targeting; fix control group contamination. |
This framework stops the rush for quick fixes. It helps your team focus on what’s truly important. You’re not chasing shadows in the data.
Now, the diplomacy part. You must explain these complex results to stakeholders who want simple answers. They want a hero story. You might have a more nuanced tale.
Explain statistical significance in business terms. “We’re 95% confident this worked” means “The risk of being wrong is 1 in 20.” Show them how incremental ROAS fits into the bigger picture of marketing spend. Highlight how ghost ads and false have skewed their view.
Remember: a “no result” is a result. It tells you what not to do. That’s valuable knowledge. Protecting budget from bad channels is a positive return.
The synthetic control output is your proof. It shows the gap between what was predicted and what happened. Use it. Point out the gap. That gap is your value—or your expensive lesson.
In the end, a careful readout process does more than measure a campaign. It builds your team’s muscle. It teaches them to think in probabilities, not certainties. To value evidence over instinct. That’s the real incrementality—not just in sales, but in wisdom.
Building a Continuous Test-and-Learn Program
A single geo split shows what worked yesterday. But a series of lift studies predicts tomorrow’s success. This is the difference between a snapshot and a movie.
Kevin Goodwin’s stair step method is key. You don’t jump to big changes all at once. Start with one solid study, then another. Soon, you’ll have a steady flow.
Your aim is to make this test-and-learn mindset a part of your company. Do you need a dedicated team or a smart partner? What matters most is their commitment.
The magic happens when you learn from your tests. Have a central place for sharing findings. Celebrate both wins and failures. Failures often teach more than successes.
This turns marketing into a growth engine, not just a cost center. A true always-on campaign needs constant measurement. Your studies show how well you’re doing over time.
The game never stops, and neither should your curiosity. Move from just running campaigns to always learning. That’s how you grow step by step.