A brand I work with ran a top-of-funnel CTV test. Brand lift study, the full setup. It was planned for ten days. The placement shut it down after the control group, the people who never saw the ads, responded to the survey at a higher rate than the people who did.
So the room’s takeaway was clean: top-of-funnel CTV doesn’t work for us.
Except that test never asked a question about CTV.
Count what was actually new
Let me total up what was new in that “one test.”
A new creative concept, and only one of them.
A new format, a brand focused ad instead of the usual direct-response, USP-led style.
A new channel.
A new objective on that channel.
A new funnel structure
A new measurement instrument nobody had used before
That was never one test. It was a bunch stacked on top.
So the only thing we can confidently say is “This combination of things - did not show results on this test”
When we already have preconceived ideas like “CTV doesn’t work” these tests cement that belief. Now, I’m not a CTV advocate, not by a long shot, but I don’t think this test was anywhere near that conclusion.
A winner is a joint event
When you get a winning ad or channel, you did not prove a variable. You proved a combination.
The product, the angle, the creator, the offer, the format, the hook, all of it lined up on the same ad. The win is the overlap, not any one piece of it.
Which means your probability of hitting on any given test is basically the product of the odds on each variable you’re holding.
P(win) ≈ P(variable 1 is right) × P(variable 2 is right) × P(variable 3 is right)...
If a variable is proven, its term is close to 1. It barely drags the number down. If a variable is new, its term is well under 1, and it multiplies everything else.
Most people get a winner and then keep exactly one ingredient from it. Same product, brand-new everything else.
You need to know which ingredients are the most important.
Not all variables weigh the same
The variables carry very different weight. A change in product or angle can decide the whole ad. A change in audio quality? Less important.
If you made me rank them, top to bottom, it looks something like this:
Product
Angle
Creator
Offer
Key selling points
Filming Location
Ad Length / CTA
Audio Quality/Lighting (*more expensive =/= better here)
We’ve run regressions across a lot of these variables but it’s different account to account and can change over time as well (Getting a A+ creator or offer may jump it up the list)
So the real rule isn’t “change one thing.” or even “only make one big change at a time” (impossible) It’s know when you are changing multiple important variables and evaluate appropriately)
Quick Example
New creative on Meta, everything else proven, hits at something like 15%, with maybe a 5% shot at a real, big winner. Those are my numbers from my own accounts.
Now stack that YouTube test back up. New creative, call it 15% (generous for new). On a new channel, my gut says maybe 30% that top-of-funnel YouTube is even a fit. On a new strategy, maybe 10% you get an untested strategy right on the first swing. The channel and strategy odds are pure priors, not measured, so don’t hold me to them. But run the multiplication:
Under half a percent for any kind of win. For a big winner, swap in the 5% and you’re at 0.15%.
You ran a test with a sub-one-percent chance of a real result, it failed, and you filed it under “the channel doesn’t work.” Do that too often and you end up over concentrated.
Mistake 1: Changing too much & Learning too quick
Sometimes you have to change a bunch of variables. There are local maxima that you find through iterations, and absolute maxima that you find through jumping and changing a bunch of things
Small iterations can help you climb your local hill, but you have to make a big jump to find a new hill that’s even taller:
Let’s imagine the axes here are channel and ad type for the X and Z axes, and the Y axis is total performance
If you’re on the small hill, let’s call that Google. Then certain keywords and angles might work
But if you want to jump to the bigger hill, let’s call it Facebook ads. You may need a completely different selling point and ad type to make that work.
Don’t hear me say that changing a lot of variables is a bad idea and you should never do it. Rather, you just need to understand that if you’re going to change multiple variables at a time, your probability of success is low. You need to take multiple swings at bat before you make any conclusive decisions about any of those individual variables.
Mistake 2: changing too little & Giving Up
The second concern is making too small a change. Either making a change on a variable that doesn’t matter (like the background color of a static ad) which has very little impact and is not likely to change. If you’re running it on Facebook, it’s going to get grouped as the same ad anyway. It won’t get spend, and it will just fatigue the original winner faster if you force it.
Do your best to understand the most important variable when you find a winning concept. Then do tests where you keep that and change other variables, or keep all the other variables and change only that.
Control Variables, but only the most important ones. Marketing doesn’t have time for scientific rigor.
When you hold the proven variables
Even when you do hold the heavy variables, the hit rate moves a lot.
In one big account, I looked at creatives that called out a specific version of the product, tied to a partner most people would recognize. The product, the offer, and that selling point stayed fixed; we varied only the execution and the creator.
Reading the dashboard out loud, they still only had a 30% hit rate. Release the selling point, hold just product and offer: I think around 15%, still strong. Release everything, the account baseline across all products and all selling points: about 11%.
I’ve watched the same thing from the other side. Take two or three top ads, push them out to ten or twenty creators with one instruction: make your version of one of these. One or two win, the other eight or nine I don’t go back to. The concept is proven, so the only new variable is the creator & execution, which is exactly what I’m trying to isolate.
Why a low bar is the right bar
First, most of marketing operates on a power law, the top 1% drives the bulk of the performance. (i.e. not a standard distribution)
Second is that creative fatigue and competitors copy you.
You need to stay in front of everyone, which means you need to be constantly coming up with new ideas, new concepts, and new angles to beat that fatigue and stay novel.
If you’re constantly increasing the bar on yourself, you won’t see an increase in the percentage of hits because your overall quality will be going up. Instead, you’ll be raising the bar as you go for either higher spend per ad or higher return on ad spend with those ads.
That constant increasing of the bar of performance is what scales a company
What I actually do
Pin one, vary the other. But only for important stuff that is worth controlling. (Again this is not the rigor of a scientific test, we still need to move fast here). OR if you vary multiple variables, test multiple times (So if you vary creator + angle but test 3-4 angles and none hit? Then I feel good moving on from the creator but not the angles).
Port proven assets into new channels. If you want to test CTV, don’t build a brand-new asset on an untested angle for it. Take something already converting and keep as many of the learnings as possible.
When you can’t isolate, sequence. Instead of teasing four confounded variables out of one messy test, run 20% off versus 30% off. If 20 wins, next round is 20 versus gifts. One variable per round, winner carries forward. Slower is better than confused.
Run explore and exploit as two portfolios. When the hit rate drops and old ads have fatigued, pull back to proven variables and rebuild the base. When it feels healthy again, spend it back down on net-new.
My rule of thumb: at $100k a month, 70% new concepts and 30% variations. Past $1M a month you can go 50/50. Never below 50% new.
Change the metric when you change the heavy variables. A brand-new pain point is not going to convert at your six-month-old CAC on day one. If the thumbstop and engagement rates are in line with your BAU creative, and CPA is within 30%, then keep it running and optimize the page while the audience warms up.
Read aggregates, not verdicts. Tag everything MECE, one and only one answer per field: static or video, what’s the hook, what’s the selling point, what’s the problem. Then roll it up. You’ll find a whole category, say educational awareness ads, is carrying you, which is a real finding. One ad dying is not.
Learn from individual wins, learn from aggregate losses.
The certainty isn’t late, it’s unavailable
A founder told me recently he was frustrated he couldn’t find where he had legs, couldn’t figure out the next ad to produce. The honest answer is that you never get there. The moment you know exactly what the next winning ad is, the bar goes up, you scale spend 50%, and now the bar is higher and the challenge is one level harder. The certainty you’re waiting for is what moves the bar and now you are in uncharted land again. Congratulations.
Considering testing a new channel?
Write down everything you are changing and evaluate the probability of success first.
Then produce the right number of tests to give a fair chance of winning BEFORE you start.
Cheers,
Curtis