A measurement tool is only as useful as the question you ask it.
A ruler, for example, is very useful if you want to know how long your coat sleeves are, but is of little benefit if you need to know whether you need to wear it. A thermometer, on the other hand, will answer the latter question but is of no help determining if the coat is going to fit.
The importance of directing a question to the appropriate instrument seems clear in these examples, and yet when it comes to statistics, in advertising we typically look at the fuel gauge to determine a car’s speed. Nowhere is this more painfully felt than when relying on determinations of statistical significance as the basis for making adjustments to improve live campaigns. The pain lies in the fact that, like using a barometer as a bathroom scale, statistical significance does not answer the question being asked when optimizing a campaign.
The question asked when optimizing is this: “Is this media tactic lifting or pulling down overall campaign performance?” The question answered by statistical significance metrics, however, is quite different. Unfortunately, the industry simply has not had the right measurement instrument to answer the question being asked by advertisers optimizing their campaigns. That is, until now.

Statistical Significance and P-Values

The concept of statistical significance is based on the calculation of a statistic known as a p-value, specifically, whether the p-value exceeds a threshold such as 0.05. The p-value is critical within scientific research communities as a way to partially validate hypotheses, and thus focus subsequent research to a narrow set of more promising hypotheses. Unfortunately, this statistic is too often misapplied when interpreting advertising campaign performance. The assumption is often that if a result is statistically significant, or “stat sig” for short, then you can be confident that it is true and therefore act accordingly. In reality, however, the p-value answers a very different and somewhat counterintuitive question.
Take this scenario for example. You are running a direct buy campaign across two publishers. One month in, the results show Publisher A way in front with a brand lift of 7% to Publisher B’s 2%. Determining the p-value only tells you what the chance is that Publisher A would show a lift of 7% or great if, in reality, it wasn’t actually driving a lift at all. Let that sink in. It doesn’t tell you that Publisher A is driving a 7% lift and it definitely doesn’t tell you if you should shift budget away from Publisher B to Publisher A. The p-value only tells you how confident you can be that 7% isn’t actually zero.

Upwave Optimization

As the leading analytics platform for brand marketers, Upwave has developed a game-changing measurement instrument for making mid-flight campaign adjustments. The Upwave Optimization dashboard refreshes daily to reveal which targeting tactics – from publisher to audience to channel, geography and more – are most likely to improve or reduce overall campaign performance. This indicator is called the Outperformance Indicator. In the above scenario with Publisher A and B, for example, the Upwave Optimization dashboard answers the urgent question that demands a real-time answer: which publisher is most likely to be helping drive overall brand lift?

The Right Tool for the Job

Despite being the industry standard for so long, p-values are simply the wrong tool when it comes to mid-campaign measurement.
P-values help advertisers answer the question, “what is the probability that this tactic drove a lift greater than zero?” when in reality, advertisers need to know “is this tactic driving more lift than the campaign average?” Is it a meaningful distance from zero, and how does it compare to other tactics?
Upwave Optimization takes into account the combination of how much data you have for each tactic, and the confidence you have that each tactic is or is not driving higher lift for the campaign as a whole. Whereas p-values can be a useful insight in a post-campaign report, optimization allows advertisers to make mid-campaign adjustments. The Outperformance Indicator thus provides marketers with the same actionable insights that have long been used in performance campaigns.
Stay tuned for additional insights into this campaign measurement innovation over the coming weeks leading up to the official launch of Upwave Optimization in March.