We use cookies to personalize content, to provide social media features and to analyze our traffic. We also share information about your use of our site with our social media, advertising and analytics partners. For information on how to change your cookie settings, please see our Privacy policy. Otherwise, if you agree to our use of cookies, please continue to use our website.

Does Statistical Significance = Business Relevance?

Marketing teams rely on marketing sciences teams to measure and optimize promotion spend. The Marketing teams use the insights from marketing sciences to make business decisions and improve promotion performance. 

However, we have noticed recently that when marketing sciences teams produce their results, they are starting to emphasize statistical significance with little to no focus on business relevance. Let’s define these two terms as follows:

  • Statistical Significance means that the measured result occurred and is not a random result. For example, if a brand spends $50 million on a television ad campaign, and the measured result of the ad campaign is statistically significant, then we can be confident that the campaign had an effect in the marketplace.

However, statistical significance says nothing about the magnitude of the effect. For that, we need to quantify the business relevance.

  • Business Relevance means that the magnitude of the effect is large enough to be important to the business owner. For example, if the $50 million television campaign generated an incremental $500 million in sales, then that campaign would be considered to have had a very relevant impact on the business. Alternatively, if the $50 million television campaign generated an incremental $1 million in sales, then most marketing teams would conclude that the campaign had little business relevance.

Continue the example. A marketing sciences team that reports a statistically significant impact from a $50 million dollar television campaign – and that’s all the information reported – will likely cause the brand team to invest another $50 million to continue the campaign. The mistake here, of course, is assuming that statistical significance is the same thing as business relevance. It’s not.

Our recommendation is that marketing sciences teams should provide results that answer both statistical significance and business relevance questions. If this is done, then the marketing team will know that the $50 million television campaign was statistically significant and generated an incremental $1 million in sales. With this information, it is doubtful that the marketing team will re-run the same campaign. 

Likewise, if the marketing sciences team reports that the $50 million campaign generated $500 million in incremental sales, then the marketing team is likely to continue the campaign as is.

What this simple example shows is that better business decisions are made when business relevance is coupled with statistical significance than when business decisions are made only relying on statistical significance. This recommendation is for all marketing sciences analyses, including promotion mix models, response models, test / control analyses, what variables to include in the models, etc.  

Join the Discussion