"Advanced Statistics" and other Meaningless Drivel

In the recent Forrester Wave on Attribution solution providers, one criterion for excellence was the use of "Advanced Statistical Models." Indeed, that phrase pops up in the context of bid management on almost every paid search agencies' website, including ours. While high-brow statistics is certainly necessary, it isn't sufficient to guarantee results. The problem is: "Advanced" doesn't necessarily mean "Accurate." Those who fail to recognize that distinction, treating stats modeling as a check-box rather than a matter of ongoing research, tend to get left behind where the rubber meets the road in performance. Take the case of attribution modeling. We've been working with our stats consulting team on how best to determine which ads should get what fraction of credit for an order. It's a very difficult problem. Truthfully, it's beyond difficult. Most folks ended their study of math at a happy point where applying math properly generated the "right" answer. Many folks assume that this must always be the case. Some of us who trudged on into the nether-world of math and quantum mechanics found to their dismay that there often isn't agreement on any one "answer" and that mathematical approaches can be valid and still not be particularly useful. Let's take a look at some real world examples of how equally "advanced" statistical models produce dramatically different results. We handed a few million rows of attribution data to some of the sharpest stats PhDs in the land and asked them to build a model for spreading credit judiciously between advertisements. These folks decided that the best approach would be to use a dynamic Bayesian Network to build a predictive model. This highly advanced statistical model produced lousy results. It turns out that Bayesian models don't handle sequential ordering well, so a sale following the pattern of activity below was mishandled badly. Example #1: This first draft of a model credited the sale almost entirely to the the affiliate ad, with less than 10% of the credit given to the competitive natural search touch that preceded it. As marketers, we're pretty sure that's wrong. The "right" answer isn't intuitively obvious, but the notion that that first touch deserves a good bit more credit than that is pretty clear. Using our experience as marketers, our internal stats folks and I helped our stats consultants think through other signals in the data that the modeling needed to sniff out. They abandoned the Bayesian approach and came back with 8 different models which produced much more reasonable, intuitively appealing results, that still remain very different from one another. Example #2: In this case, the user first came to the site twice through a single CSE ad. An hour and a half later, they came back to the site through a competitive (non-brand) paid search ad, and then they went digging for coupons through a few affiliate ads. They ended up going back to the CSE before placing the order. Here are the ways the different models parsed credit between the ads: Do any of those models strike you as "dead-on"? "Way off base?" Let's see how those same models handled a different scenario: Example 3: In this case, Affiliate ads actually initiated the contact in late April. The user then re-engaged through a competitive paid search ad, before going back to their favorite coupon site to grab a discount. How would you parse the credit? Here's what the models did: Of course, the reality is that we can't know what actually motivated each individual. The best we can hope to do is develop modeling techniques that allow us to get close to the truth in aggregate. These are all highly advanced statistical approaches to the problem and they don't reach the same "answers." That they produce different answers doesn't surprise us, but may be startling to folks who don't live in the world of statistics. This is the broader point. Whether we're talking about bidding algorithms, circulation modeling software, or attribution management the fact that the statistics under the hood are "advanced" doesn't make them "right." All advanced models are not equally predictive, and absent guidance from marketing experience and testing between modeling approaches "advanced models" can stink. Indeed, we threw out many "advanced" models along the path to our current bidding algorithm. Marketers absent mathematical know-how are stuck without the ability to manage large scale, difficult problems. However, mathematicians lacking marketing acumen don't fair much better. Finding the best solution requires both the math and the marketing intuition to make it work. Buyer's Conundrum: How do you choose between service providers given that they all claim "Advanced Statistics" and none of them will let you look under the hood? My observation is this: the folks who speak of their solution as a completed puzzle probably never even opened the box. Anyone who isn't still tinkering with their models probably never put much thought into them in the first place. Talk to their clients about results, and hire the folks who produce great results but are still curious.
Join the Discussion