In 1996, I was first introduced to the concept of incremental response modeling. The concept is simple. In direct mail, typically, there is gross response based on what’s commonly referred to as “match-back” attribution. That is, I send out my direct mail and see who has responded or purchased during a set window of time, say, 30 days. Those that have are then labeled “responders.” Easy enough. But the question then becomes…well…who would’ve responded anyway?
To address this, direct marketers created the straightforward concept of holdouts, or blackouts. Simply take a random population and do not mail them. Then do the same attribution and literally take the difference – the gross response minus the holdout response is the incremental. Simple right?
Well, now that we can measure incrementality, the next question is – can we improve upon it with predictive modeling? Well, this is where the challenge really starts. In my experience, in seeing this type of application time and time again, I’ve come to the conclusion that building an incremental model solution that consistently performs isn’t simple at all. In fact, it’s just flat out difficult. It’s not that the model itself is so hard to build, it’s more about the one consistent thing I’ve experienced: results vary.
Of course, I know there are analytic providers and software solutions out there that will contradict this and say it’s easy. But take it from someone who lives and breathes it: incremental modeling is one of the hardest predictive applications to get right over a sustained period of time.
The reality, though, is that this shouldn’t really surprise us. If it were easy, everyone would be doing it successfully, right? In this case, the difficulty is in the nature of how we define incremental measurement. By definition, incremental response is measured on a group of individuals as opposed to one person. Meaning, the incremental response rate is the difference of two rates. Therefore, for any one individual, you cannot really determine if that person was incremental. But you can determine it on a segment or total campaign level. This is the purest challenge of incremental modeling. And although we’ve designed algorithms that directly maximize this difference, we still face the challenge of balancing opportunity cost (holdouts) against campaign performance (treatments).
But if you find yourself working on this type of initiative, please don’t be discouraged. My point of view is not meant to sound defeated. Rather, I’d think this challenge something that should motivate us. It’s a solvable problem, but it will absolutely take discipline in holdout sampling and vigilance in testing and learning with every single cycle. Do that and you’ll get better over time. Just be prepared for the bump in the road, as incrementality is an elusive little thing.