3 Methods for Setting Cookie Windows

Someone clicks on an ad and visits your site. Some time later they buy from you. At what point should an ad stop receiving credit for 'driving' this order? In a subsequent post, we'll tackle the notion that "credit/no-credit" demarcations may not be sufficiently subtle, but for now, let's talk about ways to establish just such a line. It's helpful to posit that with infinite cookie windows, each ad would be credited with some fraction of buyers who were driven by the ad and some who simply bought later without the ad having any influence over the purchase. Perhaps the consumer in that later group is now on a different shopping mission, perhaps they've been influenced to visit your store again by other media or word of mouth, but particularly for well-known brands assuming that all orders preceded by an ad click were driven by that ad is hard to justify. The other cohort of buyers were influenced by the ad. However, they may not convert same session because they want to shop around, they want to consult with someone (a spouse?) about a purchase before commiting, etc. Indeed, it's reasonable to assume that as the delay between the ad click and the order gets longer the split between the "considered buyers" influenced by the ad and the "random re-visitors" will change, shifting from heavily dominated by the former, to heavily dominated by the later as the delay gets larger. This framework, if accepted, gives us three different mechanisms for determining the right cookie window. Method #1: The Click-to-Order Curve
  1. Take non-brand PPC conversion data for the past two or three months. Brand traffic converts differently from non-brand and is generally not incremental, hence we recommend focusing on the competitive ad data.
  2. Count the number of orders placed within each day subsequent to the ad click that preceded the order. To avoid edge effects and rounding issues, we recommend classifying the click to order delay in seconds, dividing by 86,400 (the number of seconds in one day) and truncating the results rather than rounding, so that any order placed within 24 hours of the ad click is binned as a "0", meaning within the first day. A "1" is then between 1 and 2 days, etc.The data should look something like this (only showing the first 20 rows): Orders by Days Delayed from Click
  3. Make a column showing the running % of total orders, such that the number is always increasing towards 100%.
  4. Graph the day versus the percent complete as below, make sure all days are represented, filling in days with no orders as needed.
Cookie Window 2 The argument goes that if the data is dominated by the "random re-visitors" that will manifest itself in a straight line graph. Essentially, if there is no connection between the click on the ad and the order, then there should be no bias towards orders happening sooner rather than later so the "rise over the run" should be a constant. When we look at data stretched over a long window (90 days or so), we often see a normal-looking curve followed by a long fairly straight line up towards 100%. We interpret this as showing the break between the buying cohorts being primarily composed of ad-driven traffic delayed for normal reasons, and the later flat line showing the traffic being primarily composed of random re-visitors not driven by the ad. Find that point by tracing a line from the right side of the graph and see where the data truly dips below that line. That day is a reasonable day to define as your cookie window. Cookie Window 3 Method #2: Average Order Size
  1. Using the data from Method 1 Step 2 calculate the Average Order Size for each day delayed. If the data is sparse you might want to take a three- or five-day rolling average to smooth out the spikes.
  2. Cookie Window 4
  3. Make a graph of Average Order Size by day
What you will often find is that the average order increases for a period of days then drops back closer to the overall average or same day average. We'd argue that this represents another flag indicating that the delay is no longer a function of more considered purchases for higher ticket sales, but is simply random repeat visits unconnected to the initial ad visit. Cookie Window 5 Method #3: Keyword to Shopping Cart Match Rates This is the hardest to pull off comprehensively, but can be done in an hour if you're willing to use sampling rather than complete data sets. The notion is simply that usually the keywords searched bear some relationship to the items purchased. Not always, and the fact that sometimes a person goes to Land's End looking for a shirt and ends up with pants instead doesn't mean the shirt ad shouldn't get credit for bringing them in the store, but it's reasonable to suspect that the fraction of orders matching a keyword thematically should be reasonably constant. However, the farther delayed the actual purchase is from the cookie, lower that match rate may become. We recommend sniffing this landscape by
  1. Take 20 orders that closed the same day as the click and see what fraction of orders had items that matched the keyword. Exclude overly general keywords like "apparel" for landsend.com since 100% of orders will match that keyword.
  2. Repeat this process with orders around the cookie window(s) suggested by Methods 1 and 2 above
  3. Repeat again with orders as far away from the ad click as you can find as a benchmark on the other end
The more samples you take the better sense you'll have of the landscape and the variability of the data, however if you're comfortable with "squishy" numbers this can help you determine if your cookie window makes sense and can confirm the numbers the other methods point to. Proper attribution is critically important, and we recommend revisiting your cookie windows periodically to make sure you're in the right neighborhood. Too long and you're liable to waste money by over advertising, too short and you may miss opportunities. Hope the above is useful and if people have other ideas let me know!
Join the Discussion