What eBay's Test Results Teach Us

A number of people in the industry have asked me what I think about the eBay research study on the incremental value of paid search. While it's flattering to be asked, I'm not sure I have a ton to add to what others have already written. First, as Chris Zaharias points out: it's really important to read the actual study, not other people's summaries of it. eBay is actually very careful to point out that they are a fairly unrepresentative advertiser in many respects and that their findings may not reflect what others would find. This is exactly right. Two physicists should get the same results using the same methodology, but that's because physical laws seem to be fairly universal; not so with marketing. What eBay found: In a carefully run geo-targeted series of control tests eBay found that advertisements on their brand name drove no incremental revenue. More surprisingly: they found that very little revenue attributed to non-brand advertisements was incremental to their business. Given those findings the ROI of non-brand paid search as measured by incremental revenue generated was terrible, leading them to cut back tremendously on paid search spend. They also found that non-brand paid search did drive a higher percentage of incremental purchases from new or infrequent buyers than did other marketing channels, and suggested that smart marketers should focus efforts on new customer acquisition. Here's the thing: you're not eBay. In so many respects, eBay is an outlier.
  1. Market share. Within the world of "auctions", and indeed in many many verticals of merchandise (think antique post cards) eBay's market-share is orders of magnitude bigger than almost any other advertiser. The notion that potential customers will check out the offering of the dominant player in a space before buying regardless of whether they have an ad running is pretty easy to understand. If you have 50% market share (fraction of total sales) in your vertical, and gigantic market penetration (the fraction of customers in the market who have bought from you), the incremental value of non-brand paid search might be low, too. While eBay found little evidence that category mattered for it, those finding might be skewed by heavy volume within each category of "auction, used, antique" prefixes.
  2. Scale. Beyond vertical specific market share, the sheer scale of eBay's business means they have huge consumer awareness. This in turn means advertising may be less necessary, particularly when measured in a short window of time. How much would Coke's sales slump if they didn't spend a dime on advertising for a couple of months? Sales might suffer over time, however. There is a great deal of fascinating research on this topic revealed in a book by Byron Sharp I'm reading on the recommendation of a friend.
  3. Customer base. I'll venture to guess that eBay's most loyal customers are also a somewhat different breed. These are people who love to shop. These are people who love to find deals. These are people who will shop around simply for the joy of doing so. Many advertisers would do well to dynamically bid less for loyal eBay shoppers because they're going to keep looking for a better deal. eBay is more "inevitable" than others would be simply because of who they attract as customers. I, on the other hand, hate to shop. I will buy from the first store that has what I want at what strikes me as a reasonable price. If you don't have a link on the page I'm not likely to look elsewhere. I'm an outlier on the other end, but that notion of range is important.
  4. Brand name differentiation. Someone typing "eBay" is going to end up at eBay.com. Will someone typing "drugstore" necessarily end up at drugstore.com, or might some of them be looking for any drugstore? Will someone searching for "Sony" always go to Sony.com, or might they be delighted with Best Buy? It is worth testing the incremental value of brand search, but understand that all brand search is not created equal.
  5. Paid search strategy and execution. Larry Kim humorously ripped eBay's ppc efforts to shreds and made a compelling case that better strategy and execution would yield better ROI for them. I've met eBay's paid search folks, and I know they're a really smart group of people. However their reliance on automation for keyword generation, copy writing, landing page choices, etc mean they had to pay more for traffic because of poor quality scores, their landing pages convert traffic less well than human driven choices, and the weight of poorly targeted keywords make the aggregate performance look awful. Smart targeting, well crafted/tested copy, good landing pages would demand much less "I" for a similar "R" in ROI. Buying the dictionary doesn't work well in paid search, just like mailing the phone book leads to disaster in direct mail. As a brand building strategy sending the message "You can get anything at eBay" might be a stroke of genius, but don't then expect the short-term ROI to look good on a branding exercise.
What should advertisers learn from eBay's rigorous test? It pays to test your assumptions. eBay identified a ton of wasteful spending by challenging the attributed sales from the channel to find out what was incremental. Good for them. Doing the same test will you find the same results? There is absolutely no reason to think you will, but testing assumptions is the only way to know for sure. The notion of "inevitable" visits in the course of shopping is a profound and interesting one. My wife likely spends 80% of her clothing budget at 5 or 6 different stores (online and off). When she's in the market for a bathing suit, she will almost certainly visit landsend.com whether they advertise to her or not. She likes to shop more than I do, but her consideration set is relatively small. Should Land's End target ads to her to capture wallet share from others, or would they be wasting money? Re-targeting efforts can look artificially productive because of this behavior exactly. Indeed, many marketers have it backwards by pushing offers to their most loyal customers who will buy from them without the offers. I applaud eBay for applying the scientific method to their marketing efforts and encourage others to do so as well. We can all raise our game by testing assumptions, but we must be very careful interpreting the results of our own tests and should never assume that what one company learns will translate directly to your company. Other good articles on the topic from Harvard Business Review, Quartz, SEL.
Join the Discussion