We've all seen case studies and white papers showing impressive results of all manner of activities from changing from vendor A to vendor B, to new software product X, to measuring offline impact of online marketing, to...anything you can think of. I've never had much use for case studies because they mean so little and they're so easily 'cooked.' Until recently we didn't have any on our website or in our sales pitches because I figured that everyone viewed them the way I did. Our sales team has taught me that they have great value in the sales process and are therefore important. People want to see concrete evidence of results, and even if they get the questionable value of case studies as a class, they may need something to get the bosses to sign off. 4 reasons I've never put stock in case studies:
- Large improvements can be made by either doing great work to raise the bar, or by doing mediocre work on a badly broken program. The lower the bar the better the case study, but does that really tell us anything?
- Often times the impressive results may be attributable to factors not mentioned in the study. Advertisers may dramatically change their product or service offerings, their marketing objectives and ROI needs, their methodology for measuring success, or other changes in the marketplace any of which can create a totally different result set having nothing to do with the "before/after" comparison presented.
- The study methodology can be grievously flawed making the results meaningless. Sadly, this nevertheless proves interesting information to folks who don't look at or think critically about study methodologies.
- Interested parties often bury the bad results. Few companies are willing to publish the results of studies that run counter to the story they'd like to tell.
Join the Discussion