While the basics of using Google's Paid and Organic report in AdWords have been covered well by the industry, there is a lesser known issue that prevents the Paid and Organic report from being as powerful as it could be: Google does not filter organic data by the regions being targeted with ads. In terms of how this impacts the data, we've found that it rears its head in a couple of different, very important, ways. But before we dig into these issues, let's do a quick overview of the Paid and Organic report and what it entails.
What is the Paid and Organic Report?
Google released the Paid and Organic report last August and, as the name suggests, it provides traffic metrics at the query level for both paid and organic search. The key feature is that the report provides the data broken down in three ways:
- When only an ad was shown.
- When both an ad and organic link were shown.
- When just an organic link was present.
To get the full data set advertisers must link their Google Webmaster Tools and AdWords accounts. Basic uses of the P&O report include recouping some of the query data lost because of Not Provided and identifying queries which are only serving organic links and that advertisers may want to consider bidding on in AdWords. While there is a lot of potential for more powerful uses, many of the most interesting possibilities are trampled by the unfiltered state of the organic data.
Measuring the Incrementality of Paid Ads
One such possible use of the Paid and Organic report is that of calculating the impact paid search ads have on the propensity for searchers to click through to an advertiser's site. To do this, a natural first thought would be to compare the click-through rate (CTR) by query of organic links when ads are not present to the cumulative CTR of the ad and organic link when both are present. This should represent the increase in the chance a query ends with the searcher getting to the advertiser's website due to the presence of the ad. However, the CTR of organic only rows in the Paid and Organic report does not appear to be very representative of the CTR one can expect for their organic link were they to suspend paid advertising altogether.
For example, one advertiser found that the CTR for one of their brand terms when no ad was shown was 53%, as measured by the organic only rows in the Paid and Organic report when ads were running. However, during a holdout test in which paid ads were paused, the CTR of the organic link by itself was 75%! This reduced the ad's estimated increase in the chance a searcher clicked through to the site from over 50%, comparing organic only and both rows, to less than 10%, based on the CTR of the organic link during the holdout test. That's a huge difference. In this case, this was a result of the organic listing having a much lower CTR in the international markets that were not being targeted by ads. So the CTR of organic only rows when the ad was running appeared artificially low. When the ad was paused for the holdout test, the organic link picked up a lot more clicks from the U.S. market where CTR was higher.
This appears to be a common scenario, so comparing organic-only CTR to the CTR of both ads and organic to estimate the incrementality of the ad is not wise. A holdout test seems to be the only way to accurately measure the true impact ads have. Due to the loss of query data brought about by Not Provided, the Paid & Organic report is still valuable to analyzing the impact of an ad holdout test at the query level, but this analysis is much more complicated than I think most marketers imagine because of the unfiltered nature of the organic data.
Analyzing a Holdout Test Using the Paid and Organic Report
To explain the details of how to analyze the results of a holdout test, let's walk through sample data from an RKG advertiser's actual test. Ad impression share for the queries tested is near 100%.
Now, it may seem in this example like you should compare the cumulative CTR when ads were running, 79.6%, to the CTR of the organic link during the holdout test, 74.7%. However, as noted above this fails to recognize that with close to 100% impression share, nearly all traffic attributed to 'organic only' prior to the test comes from outside the areas targeted with paid search ads. Thus, the CTR prior to the test should actually be measured using only the 'Both Shown' and 'Ad Shown Only' rows, which comes to 83.6% compared to 79.6% when including the 'Organic Shown Only' rows. This effectively excludes the data from markets where we are not targeting ads.
But that's not all. When looking at the data for the period of the holdout test, we need to adjust the organic data to account for the approximate amount of traffic which occurred outside of the areas targeted by ads. Prior to the test, 13.1% of organic queries were coming from outside of the areas targeted by ads, so we reduce the number of organic queries observed during the test from 28,177 to 24,495. The CTR of those organic queries from outside of the targeted regions was 53.3%, so we also reduce the number of organic clicks that occurred during the test from 21,041 to 19,078. Taking our updated click and query totals for during the holdout test, we arrive at an estimated CTR of 77.9% for the areas targeted by ads, compared to 74.7% prior to accounting for those clicks occurring outside the targeted areas.
After all of this, we finally arrive at the two CTRs that matter to us: 83.6% for ads and organic prior to the test, and 77.9% during the test. Using these figures, the ads for these queries increase the chance that a searcher will end up clicking through to the advertiser's site by 7.3%. Unfortunately, this analysis only truly works for queries with ad impression share at or near 100%. This is because queries with less impression share have more organic impressions and clicks from areas that are targeted by ads in the organic only rows when ads are running, impacting the share of organic impressions and clicks that this bucket accounts for and the measured CTR of the bucket. This then muddies the adjustments we make in analyzing the data after the test.
Percentage of an Advertiser's SERP Impressions that Occur Without an Ad
Another potential use for the Paid and Organic report is to measure what percentage of the time your ad is not being triggered for key queries. However, comparing the percentage of impressions that occur when just an organic link is present, just a paid ad link, or a combination of the two, the average advertiser sees about 40% of their SERP impressions without one of their ads present according to the Paid and Organic report. This is significantly higher than many industry estimates of the percentage of queries that don't return an ad, such as that of Moz, which currently pegs the percentage of queries not returning an ad at around 20%. This is also a result of Google including organic impressions and clicks from areas that advertisers are not targeting in their AdWords campaigns. So while it may be a good indicator that there are additional countries you may want to consider targeting with your ads, we're taking the amount of traffic for queries which our ads don't serve for with a grain of salt, and utilizing Google's impression share data to identify if the ad for a query is really failing to trigger for as many auctions as we'd like in the targeted areas.
There's a lot of cool data in the Paid & Organic report that could give advertisers interesting insights into the synergy between their paid search and organic presence. To truly enable advertisers to take full advantage of this data, however, Google must provide a filter to restrict organic impressions and clicks just to those regions being targeted in AdWords for real apples-to-apples comparisons. Maybe this is a more difficult fix than we imagine, but without it, any comparisons of situations where ads are showing to those where they aren't are made fruitless, and even assessing holdout tests is more complicated than it should be. Further, advertisers who fail to recognize this issue may be incorrectly reading into the report, as well as incorrectly analyzing results from holdout tests.