George Michie from the Rimm Kaufman Group (RKG) recently wrote a two-part post titled the Geographic Impact of PPC. In it, he analyzed the data from a report RKG conducted to determine, as George put it, "if some well-known truths from the catalog industry also apply to the world of Paid Search, namely that geography matters." I was intrigued. Of course geography matters, the only question is how much. Maybe the study can help shed some light on that. Right?
Not so, at least not according to this report. In his description of the findings, George states that "we have to conclude that knowing the zip code of the user doesn’t have much value for retailers in Paid Search bid management." Huh?
Let me start out by saying that George is very honest that there are major limitations to the study (see Part 2), and basically admits that no conclusions should be drawn from it. He does go on to draw conclusions, however. One of which is that ads placed for national retailers do as well or better when served to online users in the same zip code as a store. So... does geography matter or doesn't it? If the second conclusion is true and I were a retailer, I'd place more ads where I have stores, right?
I like the fact that George and RKG are asking these questions. Some of their findings are shocking, and they will evoke some discussion, which is good, but the study is flawed. I find it hard to believe that either RKG or George actually thinks that geography doesn't matter in online advertising. For anyone who's in the online advertising business, experience and intuition tell us otherwise. Almost every online marketer includes some geotargeting options. The largest advertisers who make the most money for their customers and analyze performance more than anyone have the most granular geotargeting, and charge more for it (you can draw custom shapes with Google and Yahoo! added zip targeting last year). Quova has case studies from advertisers like 24/7 RealMedia as well as large and small organizations like Continental Airlines and WyzAnt that show that local ads and offers can increase conversions from 50-200% (WyzAnt and Continental respectively). And that's not just for local businesses, Melissa Mackey at Search Engine Watch just two months ago wrote about how national advertisers can reap the same benefits local advertisers have for years.
So where did the study go wrong? Again, George is honest and admits there is very little correlation between the factors in their data when looking at traffic value. Even in their best model, 85% of the variance was caused by random noise or other factors not available in the data. They did get somewhat better results (though still effectively useless) when looking at rural vs. densely populated areas, but even then 55% of the variance was unexplainable. He compares those results to what Johannes Kepler found when trying to explain planetary orbits. Kepler completely rejected one of his early attempts because the variance was just 2%, and kept working until he was able to describe the true model of elliptical planetary orbits.
I think it's safe to say that the reason there is so little correlation in RKG's data is because one of the original assumptions was wrong. RKG mapped the IP addresses of 3M PPC clicks to individual zip codes based on data from another geolocation provider. To be fair, it's unclear whether that provider knew how the data was going to be used, but if they did, they should have made it clear that mapping IP addresses to user zip codes can be problematic. Lots of traffic is aggregated through ISP proxies or other routing points. Many IP addresses are dynamic, so can be in one location one day and somewhere else the next. Users often browse at work and shop close to home, and don't live and work in the same zip. Even how users connect to the Internet can affect location (e.g. it's much more difficult to locate dial-up and satellite connections). George says that they originally tried to do the study with a cheaper provider and found the data unreliable. So how do you describe data that results in 85% variance? I'm not saying that IP addresses cannot be mapped to zip codes. They can. But if you don't know which IP addresses can be reliably mapped and which can't, then you can't make any real conclusions based on that data. RKG's results show how starting with one bad assumption can lead to wildly unexplainable results.
Also, one of the limitations of the study George lists is important: that the zip codes are "zip codes of the IP, not the user. We really don’t know the location of the user, we know the location of the IP address." That can obviously make a huge difference. Most IP geolocation technology locates IP addresses somewhere upstream of the ultimate end-point. That location might be a DSLAM (for DSL), an ISP, or even a satellite. It's rare that all the downstream IP addresses all share the zip code and almost a certainty that they don't share the same demography. DSL connections can be located fairly accurately because of the physical distance limitation to the DSLAM of about 5km. As an experiment, draw a circle with an 5 km radius around where you live and figure out how many zip codes and demographics that covers.
Quova actually provides our customers with tools to help them determine the likelihood that a user is actually where the IP address is. That likelihood varies on several factors (the "factors not available in the data" George mentioned). At Quova, we provide a value called the Confidence Factor, that gives our customers an indication of how likely it is that the user is located were the IP is. That Confidence Factor is an aggregate of the other information and evidence we collect and of our Network Geography Analysts' knowledge of how the internet works. We also provide most of that evidence as raw data, so you can find correlations, if they exist, with any one piece of evidence.
The fact that RKG didn't have the information to tell them what data to trust and what not to unfortunately makes their findings only interesting talking points, not real conclusions. I might sound critical of the study and I don't mean to be. I don't think George would disagree with anything I've said and don't think even he would claim that the findings are anything other than anecdotal. If nothing else, he raises some very interesting questions about how national retailers should think about online advertising.
If RKG has a real interest in this, I hope they follow Kepler's lead and try again. I look forward to more studies from RKG and others.