Growth of ‘Not Provided’
In October 2011, Google started to encrypt the query data of signed-in users. At that time Google said that it will only impact small portion of keywords data. According to Google Analytics blog:
“When a signed-in user visits your site from an organic Google search, all web analytics services, including Google Analytics, will continue to recognize the visit as Google “organic” search, but will no longer report the query terms that the user searched on to reach your site. Keep in mind that the change will affect only a minority of your traffic. You will continue to see aggregate query data with no change, including visits from users who aren’t signed in and visits from Google cpc”
At that time who knew that this ‘impact on minority of traffic’ will eventually turn into ‘impact on majority of traffic’
After the initial announcement of data encryption for signed in users, this ‘monster’ of not-provided has continuously expanded since.
Going forward Google announced that searches made from SSL connection, will also be counted as ‘not provided’, even if the user is not signed-in.
To add more in this in 2012, Firefox had started using Google SSL Search by default, in result of that a huge chunk of search queries got withheld.
Following graph from an e-commerce website shows how ‘not-provided’ grows over the years since its inception in Oct, 2011.
Danny Sullivan used the term Dark Google for this. Dark Google refers to visits from Google’s search engine that can no longer be tied to a particular search term.
So, in order to find a ray of light in these ‘dark days’ , SEO community around the world has experimented and tested different ways to discover keywords data hidden under the cover of ‘not provided’
Luckily we have webmaster tools – Hoping Google will not take it back!
In Google webmaster tool we have access to keywords data , the data is for last 3 months, even though this data is very less as compared to what we had in Google Analytics in pre ‘not provided’ era, but still we can make good use of this data.
Sub-Directory level webmaster tool configuration gives you more keywords data!
Usually we configure webmaster tools on root domain level i.e. www.example.com. With this configuration we can get query and impressions data for whole site. Webmaster only shows limited amount of queries, still there is large data that can be explored.
An idea struck into my mind, why not we should configure separate webmaster tool on sub- directory level (www.example.com/abc/),so that we can access more keywords data for that particular directory. Separate webmaster tool configuration gave a large count of queries that generated impressions against only the specific sub-directory.
It’s like if we are getting 2000 queries against whole site, we can get same level of details against subdirectory level i.e. 2000 queries for each of the sub-directory for whom we have configured separate webmaster tool.
Advantage here is that, the query data at sub-directory level covers much more queries then we can find in combined webmaster (root domain level). This provides us opportunity to have detailed ‘keywords’ data against our targeted directories and sub-directories that otherwise we can’t get in GA.
How I managed to extract more keywords data from Google Webmaster!
I have tried this method for one of my client’s website. They want to focus on one of their sports directory that is related to Super Bowl Tickets. For gaining access to more keywords data I had configured a separate webmaster for that particular directory. After couple of days when I saw data in webmaster, I was amazed to see the large number of queries that generate impressions against this particular directory. Clearly this data was much more than what we had in root domain level webmaster configuration.
Following is the comparison for both levels Webmaster data for same date range and same filters. You can see the difference in number of queries and impressions.
Figure 1: Data from root level webmaster configuration
Figure 2: Data from Sub-Directory level webmaster configuration
More Keywords, More Opportunities, More Traffic!
Once i got access to that detailed query data, i planned to use these queries for generating more traffic. There were many hidden opportunities in terms of keyword targeting, that we were missing before sub-directory level configuration. I have worked on such queries on which the client’s site is ranking on second page, as these were the low hanging fruits. That resulted in traffic growth for that particular directory. Following image shows the growth of organic traffic:
Figure 3: Organic Traffic growth for Super Bowl
So here we are, we have seen how we can get more insights from webmaster query data by configuring on sub-directories level. This method can be useful especially if you want to focus on specific areas of your websites, you can configure as many sub-directories as you want. If anyone has tried some other method that proves successful in obtaining keywords, you are welcome to share in comments below.
Pro-Tip: If you have verified webmaster tool for your root domain, then there is no need of separate verification process for sub-directories of that domain, all you have to add subdirectory and press verify.