Keyword metrics, such as search volume and keyword difficulty, drive major decisions about what keywords to target for SEO. It would be a waste of time and effort to target keywords with a small search volume and high keyword difficulty. Spotting those rare golden opportunities of relevant keywords to target that have both a high search volume and low keyword difficulty can make a huge difference to the traffic to your site. Sourcing keyword metrics at no charge can be challenging if not impossible. Within this blog post I will outline how it is possible to source keyword search volume in bulk for free as long as you have a Google Ads manager account and a bit of python knowledge. So lets begin. The AdWords API At the time of writing, the Google Ads API was still in beta and the AdWords API was the recommended alternative in official documentation. The AdWords API is a SOAP API and, as it is a Google API, it uses the OAuth2.0 protocol for authentication and authorisation. Both of these features add extra complexity to API calls and OAuth2.0 authentication means we need to first obtain a few authentication credentials before we begin. Obtaining Our Credentials Developer Token The first token we need to make calls to the API is a developer token. This can be attained through the Google Ads interface itself. Simply navigate to your manager account, then select TOOLS & SETTINGS > SETUP > API CENTRE. Fill out the form and apply for your developer token. Approval of this token may take a few days however you can create a test manager account that allows you to make API calls straight away until then. Bare in mind, for the service we will be using, API calls to the test account will return dummy data. I was not under any particular time pressure, and wanted to explore other areas of the API, so decided to wait for approval for use with our production account. For more information on setting up test accounts see here. OAuth2.0 Credentials Now for our OAuth2.0 credentials. More specifically we need to now generate our client ID and client secret. To do this we need to access the Google Developers Console and navigate to the credentials page. Here we need to: Create a new project. Select Create Credentials then OAuth client ID, on the credentials page. Select Desktop App for the Application type. Click Create. On the page that appears you need to copy the client ID and client secret and store them safely for when we configure our client library. When we eventually send an API request our script will use these credentials to request an access token from the Google Authorisation Server. Once we have this access token our client, which we will set up, sends it to the AdWords API , along with our query, to which we will hopefully receive a response with the data we desire. These access tokens only have limited lifetimes so if we want to be able to reuse the script in the future we need something called a refresh token. This refresh token allows us to renew our access token and therefore allows us to send further requests to the AdWords API. Refresh Token The easiest way to generate our refresh token is to run generate_refresh_token.py (python 3+) from the command line with your client ID and client secret as command line arguments like so: python generate_refresh_token.py --client_id INSERT_CLIENT_ID --client_secret INSERT_CLIENT_SECRET This will prompt you to visit a URL where you setup your OAuth2.0 credentials to access the AdWords API on your behalf. Click Allow which will generate an authorisation code. Copy this code into the command line where you are running the python script and press enter. This generates your refresh token. Client Customer ID The final credential we need is the client customer ID. This can be found next to the account name in most places on the Google Ads interface, for example the accounts drop down menu, in the format xxx-xxx-xxxx. Select the number corresponding to your manager account. Finding Keyword Search Volume Now that we finally have all our necessary credentials lets start coding. Firstly lets install all the necessary packages we will be using using pip and the command line. pip install googleads pandas Pandas will help to handle any data we need using DataFrames and the Google Ads client library makes handling SOAP requests/responses easy with no need for us to work with any XML at all. First of all lets start by importing the libraries we will be using: import _locale import googleads import pandas as pd import traceback from googleads import adwords from googleads import oauth2 When I first created the script I ran into some encoding issues so lets handle that first : _locale._getdefaultlocale = (lambda *args: ['en_UK', 'UTF-8']) Now lets create our search volume puller class and initialise it with our credentials: class searchVolumePuller(): def __init__(self,client_ID,client_secret,refresh_token,developer_token,client_customer_id): self.client_ID = client_ID self.client_secret = client_secret self.refresh_token = refresh_token self.developer_token = developer_token self.client_customer_id = client_customer_id Now lets create a function that will initialise our AdWords client. To do this we need to first request our access token using our client ID, client secret and refresh token. Then we create our AdWords client using our developer token, our newly granted access token and our client customer ID. I ran into some caching issues when I was first exploring the API hence the cache argument when creating our AdWords client. class searchVolumePuller(): def __init__(self,client_ID,client_secret,refresh_token,developer_token,client_customer_id): self.client_ID = client_ID self.client_secret = client_secret self.refresh_token = refresh_token self.developer_token = developer_token self.client_customer_id = client_customer_id def get_client(self): access_token = oauth2.GoogleRefreshTokenClient(self.client_ID, self.client_secret, self.refresh_token) adwords_client = adwords.AdWordsClient(self.developer_token, access_token, client_customer_id = self.client_customer_id, cache=googleads.common.ZeepServiceProxy.NO_CACHE) return adwords_client Next we need to create a function to create our service client for our desired service. class searchVolumePuller(): def __init__(self,client_ID,client_secret,refresh_token,developer_token,client_customer_id): self.client_ID = client_ID self.client_secret = client_secret self.refresh_token = refresh_token self.developer_token = developer_token self.client_customer_id = client_customer_id def get_client(self): access_token = oauth2.GoogleRefreshTokenClient(self.client_ID, self.client_secret, self.refresh_token) adwords_client = adwords.AdWordsClient(self.developer_token, access_token, client_customer_id = self.client_customer_id, cache=googleads.common.ZeepServiceProxy.NO_CACHE) return adwords_client def get_service(self,service,client): return client.GetService(service) Now to create our function that will actually pull search volume for keywords. We can bulk request the search volume of hundreds of keywords in a single API request at a time, so we will split up our keyword list into sub-lists of 700 keywords using list comprehension. This reduces the number of API calls needed so massively speeds up the script and vastly reduces the chance of returning a ‘RateLimitExceeded’ error in comparison to hitting the API for one keyword at a time. We also need to configure a selector to request the search volume data for each keyword. Adding a few comments to this function so its easier to follow our class now becomes : class searchVolumePuller(): def __init__(self,client_ID,client_secret,refresh_token,developer_token,client_customer_id): self.client_ID = client_ID self.client_secret = client_secret self.refresh_token = refresh_token self.developer_token = developer_token self.client_customer_id = client_customer_id def get_client(self): access_token = oauth2.GoogleRefreshTokenClient(self.client_ID, self.client_secret, self.refresh_token) adwords_client = adwords.AdWordsClient(self.developer_token, access_token, client_customer_id = self.client_customer_id, cache=googleads.common.ZeepServiceProxy.NO_CACHE) return adwords_client def get_service(self,service,client): return client.GetService(service)

Blog

Find Keyword Search Volumes for Free using Python and the AdWords API

Home

Blog

Simran Gill

09 April 2020

Sourcing keyword metrics at no charge can be challenging if not impossible. Within this blog post I will outline how it is possible to source keyword search volume in bulk for free as long as you have a Google Ads manager account and a bit of python knowledge. So lets begin.

The AdWords API

At the time of writing, the Google Ads API was still in beta and the AdWords API was the recommended alternative in official documentation. The AdWords API is a SOAP API and, as it is a Google API, it uses the OAuth2.0 protocol for authentication and authorisation. Both of these features add extra complexity to API calls and OAuth2.0 authentication means we need to first obtain a few authentication credentials before we begin.

Obtaining Our Credentials

Developer Token

The first token we need to make calls to the API is a developer token. This can be attained through the Google Ads interface itself. Simply navigate to your manager account, then select TOOLS & SETTINGS > SETUP > API CENTRE. Fill out the form and apply for your developer token.

Approval of this token may take a few days however you can create a test manager account that allows you to make API calls straight away until then. Bare in mind, for the service we will be using, API calls to the test account will return dummy data. I was not under any particular time pressure, and wanted to explore other areas of the API, so decided to wait for approval for use with our production account. For more information on setting up test accounts see here.

OAuth2.0 Credentials

Now for our OAuth2.0 credentials. More specifically we need to now generate our client ID and client secret. To do this we need to access the Google Developers Console and navigate to the credentials page.

Here we need to:

Create a new project.
Select Create Credentials then OAuth client ID, on the credentials page.
Select Desktop App for the Application type.
Click Create.

On the page that appears you need to copy the client ID and client secret and store them safely for when we configure our client library.

When we eventually send an API request our script will use these credentials to request an access token from the Google Authorisation Server. Once we have this access token our client, which we will set up, sends it to the AdWords API , along with our query, to which we will hopefully receive a response with the data we desire.

These access tokens only have limited lifetimes so if we want to be able to reuse the script in the future we need something called a refresh token. This refresh token allows us to renew our access token and therefore allows us to send further requests to the AdWords API.

Refresh Token

The easiest way to generate our refresh token is to run generate_refresh_token.py (python 3+) from the command line with your client ID and client secret as command line arguments like so:

python generate_refresh_token.py --client_id INSERT_CLIENT_ID --client_secret INSERT_CLIENT_SECRET

This will prompt you to visit a URL where you setup your OAuth2.0 credentials to access the AdWords API on your behalf. Click Allow which will generate an authorisation code. Copy this code into the command line where you are running the python script and press enter. This generates your refresh token.

Client Customer ID

The final credential we need is the client customer ID. This can be found next to the account name in most places on the Google Ads interface, for example the accounts drop down menu, in the format xxx-xxx-xxxx. Select the number corresponding to your manager account.

Finding Keyword Search Volume

Now that we finally have all our necessary credentials lets start coding.

Firstly lets install all the necessary packages we will be using using pip and the command line.

pip install googleads pandas

Pandas will help to handle any data we need using DataFrames and the Google Ads client library makes handling SOAP requests/responses easy with no need for us to work with any XML at all.

First of all lets start by importing the libraries we will be using:

import _locale
import googleads
import pandas as pd
import traceback
from googleads import adwords
from googleads import oauth2

When I first created the script I ran into some encoding issues so lets handle that first :

_locale._getdefaultlocale = (lambda *args: ['en_UK', 'UTF-8'])

Now lets create our search volume puller class and initialise it with our credentials:

class searchVolumePuller():
      def __init__(self,client_ID,client_secret,refresh_token,developer_token,client_customer_id):
            self.client_ID = client_ID
            self.client_secret = client_secret
            self.refresh_token = refresh_token
            self.developer_token = developer_token
            self.client_customer_id = client_customer_id

Now lets create a function that will initialise our AdWords client. To do this we need to first request our access token using our client ID, client secret and refresh token. Then we create our AdWords client using our developer token, our newly granted access token and our client customer ID. I ran into some caching issues when I was first exploring the API hence the cache argument when creating our AdWords client.

class searchVolumePuller():
      def __init__(self,client_ID,client_secret,refresh_token,developer_token,client_customer_id):
            self.client_ID = client_ID
            self.client_secret = client_secret
            self.refresh_token = refresh_token
            self.developer_token = developer_token
            self.client_customer_id = client_customer_id
            
      def get_client(self):
          access_token = oauth2.GoogleRefreshTokenClient(self.client_ID,
                                                         self.client_secret,
                                                         self.refresh_token)
          adwords_client = adwords.AdWordsClient(self.developer_token,
                                                 access_token,
                                                 client_customer_id = self.client_customer_id,
                                                 cache=googleads.common.ZeepServiceProxy.NO_CACHE)
 
          return adwords_client

Next we need to create a function to create our service client for our desired service.

class searchVolumePuller():
      def __init__(self,client_ID,client_secret,refresh_token,developer_token,client_customer_id):
            self.client_ID = client_ID
            self.client_secret = client_secret
            self.refresh_token = refresh_token
            self.developer_token = developer_token
            self.client_customer_id = client_customer_id
            
      def get_client(self):
          access_token = oauth2.GoogleRefreshTokenClient(self.client_ID,
                                                         self.client_secret,
                                                         self.refresh_token)
          adwords_client = adwords.AdWordsClient(self.developer_token,
                                                 access_token,
                                                 client_customer_id = self.client_customer_id,
                                                 cache=googleads.common.ZeepServiceProxy.NO_CACHE)
 
          return adwords_client
      
      def get_service(self,service,client):
 
          return client.GetService(service)

Now to create our function that will actually pull search volume for keywords. We can bulk request the search volume of hundreds of keywords in a single API request at a time, so we will split up our keyword list into sub-lists of 700 keywords using list comprehension. This reduces the number of API calls needed so massively speeds up the script and vastly reduces the chance of returning a ‘RateLimitExceeded’ error in comparison to hitting the API for one keyword at a time.

We also need to configure a selector to request the search volume data for each keyword. Adding a few comments to this function so its easier to follow our class now becomes :

class searchVolumePuller():
      def __init__(self,client_ID,client_secret,refresh_token,developer_token,client_customer_id):
            self.client_ID = client_ID
            self.client_secret = client_secret
            self.refresh_token = refresh_token
            self.developer_token = developer_token
            self.client_customer_id = client_customer_id
            
      def get_client(self):
          access_token = oauth2.GoogleRefreshTokenClient(self.client_ID,
                                                         self.client_secret,
                                                         self.refresh_token)
          adwords_client = adwords.AdWordsClient(self.developer_token,
                                                 access_token,
                                                 client_customer_id = self.client_customer_id,
                                                 cache=googleads.common.ZeepServiceProxy.NO_CACHE)
 
          return adwords_client
      
      def get_service(self,service,client):
 
          return client.GetService(service)

      def get_search_volume(self,service_client,keyword_list):
            #empty dataframe to append data into and keywords and search volume lists#
            keywords = []
            search_volume = []
            keywords_and_search_volume = pd.DataFrame()
            #need to split data into lists of 700#
            sublists = [keyword_list[x:x+700] for x in range(0,len(keyword_list),700)]
            for sublist in sublists:
                  # Construct selector and get keyword stats.
                  selector = {
                  'ideaType': 'KEYWORD',
                  'requestType' : 'STATS'
                    }
                    
                  #select attributes we want to retrieve#
                  selector['requestedAttributeTypes'] = [
                    'KEYWORD_TEXT',
                    'SEARCH_VOLUME'
                    ]
                    
                  #configure selectors paging limit to limit number of results#
                  offset = 0
                  selector['paging'] = {
                  'startIndex' : str(offset),
                  'numberResults' : str(len(sublist))
                        }
                    
                  #specify selectors keywords to suggest for#
                  selector['searchParameters'] = [{
                  'xsi_type' : 'RelatedToQuerySearchParameter',
                  'queries' : sublist
                        }]
                  
                  #pull the data#
                  page = service_client.get(selector)
                  #access json elements to return the suggestions#
                  for i in range(0,len(page['entries'])):
                        keywords.append(page['entries'][i]['data'][0]['value']['value'])
                        search_volume.append(page['entries'][i]['data'][1]['value']['value'])
                        
            keywords_and_search_volume['Keywords'] = keywords
            keywords_and_search_volume['Search Volume'] = search_volume
            
            return keywords_and_search_volume

Now to utilise the above class we need a keyword list to find search volumes for as well as our credentials.

We have set up the API calls to be able to handle thousands of keywords in a short space of time however for the purposes of testing lets keep it simple:

if __name__ == '__main__':
     CLIENT_ID = YOUR_CLIENT_ID
     CLIENT_SECRET = YOUR_CLIENT_SECRET
     REFRESH_TOKEN = YOUR_REFRESH_TOKEN
     DEVELOPER_TOKEN = YOUR_DEVELOPER_TOKEN
     CLIENT_CUSTOMER_ID = YOUR_CLIENT_CUSTOMER_ID
 
     keyword_list = ['SEO','Leeds','Google']

Now lets initialise our volume puller class with our credentials:

if __name__ == '__main__':
     CLIENT_ID = YOUR_CLIENT_ID
     CLIENT_SECRET = YOUR_CLIENT_SECRET
     REFRESH_TOKEN = YOUR_REFRESH_TOKEN
     DEVELOPER_TOKEN = YOUR_DEVELOPER_TOKEN
     CLIENT_CUSTOMER_ID = YOUR_CLIENT_CUSTOMER_ID

     keyword_list = ['SEO','Leeds','Google']

     volume_puller = searchVolumePuller(CLIENT_ID,
                                        CLIENT_SECRET,
                                        REFRESH_TOKEN,
                                        DEVELOPER_TOKEN,
                                        CLIENT_CUSTOMER_ID)

Time to initialise our AdWords client and create our service client. The service we want to use is the ‘TargetingIdeaService’.

if __name__ == '__main__':
     CLIENT_ID = YOUR_CLIENT_ID
     CLIENT_SECRET = YOUR_CLIENT_SECRET
     REFRESH_TOKEN = YOUR_REFRESH_TOKEN
     DEVELOPER_TOKEN = YOUR_DEVELOPER_TOKEN
     CLIENT_CUSTOMER_ID = YOUR_CLIENT_CUSTOMER_ID

     keyword_list = ['SEO','Leeds','Google']

     volume_puller = searchVolumePuller(CLIENT_ID,
                                        CLIENT_SECRET,
                                        REFRESH_TOKEN,
                                        DEVELOPER_TOKEN,
                                        CLIENT_CUSTOMER_ID)

     adwords_client = volume_puller.get_client()
            
            
     targeting_service = volume_puller.get_service('TargetingIdeaService', adwords_client)

And finally lets pull the search volume for our keywords:

if __name__ == '__main__':
     CLIENT_ID = YOUR_CLIENT_ID
     CLIENT_SECRET = YOUR_CLIENT_SECRET
     REFRESH_TOKEN = YOUR_REFRESH_TOKEN
     DEVELOPER_TOKEN = YOUR_DEVELOPER_TOKEN
     CLIENT_CUSTOMER_ID = YOUR_CLIENT_CUSTOMER_ID

     keyword_list = ['SEO','Leeds','Google']

     volume_puller = searchVolumePuller(CLIENT_ID,
                                        CLIENT_SECRET,
                                        REFRESH_TOKEN,
                                        DEVELOPER_TOKEN,
                                        CLIENT_CUSTOMER_ID)

     adwords_client = volume_puller.get_client()
            
            
     targeting_service = volume_puller.get_service('TargetingIdeaService', adwords_client)

     kw_sv_df = volume_puller.get_search_volume(targeting_service,keyword_list)

The kw_sv_df variable will be a dataframe with ‘Keywords’ and ‘Search Volume’ Columns that contain your data like so:

  Keywords   Search Volume
0   seo         1220000
1  leeds         673000
2  google      618000000

Here is our final code that brought us to this point:

import _locale
import googleads
import pandas as pd
import traceback
from googleads import adwords
from googleads import oauth2

_locale._getdefaultlocale = (lambda *args: ['en_UK', 'UTF-8'])

class searchVolumePuller():
      def __init__(self,client_ID,client_secret,refresh_token,developer_token,client_customer_id):
            self.client_ID = client_ID
            self.client_secret = client_secret
            self.refresh_token = refresh_token
            self.developer_token = developer_token
            self.client_customer_id = client_customer_id
            
      def get_client(self):
          access_token = oauth2.GoogleRefreshTokenClient(self.client_ID,
                                                         self.client_secret,
                                                         self.refresh_token)
          adwords_client = adwords.AdWordsClient(self.developer_token,
                                                 access_token,
                                                 client_customer_id = self.client_customer_id,
                                                 cache=googleads.common.ZeepServiceProxy.NO_CACHE)
 
          return adwords_client
      
      def get_service(self,service,client):
 
          return client.GetService(service)

      def get_search_volume(self,service_client,keyword_list):
            #empty dataframe to append data into and keywords and search volume lists#
            keywords = []
            search_volume = []
            keywords_and_search_volume = pd.DataFrame()
            #need to split data into smaller lists of 700#
            sublists = [keyword_list[x:x+700] for x in range(0,len(keyword_list),700)]
            for sublist in sublists:
                  # Construct selector and get keyword stats.
                  selector = {
                  'ideaType': 'KEYWORD',
                  'requestType' : 'STATS'
                    }
                    
                  #select attributes we want to retrieve#
                  selector['requestedAttributeTypes'] = [
                    'KEYWORD_TEXT',
                    'SEARCH_VOLUME'
                    ]
                    
                  #configure selectors paging limit to limit number of results#
                  offset = 0
                  selector['paging'] = {
                  'startIndex' : str(offset),
                  'numberResults' : str(len(sublist))
                        }
                    
                  #specify selectors keywords to suggest for#
                  selector['searchParameters'] = [{
                  'xsi_type' : 'RelatedToQuerySearchParameter',
                  'queries' : sublist
                        }]
                  
                  #pull the data#
                  page = service_client.get(selector)
                  #access json elements to return the suggestions#
                  for i in range(0,len(page['entries'])):
                        keywords.append(page['entries'][i]['data'][0]['value']['value'])
                        search_volume.append(page['entries'][i]['data'][1]['value']['value'])
                        
            keywords_and_search_volume['Keywords'] = keywords
            keywords_and_search_volume['Search Volume'] = search_volume
            
            return keywords_and_search_volume
        
if __name__ == '__main__':
     CLIENT_ID = YOUR_CLIENT_ID
     CLIENT_SECRET = YOUR_CLIENT_SECRET
     REFRESH_TOKEN = YOUR_REFRESH_TOKEN
     DEVELOPER_TOKEN = YOUR_DEVELOPER_TOKEN
     CLIENT_CUSTOMER_ID = YOUR_CLIENT_CUSTOMER_ID

     keyword_list = ['SEO','Leeds','Google']

     volume_puller = searchVolumePuller(CLIENT_ID,
                                        CLIENT_SECRET,
                                        REFRESH_TOKEN,
                                        DEVELOPER_TOKEN,
                                        CLIENT_CUSTOMER_ID)

     adwords_client = volume_puller.get_client()
            
            
     targeting_service = volume_puller.get_service('TargetingIdeaService', adwords_client)

     kw_sv_df = volume_puller.get_search_volume(targeting_service,keyword_list)

Final Thoughts

The scripting above is designed to show just one way in which the AdWords API can be utilised. Links to other resources that show other features of the API can be found in the notes section at the bottom of this blog.

Ideally the above script would be modified to take in something like an excel file of 1000+ keywords for processing. This could even be packaged into a .exe file, where a user enters a file path to the file, using libraries like Pyinstaller or make up part of a webapp using a web framework like Django.

Notes

The only set limit for API requests using the AdWords API is the number of requests per day. Basic access level developer tokens can perform 10,000 operations per day and 1,000 report downloads. This limit can be raised by applying for Standard access. The rate limiting you may experience when running this script doesn’t have a set quota of requests per second. The quota depends on factors out of your control like server load. This means if you plan on using the AdWords API in applications that need to be reliable then you need to account for rate limiting errors that you will likely encounter.
The search volume figures retrieved from the API are only estimates and the database often groups keywords and assigns each to a particular volume. For example the terms ‘washing machine’ and ‘washing mashine’ are both returned with the same search volume when in reality this is far from true.
The data returned may differ from that of keyword planner due to differing settings. For more info see here.
Other types of services are also offered by the API.
More info on the targeting ideas service we used.

To discuss the ways in which our SEO Agency can help develop a strong keyword strategy for your business, get in touch.

Written by

Simran Gill

Contact.

We’re always keen to talk search marketing.

We’d love to chat with you about your next project and goals, or simply share some additional insight into the industry and how we could potentially work together to drive growth.

from XLMedia

We believe that great work comes from a passion for all things digital and an absolute commitment to excellence.