Programmatic Google Trends API

By Sal Uryasev
June 11, 2008
Find more about: google trends api programmatic

Updated October 21, 2009

Yesterday, Google released an update to their popular Google Trends tool. There are improvements over the previous version, but the biggest new feature is a new shiny button that lets you download all your data in the format of a CSV. This is a very cool enhancement. Where Google Trends was a geeky toy, it now takes the leap to integrate into analysts' reports and with that, edge its way onto managerial desks.

This python module is a quasi-API to make it easier to authenticate into Google Trends for those who want to squeeze the extra level of functionality out of their data. The advantage of programmatic access is that the data can be automatically trended and merged. It can be snuck into a 9:00 AM daily email to the VP of Marketing so that she knows to ramp up Google Adwords campaigns for some specific keyword. Also, by programatically pulling multiple reports, it is possible to create a wealth of data not visible in a single report. Using one keyword as a benchmark to merge multiple reports, we can do a meaningful comparison on tens or hundreds of relevant keywords.

To use the pyGTrends, the quasi-Google-Trends-API, you can download the latest version from github.

Here is an example of the most basic basic report that you can pull down from Google Trends. The connector function needs authentication info, and download_report needs to be passed a list of keywords.

from pyGTrends import pyGTrends

connector = pyGTrends('google username','google password')
connector.download_report(('keyword1', 'keyword2'))
print connector.csv()

You can, however, use pyGTrends to get any slice of data that you can pull down from Google Trends. To see the exact parameters that you should use, go to Google Trends, and navigate to the specific sufficiently-narrow report that you are interested in. Then, right-click on the CSV download, and save the link location. The different parameters should be discernible from the link. The following code downloads a report for banana, bread, and bakery keywords from April 2008, originating from the magnificent nation of Austria, and scaled using fixed scaling (aka the second download link). 

connector.download_report(('banana', 'bread', 'bakery'), 
                          date='2008-4', 
                          geo='AT', 
                          scale=1)

By default, the csv() function downloads the main part of the report, but there are a few additional parts stuck to the bottom of the CSV file. If you are interested in those, pass the section parameter to the csv() function. The following will return the Language section.

print connector.csv(section='Language')

Full recommended usage includes using either the csv.reader or csv.DictReader module.

from csv import DictReader
print DictReader(connector.csv().split('\n'))

Here is a snapshot from the new Google Trends to add some eye-candy to the post: Google Trends Eye-Candy

This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License. All source code is released under a BSD License unless otherwise specified.

28 comments | Show all comments only the last 5 are shown


June 26, 2009
pstobbs said:

Hi Sal
Great tool- very useful indeed, many thanks.

I'm new to Python but pulled together a little code to cycle through different keyword inputs. It seemed to be working fine but then I ran into the issue mentioned by redneckjedi unfortunaltely. I had included a sleep(10) but clearly that wasn't enough and I am now getting Http Error 503: Service Unavailable. When I access the site manually I am greeted with a captcha so I can still run the queries manually just not through the script- has anyone else come across this? I hope the block will stop soon and I will then add a much longer sleep. I only have about 20 or so keyword sets which should be ok I think?

Anyone any ideas how long the ban lasts?
cheers
Piers

Below is my rudimentary loop which I'm sure could be improved upon

#----- import pygtrends
import sys
sys.path.append('c:\\python_code')
from pyGTrends import pyGTrends
connector = pyGTrends('username','password')

#----- initiate output file
import time
final_output = str()

#-----load csv inputs file
import csv
gtrends_input = csv.reader(open("google_trends_inputs.csv","rb"))
gtrends_input_list = []
gtrends_input_list.extend(gtrends_input)

#---- define loop
loop_range = range(1,len(gtrends_input_list))

for x in loop_range:

#---- parameters
tmp_row = gtrends_input_list[x]
tmp_terms = tmp_row[:5]
tmp_geo = tmp_row[5]

#---- save parameters to output
final_output = final_output + tmp_geo + '\n'

#---- get data
connector.download_report((tmp_terms),geo=tmp_geo,scale=1)

#---- append output
tmp_output = connector.csv() + '\n'
final_output = final_output + tmp_output
time.sleep(100)
print 'done',x, tmp_terms, tmp_geo, tmp_symbol

#---- end loop

#---- output

outfile = open('output.csv','w')
outfile.write(final_output)
outfile.close()


September 16, 2009
Dheeru said:

My IP was blocked for a day after submitting less than 100 queries.


October 16, 2009
Bas said:

resp = self.opener.open(self.url_ServiceLoginBoxAuth,params).read();

print " resp ::"+resp;

Im getting the response from google that says "Your browser's cookie functionality is turned off. Please turn it on". ie, not able to login to google account for trends.

Please help me out.


October 21, 2009
Sal Uryasev said:

Alright.
It turns out that Google changed their Login algorithm. Where in the past I could construct a unique 11-digit ID out of thin air, the new algorithm matched the 11-digit ID against one written into a cookie.

The reason why jcp20 were failing some of the time is probably due to Google rolling out their new login algorithm, and some of the calls would hit an older server, while others would fail on a new server. The data pulls should work consistently at this point.

I have also simplified the convenience functions, with the expectation that the user uses the csv.reader or the csv.DictReader features of the standard library.

Recommended example:

from csv import DictReader
r = pyGTrends(username, password)
r.download_report(('pants', 'skirt'))
d = DictReader(r.csv().split('\n'))


October 25, 2009
Josh said:

This is a great idea--I've been waiting for google to roll out their long promised API...

Has anybody but Dheeru experienced the limitations of this API. I would like to set up a datasource to integrate cleanly into a cakePHP app, but it doesn't seem worth the while if google cuts off access to an account/ip(?) after a small number of queries.

Anybody have any specifics on this?

Your name

Email (optional, will not be shared)

Type the word "juice" (required to confuse the spammers)

Your comment


Add a comment






Joining Juice | Contact Us | Privacy Policy
© 2005—2009 Juice Inc.