Programmatic Google Trends API

By Sal Uryasev
June 11, 2008
Find more about: google trends api programmatic

Updated October 21, 2009

Yesterday, Google released an update to their popular Google Trends tool. There are improvements over the previous version, but the biggest new feature is a new shiny button that lets you download all your data in the format of a CSV. This is a very cool enhancement. Where Google Trends was a geeky toy, it now takes the leap to integrate into analysts' reports and with that, edge its way onto managerial desks.

This python module is a quasi-API to make it easier to authenticate into Google Trends for those who want to squeeze the extra level of functionality out of their data. The advantage of programmatic access is that the data can be automatically trended and merged. It can be snuck into a 9:00 AM daily email to the VP of Marketing so that she knows to ramp up Google Adwords campaigns for some specific keyword. Also, by programatically pulling multiple reports, it is possible to create a wealth of data not visible in a single report. Using one keyword as a benchmark to merge multiple reports, we can do a meaningful comparison on tens or hundreds of relevant keywords.

To use the pyGTrends, the quasi-Google-Trends-API, you can download the latest version from github.

Here is an example of the most basic basic report that you can pull down from Google Trends. The connector function needs authentication info, and download_report needs to be passed a list of keywords.

from pyGTrends import pyGTrends

connector = pyGTrends('google username','google password')
connector.download_report(('keyword1', 'keyword2'))
print connector.csv()

You can, however, use pyGTrends to get any slice of data that you can pull down from Google Trends. To see the exact parameters that you should use, go to Google Trends, and navigate to the specific sufficiently-narrow report that you are interested in. Then, right-click on the CSV download, and save the link location. The different parameters should be discernible from the link. The following code downloads a report for banana, bread, and bakery keywords from April 2008, originating from the magnificent nation of Austria, and scaled using fixed scaling (aka the second download link). 

connector.download_report(('banana', 'bread', 'bakery'), 
                          date='2008-4', 
                          geo='AT', 
                          scale=1)

By default, the csv() function downloads the main part of the report, but there are a few additional parts stuck to the bottom of the CSV file. If you are interested in those, pass the section parameter to the csv() function. The following will return the Language section.

print connector.csv(section='Language')

Full recommended usage includes using either the csv.reader or csv.DictReader module.

from csv import DictReader
print DictReader(connector.csv().split('\n'))

Here is a snapshot from the new Google Trends to add some eye-candy to the post: Google Trends Eye-Candy

This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License. All source code is released under a BSD License unless otherwise specified.

35 comments | Show all comments only the last 5 are shown


February 16, 2010
David said:

I have the same questions as the above.
also, is there a document for using this API.
Thanks


February 23, 2010
Aloysius Adrian said:

I want to ask about section in csv function. I was trying to get a certain csv, but the interpreter stops at line 115 that produces error message : "KeyError: 'main'"

I want to ask about the other parameter. I can not print the csv because of the section parameter there.

Thanks.


February 24, 2010
Sal Uryasev said:

The DictReader module is a Python convenience module for reading data into a dictionary. Its use is optional.

One thing that can be done if there are any issues is to print connector.raw_data, and that would display the direct result from Google Analytics. Google sometimes displays additional username/login related problems that the module may not have accounted for.


March 1, 2010
David said:

From yesterday, This script can't do print connect.csv(). It always returns "Could not find requested section" error. Do you know where the problem is?
I also download the original script in case I modify anything but the error remains.
Thanks


March 2, 2010
Aloysius Adrian said:

@David
I, also, can not get the connect.csv()

I contacted Mr. Uryasev, and he suggested that I try print connect.raw_data

I did that, but the result/output was : "You must be signed in to export data from Google Trends"

I wonder if you also get that kind of output..

Your name

Email (optional, will not be shared)

Type the word "juice" (required to confuse the spammers)

Your comment


Add a comment




How Did We Mash Data into Google Analytics?

By Sal Uryasev
June 6, 2008
Find more about: googleanalytics google

This post is the code behind how we mashed external data into Google Analytics.

The first step is to yank reference data from the Google Analytics site to reference against Kampyle's data. We specifically want to gather individual names of websites (index.html, /index2.html), and the current selected daterange. The cell references to the website names in the table can be found using a neat Javascript Shell popular among Greasemonkey and Javascript developers. I will not go into detail about the Javascript Shell, but by checking out the various child nodes for the table object we can track down that document.getElementById('f_table_data').childNodes[3].rows[1].cells[1].textContent points at the text in the first cell of the first row. While the syntax looks long, it is just nested HTML in a more elegant programmatic fashion.

For the date, Google Analytics uses a slightly peculiar hybrid system where the date is drawn initially from the URL, but if the date is modified with the java date tool in the upper right hand corner, it uses that instead. From our end, document.getElementById('f_primaryBegin').value and document.getElementById('f_primaryEnd').value are the java date tool values that only start existing if the date tool is used. Pull these two values if they exist, and simply parse the date from the URL otherwise.

The clickable tab we created is essentially the equivalent of a little Greasemonkey button with a few frills that can be created in the standard Greasemonkey fashion. Wherever possible, I use Google-defined layouts for consistency with the site.

Next, we want to send out our reference data to some external server. Greasemonkey has good functionality for pulling data from other sites and servers through the use of the GM_xmlhttpRequest command. A server-end PHP or Django service might be easiest to implement. In this specific example, Kampyle wanted to use the SOAP protocol. While there is an excellent overall SOAP client for javascript by Matteo Casati, this client does not work in a plug and play fashion with Greasemonkey, and needed some modification. For any devoted SOAPers who want to try Greasemonkey, the revised javascript-soap-client code can be found in the attached file. We use the SHA256 encryption function written by Angel Marin and Paul Johnston, but that is accomplished by just copying and pasting the function into our code.

The result comes back in the form of an xml object describing each row in the table, which we parse using native Javascript/Greasemonkey methods, and pop back into the table in the way that we extracted the individual website names. A neat trick here is to call each individual row individually, and not to wait for the data to come back before calling the next row from the server. Separate listeners can wait and insert the data at their leisure. This allows our page to load up faster, and in case there is an error with one data element, it could potentially allow the rest of the rows to load in peace.

You can play around with my code here. This code is released under the BSD License. You won't be able to run the code verbatim without Kampyle's compliance, since they have changed the API calls on their server. However, much of it should be very portable to other data sources.

This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License. All source code is released under a BSD License unless otherwise specified.

2 comments


September 13, 2008
Gotham said:

This post have great information but I needed a few clarifications. For one of my clients I have usual web analytics info displayed in GA. Additionally the client has call tracking data in its own database. Can I pull that info into GA in a new tab? Your mashup indicates that you added a new tab called "Kampyle", are the names of the table which shows up configurable? (e.g URL, avg grade)


September 16, 2008
Sal Uryasev said:

Hey Gotham,

Yes - as long as you have easy access to the data, you can push any data that you want into Google Analytics. If the data is completely static, you can even add it to the script. Alternatively, you could have a hosted file somewhere. In our case, the data was very dynamic, so we used a server with another web service to fetch the data.

If you click on the picture above, it'll show you the entire table, including column names that I changed around. Essentially, you have the power to change any text that you can select by a mouse. It is just a matter of knowing where to point your code.

Your name

Email (optional, will not be shared)

Type the word "juice" (required to confuse the spammers)

Your comment


Add a comment




Juiced Google Analytics Python API

By Sal Uryasev
May 2, 2008
Find more about: google analytics python api pygapi

Due to the release of an official Google Analytics Data Export API, this module is now deprecated. We have an alternative python module based upon the real analytics API here, and an exploring tool with an automatic code generation capability here.

It is not official. It is not from Google. It is, however, very functional and very here. I present to you pyGAPI, the Juiced Google Analytics Python API. This module allows you to pull information from your incarnation of Google Analytics and employ it programatically into your reporting code.

Let us use iPython to peek through some code using pyGAPI.

In [3]: from datetime import date
In [4]: import pyGAPI
In [5]: connector = pyGAPI.pyGAPI(username, password, website_id="1234567")

Here we create a pyGAPI object. Behind the scenes, pyGAPI logs into Google Analytics, and downloads an identifier cookie. website_id is optional. If omitted, pyGAPI accesses the first website on the account's list. To get a list of all the site IDs to which your site has access, run the function connector.list_sites().

In [6]: connector.download_report('KeywordsReport', (date(2008,3,10), date(2008,3,31)), limit=5)

Download a report into your pyGAPI object. KeywordsReport is the name of the report. It is followed by a tuple containing the start and end dates in python date format. limit is an optional parameter that specifies the number of entries that pyGAPI should pull down. By default, it will pull in all the entries up to a maximum of 10000. Lowering this number will certainly improve performance. The entries returned are ranked by Visits, so you should get the most significant values of the bunch.

In [7]: print connector.csv()
Keyword,Visits,Pages/Visit,Avg. Time on Site,% New Visits,Bounce Rate,Visits,Subscribe,Solutions,Goal Conversion Rate,Per Visit Goal Value
juice analytics,356,5.935393258426966,314.061797752809,0.38764044642448425,0.29494380950927734,356,1.0,0.16292135417461395,1.1629213094711304,0.0
excel training,142,1.971830985915493,98.0774647887324,0.908450722694397,0.6901408433914185,142,1.0,0.0211267601698637,1.0211267471313477,0.0
excel charts,77,1.7922077922077921,95.0,0.9090909361839294,0.7792207598686218,77,1.0,0.03896103799343109,1.0389610528945923,0.0
excel skills,72,1.6527777777777777,75.29166666666667,0.9444444179534912,0.7083333134651184,72,1.0,0.0,1.0,0.0
colbert bump,70,1.3142857142857143,113.77142857142857,0.6428571343421936,0.8428571224212646,70,1.0,0.0,1.0,0.0

This function displays your report in a nice excel-ready CSV format.

In [8]: print connector.parse_csv_as_dicts(convert_numbers=True)
[{'Avg. Time on Site': 314.06179775280901, 'Per Visit Goal Value': 0.0, 'Bounce Rate': 0.29494380950927734, 'Keyword': 'juice analytics', 'Visits': 356.0, 'Pages/Visit': 5.9353932584269664, 'Subscribe': 1.0, 'Solutions': 0.16292135417461395, '% New Visits': 0.38764044642448425, 'Goal Conversion Rate': 1.1629213094711304}, {'Avg. Time on Site': 98.077464788732399, 'Per Visit Goal Value': 0.0, 'Bounce Rate': 0.69014084339141846, 'Keyword': 'excel training', 'Visits': 142.0, 'Pages/Visit': 1.971830985915493, 'Subscribe': 1.0, 'Solutions': 0.021126760169863701, '% New Visits': 0.90845072269439697, 'Goal Conversion Rate': 1.0211267471313477}, {'Avg. Time on Site': 95.0, 'Per Visit Goal Value': 0.0, 'Bounce Rate': 0.77922075986862183, 'Keyword': 'excel charts', 'Visits': 77.0, 'Pages/Visit': 1.7922077922077921, 'Subscribe': 1.0, 'Solutions': 0.038961037993431091, '% New Visits': 0.90909093618392944, 'Goal Conversion Rate': 1.0389610528945923}, {'Avg. Time on Site': 75.291666666666671, 'Per Visit Goal Value': 0.0, 'Bounce Rate': 0.70833331346511841, 'Keyword': 'excel skills', 'Visits': 72.0, 'Pages/Visit': 1.6527777777777777, 'Subscribe': 1.0, 'Solutions': 0.0, '% New Visits': 0.94444441795349121, 'Goal Conversion Rate': 1.0}, {'Avg. Time on Site': 113.77142857142857, 'Per Visit Goal Value': 0.0, 'Bounce Rate': 0.84285712242126465, 'Keyword': 'colbert bump', 'Visits': 70.0, 'Pages/Visit': 1.3142857142857143, 'Subscribe': 1.0, 'Solutions': 0.0, '% New Visits': 0.6428571343421936, 'Goal Conversion Rate': 1.0}]

This function goes the extra step and converts the CSV into a dictionary for easier programmatic use. By default, all entries will be returned as python strings. Setting convert_numbers to True, as we did here, will additionally parse the dictionary to turn all numbers into float values. 

In [9]: print connector.list_reports()
('ReferringSourcesReport', 'SearchEnginesReport', 'AllSourcesReport', 'KeywordsReport', 'CampaignsReport', 'AdVersionsReport', 'TopContentReport', 'ContentByTitleReport', 'ContentDrilldownReport', 'EntrancesReport', 'ExitsReport', 'GeoMapReport', 'LanguagesReport', 'HostnamesReport', 'SpeedsReport')

This gets a list of all the reports that I have successfully tested thus far. All site-specific reports should work. A couple site-section specific reports should be included in the next update of pyGAPI.

Google is great and will release a real API soon, but until then you can download pyGAPI.

This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License. All source code is released under a BSD License unless otherwise specified.

13 comments | Show all comments only the last 5 are shown


August 7, 2008
Ludovic said:

Very nice work. Very useful to, let's say get your most visited pages without having to maintain parallel accounting. May I ask you to licence it to an OSS licence and put it on Google Code ? Would be great.


August 20, 2008
Sebastian said:

Hello,

it work well! Great.
How can i pull the "keyword" or "country" report for a specific URL?
(use segmention)

Thanks


September 5, 2008
Thierry said:

Awesome work !


April 21, 2009
Random said:

There is now an actual analytics API:
http://code.google.com/apis/analytics/docs/gdata/gdataDeveloperGuide.html


April 22, 2009
Sal said:

I wrote a Python API wrapper that I call 'degapi' for the new analytics API to replace this old code. I have yet to put up a post and link about it, but it can be found here: http://suryasev.github.com/python-degapi/

There is an automatic python code generator for this API at http://vascodegapi.juiceanalytics.com

Your name

Email (optional, will not be shared)

Type the word "juice" (required to confuse the spammers)

Your comment


Add a comment




Keyword Trends in Google Analytics With Greasemonkey

By Sal Uryasev
April 23, 2008
Find more about: webanalytics google analytics hack greasemonkey

There is a new post that re-releases the script as a Firefox Plugin. Find it here.

After the warm reception for the first version of our Enhanced Google Analytics, we decided to add some new functionality. (Nothing like a few kinds words to keep us in the giving mood.) The first script created a couple new tables in the Google Analytics interface that highlight recent changes in referral visits. It uses Greasemonkey, an add-on for Firefox that allows a user to insert javascript directly into a webpage.

Our update gives you even more ability to understand the data in Google Analytics:

  • At the suggestion of Avinash Kaushik, the new script works for keyword data, helping you see how organic search traffic is changing. An increase in a keyword may indicate a general change in user interests and/or improved performance on search results.
  • My coworker Pete Skomoroch also suggested that I add the ability to see declines in referrals and new keyword searches.
  • With the help of Paul Irish, the script is now better able to interface with the date widget on the Google Analytics site.

(Click the above button for a simulation.)

Keyword Growth Keyword Decline

When you click the button, your browser will download some historical data behind the scenes, and display a nice summary of the best and worst performing keywords/referring domains.

Installation Instructions:
Firefox 2.0+
Greasemonkey
googleanalyticsdownloade.user.js

If you don't already have Firefox, install it. Install Greasemonkey, and do the required Firefox restart. You should see a handsome monkey peeking at you from the bottom right hand corner of your browser. Open the script file in your firefox browser, and Greasemonkey should give you an option to install the script.

Afterwards, log into Google Analytics, and navigate to your Referring Sources or Keywords Tab. Click the button.

Configuring the script:

We spent some time trying to find convenient default settings here at Juice Analytics, so the script should work straight out of the box. Some users, however, may find it convenient to alter some of these configurations. To do so, in Firefox, go to Tools=>Greasemonkey=>Manage User Scripts..., select Google Analytics Downloader, and then click Edit in the lower left corner of the window. This should open up the script file in a text editor. If your computer does not have a default text editor configured, you may have to choose one. 'c:\windows\notepad' is a good bet for Windows machines.

This is what you should see:

Code Blurb

The bracket labeled 'keywords?' controls defaults for the Keywords page, and correspondingly, 'referring_sources?' controls the Referring Sources page.

To change the settings, simply change the corresponding variable to your preferred default. Make sure to refresh your Google Analytics webpage, if you have it open, so the new settings are loaded.

Now for the nitty gritty configuration details:

  • display_limit: This controls the maximum entries that each table will contain. This may be useful for large, sprawling sites.
  • growth_tolerance: This is the percentage growth parameter. Changing it to .10, for example, will catch everything that has grown by 10%, as opposed to the default 50% and 20%, respectively.
  • minimum_number_elements: This is a significance benchmark that can be used to limit what is displayed upon the screen. By default, only keywords with at least 10 elements are displayed upon the screen. Referring Sites does not have a minimum by default, but one can be set if desired.
  • limit: Limit is more of an internal parameter that determines how many entries should be downloaded from Google in order to get the results that are visible here on the page. Lower the limit to increase speed. If the limit is set to a very high number, you will get the largest result set, but you will have to sit around for a while for the results to load. Since the results are downloaded ordered by volume, raising the limit from the default numbers will not actually give more significant results. You will simply get more of the smaller results, such as keywords with only 1 hit.
  • look_back: This is a very important parameter. The script uses the date displayed upon your Google Analytics page to determine the full range that you want to consider in your results, but 'look_back' determines how many of those days are used for the significance test. So, say the range you have displayed in Google is March 23 - April 22 and your look_back is 7 days. The script will compare the average referrals for a given keyword from April 16-22 to the average from March 23-April 15, and will return the keyword only if the recent average is 20% higher than the rest of the time period. Thus, if you want to increase the total range of the data, change the dates on the actual webpage. Change 'look_back' only if you want to change the period of significance.

Happy analyzing!

This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License. All source code is released under a BSD License unless otherwise specified.

43 comments | Show all comments only the last 5 are shown


March 26, 2009
Dave H said:

Hi,
Installed fine both using Greasemonkey/FF Add-In, however when I push the button I can see the table briefly appear but almost instantly disappear. Instead, multiple sort asc/desc triangles appear in the main Site Usage table. Does it interfere with other plug-ins?
(Also it only works when you access GA directly (the orange version) as opposed to via Adwords (the green version) - but that's not the main problem!)


March 30, 2009
Sal Uryasev said:

Hey Dave,
My particular plugin is very non-intrusive. While it should not interfere with any other plugins that I know about, if you have something that is quite intrusive, there is always a chance. It does sound as if you have something extra installed that kicks off after the script runs... maybe some kinds of special scripts to neaten up webpages?


June 5, 2009
steve said:

Hey Dave,
Thanks for the plugin, but I can't get it to work.:( I'm on firefox 3.0.1 and I can see it installed and I restarted like it asked. Then I go to google analytics page and no blue button.

Can you help me? I realize this is in Beat and I should expect this.


July 4, 2009
norad73 said:

The button shows 3 tables but they are empty... I tried changing the date period but they are still empty... any ideas?


December 15, 2009
Lee said:

The question is why the heck doesn't GA do this already? Been beating my head trying to find out how to do this with GA but you've provided the only solution.

I'd really love to be able to change the ranges -- so be able to compare against the same time last year, 30 days and then the default 7 days. Any help on how to do this?

Thanks for an awesome and life saving job.

Your name

Email (optional, will not be shared)

Type the word "juice" (required to confuse the spammers)

Your comment


Add a comment




Enhancing Google Analytics Using Greasemonkey

By Sal Uryasev
April 11, 2008
Find more about: webanalytics google analytics hack greasemonkey

There is a new post that re-releases the script as a Firefox Plugin. Find it here.

My boss Zach has a problem. Every four hours the craving strikes him. No matter where he is, he pulls out his shiny Macbook Pro and navigates to Juice's Google Analytics site. He pulls up the list of referrers to our site and meticulously searches for new domains. He has an freakish ability to pick out IP addresses that have never linked to us before. Even so, there had to be a better way.

I wondered whether Greasemonkey might be able to help. Greasemonkey is an extension for Firefox that allows users to install custom javascript when you visit a specific website. These scripts can add a delete button for Gmail, automatically display lyrics to your YouTube music video, or do pretty much anything else you would want to enhance the functionality of a website.

After poking around the subtleties of the Google Analytics interface, I came up with a little script that can identify the new referrals that Zach so desperately craves. When navigating to the "Referring Sites" section of Google Analytics, the script add the following button to the interface.

Google Analytics Button

Pushing the button downloads all the referrer data for the date displayed in the Google Analytics range, as well as a similar set of data for the range up to, but not including, the last three days. The difference between the two data sources is used to calculate all of the results. The specific number of days can be changed by editing the first line of the script. Greasemonkey then displays the results in two tables above the original Referrer table. (Greasemonkey works entirely within your browser shell, so your data should be quite secure.)

Google Analytics Data

The first table shows any sites that have displayed more than a 50% increase in visits over the last 3 days as compared to the rest of the time range. The second shows all new recent sites that do not appear at all more than 3 days ago. This can be quite useful to anyone, who, like Zach, absolutely needs to know about any new and exciting inbound links.

Installation Instructions:
Firefox 2.0+
Greasemonkey
googleanalyticsdownloade.user.js

If you don't already have Firefox, install it. Install Greasemonkey, and do the required Firefox restart. You should see a handsome monkey peeking at you from the bottom right hand corner of your browser. Open the script file in your firefox browser, and Greasemonkey should give you an option to install the script.

Afterwards, log into Google Analytics, and navigate to your Referring Sources Tab. Click the button.

This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License. All source code is released under a BSD License unless otherwise specified.

21 comments | Show all comments only the last 5 are shown


May 1, 2008
Tim said:

Hi Sal,
thank you so much - I just reinstalled it and now it works! This is really fantastic!
Thanks,
Tim


May 7, 2008
Nate Sidmore said:

Awesome tool Sal, (thanks to Avinash for the tip).

I did run into a problem with the Firefox pop-up message "Warning: Unresponsive script". However that problem can be solved by lengthening the time allowed for scripts to run. For more details go to http://lifehacker.com/software/firefox/put-off-firefox-15s-unresponsive-script-dialogue-162574.php

However I was bummed when after setting the time allowance to 10 minutes, and clicking the "Who Sent Me Unusual Traffic" button in GA, the script ran for 9 min 38 sec before returning results. Any tips on getting quicker returns?


May 7, 2008
Chris Gemignani said:

Nate:

Thanks for the encouragement. If you check our "Keyword Trends" Greasemonkey script (linked at the start of this post), we write about how to change the parameters in the script to make things run faster.


June 19, 2009
Edwin said:

How can I only bring up the report for non-paid keywords? Selecting it and then returning the results, still brings up cpc words as well.


February 22, 2010
Shankar said:

This is a nice tips, I 'll use it fro my site http://www.onlinegk.com

Your name

Email (optional, will not be shared)

Type the word "juice" (required to confuse the spammers)

Your comment


Add a comment








Joining Juice | Contact Us | Privacy Policy
© 2005—2009 Juice Inc.