Programmatic Google Trends API
By Sal Uryasev
June 11, 2008
Find more about:
google
trends
api
programmatic
Updated October 21, 2009
Yesterday, Google released an update to their popular Google Trends tool. There are improvements over the previous version, but the biggest new feature is a new shiny button that lets you download all your data in the format of a CSV. This is a very cool enhancement. Where Google Trends was a geeky toy, it now takes the leap to integrate into analysts' reports and with that, edge its way onto managerial desks.
This python module is a quasi-API to make it easier to authenticate into Google Trends for those who want to squeeze the extra level of functionality out of their data. The advantage of programmatic access is that the data can be automatically trended and merged. It can be snuck into a 9:00 AM daily email to the VP of Marketing so that she knows to ramp up Google Adwords campaigns for some specific keyword. Also, by programatically pulling multiple reports, it is possible to create a wealth of data not visible in a single report. Using one keyword as a benchmark to merge multiple reports, we can do a meaningful comparison on tens or hundreds of relevant keywords.
To use the pyGTrends, the quasi-Google-Trends-API, you can download the latest version from github.
Here is an example of the most basic basic report that you can pull down from Google Trends. The connector function needs authentication info, and download_report needs to be passed a list of keywords.
from pyGTrends import pyGTrends
connector = pyGTrends('google username','google password')
connector.download_report(('keyword1', 'keyword2'))
print connector.csv()
You can, however, use pyGTrends to get any slice of data that you can pull down from Google Trends. To see the exact parameters that you should use, go to Google Trends, and navigate to the specific sufficiently-narrow report that you are interested in. Then, right-click on the CSV download, and save the link location. The different parameters should be discernible from the link. The following code downloads a report for banana, bread, and bakery keywords from April 2008, originating from the magnificent nation of Austria, and scaled using fixed scaling (aka the second download link).
connector.download_report(('banana', 'bread', 'bakery'),
date='2008-4',
geo='AT',
scale=1)
By default, the csv() function downloads the main part of the report, but there are a few additional parts stuck to the bottom of the CSV file. If you are interested in those, pass the section parameter to the csv() function. The following will return the Language section.
print connector.csv(section='Language')
Full recommended usage includes using either the csv.reader or csv.DictReader module.
from csv import DictReader
print DictReader(connector.csv().split('\n'))
Here is a snapshot from the new Google Trends to add some eye-candy to the post:

How Did We Mash Data into Google Analytics?
By Sal Uryasev
June 6, 2008
Find more about:
googleanalytics
google
This post is the code behind how we mashed external data into Google Analytics.
The first step is to yank reference data from the Google Analytics site to reference against Kampyle's data. We specifically want to gather individual names of websites (index.html, /index2.html), and the current selected daterange. The cell references to the website names in the table can be found using a neat Javascript Shell popular among Greasemonkey and Javascript developers. I will not go into detail about the Javascript Shell, but by checking out the various child nodes for the table object we can track down that document.getElementById('f_table_data').childNodes[3].rows[1].cells[1].textContent points at the text in the first cell of the first row. While the syntax looks long, it is just nested HTML in a more elegant programmatic fashion.
For the date, Google Analytics uses a slightly peculiar hybrid system where the date is drawn initially from the URL, but if the date is modified with the java date tool in the upper right hand corner, it uses that instead. From our end, document.getElementById('f_primaryBegin').value and document.getElementById('f_primaryEnd').value are the java date tool values that only start existing if the date tool is used. Pull these two values if they exist, and simply parse the date from the URL otherwise.
The clickable tab we created is essentially the equivalent of a little Greasemonkey button with a few frills that can be created in the standard Greasemonkey fashion. Wherever possible, I use Google-defined layouts for consistency with the site.
Next, we want to send out our reference data to some external server. Greasemonkey has good functionality for pulling data from other sites and servers through the use of the GM_xmlhttpRequest command. A server-end PHP or Django service might be easiest to implement. In this specific example, Kampyle wanted to use the SOAP protocol. While there is an excellent overall SOAP client for javascript by Matteo Casati, this client does not work in a plug and play fashion with Greasemonkey, and needed some modification. For any devoted SOAPers who want to try Greasemonkey, the revised javascript-soap-client code can be found in the attached file. We use the SHA256 encryption function written by Angel Marin and Paul Johnston, but that is accomplished by just copying and pasting the function into our code.
The result comes back in the form of an xml object describing each row in the table, which we parse using native Javascript/Greasemonkey methods, and pop back into the table in the way that we extracted the individual website names. A neat trick here is to call each individual row individually, and not to wait for the data to come back before calling the next row from the server. Separate listeners can wait and insert the data at their leisure. This allows our page to load up faster, and in case there is an error with one data element, it could potentially allow the rest of the rows to load in peace.
You can play around with my code here. This code is released under the BSD License. You won't be able to run the code verbatim without Kampyle's compliance, since they have changed the API calls on their server. However, much of it should be very portable to other data sources.
2 comments
Gotham said:
This post have great information but I needed a few clarifications. For one of my clients I have usual web analytics info displayed in GA. Additionally the client has call tracking data in its own database. Can I pull that info into GA in a new tab? Your mashup indicates that you added a new tab called "Kampyle", are the names of the table which shows up configurable? (e.g URL, avg grade)
Sal Uryasev said:
Hey Gotham,
Yes - as long as you have easy access to the data, you can push any data that you want into Google Analytics. If the data is completely static, you can even add it to the script. Alternatively, you could have a hosted file somewhere. In our case, the data was very dynamic, so we used a server with another web service to fetch the data.
If you click on the picture above, it'll show you the entire table, including column names that I changed around. Essentially, you have the power to change any text that you can select by a mouse. It is just a matter of knowing where to point your code.
Add a comment
Juiced Google Analytics Python API
By Sal Uryasev
May 2, 2008
Find more about:
google
analytics
python
api
pygapi
Due to the release of an official Google Analytics Data Export API, this module is now deprecated. We have an alternative python module based upon the real analytics API here, and an exploring tool with an automatic code generation capability here.
It is not official. It is not from Google. It is, however, very functional and very here. I present to you pyGAPI, the Juiced Google Analytics Python API. This module allows you to pull information from your incarnation of Google Analytics and employ it programatically into your reporting code.
Let us use iPython to peek through some code using pyGAPI.
In [3]: from datetime import date
In [4]: import pyGAPI
In [5]: connector = pyGAPI.pyGAPI(username, password, website_id="1234567")
Here we create a pyGAPI object. Behind the scenes, pyGAPI logs into Google Analytics, and downloads an identifier cookie. website_id is optional. If omitted, pyGAPI accesses the first website on the account's list. To get a list of all the site IDs to which your site has access, run the function connector.list_sites().
In [6]: connector.download_report('KeywordsReport', (date(2008,3,10), date(2008,3,31)), limit=5)
Download a report into your pyGAPI object. KeywordsReport is the name of the report. It is followed by a tuple containing the start and end dates in python date format. limit is an optional parameter that specifies the number of entries that pyGAPI should pull down. By default, it will pull in all the entries up to a maximum of 10000. Lowering this number will certainly improve performance. The entries returned are ranked by Visits, so you should get the most significant values of the bunch.
In [7]: print connector.csv()
Keyword,Visits,Pages/Visit,Avg. Time on Site,% New Visits,Bounce Rate,Visits,Subscribe,Solutions,Goal Conversion Rate,Per Visit Goal Value
juice analytics,356,5.935393258426966,314.061797752809,0.38764044642448425,0.29494380950927734,356,1.0,0.16292135417461395,1.1629213094711304,0.0
excel training,142,1.971830985915493,98.0774647887324,0.908450722694397,0.6901408433914185,142,1.0,0.0211267601698637,1.0211267471313477,0.0
excel charts,77,1.7922077922077921,95.0,0.9090909361839294,0.7792207598686218,77,1.0,0.03896103799343109,1.0389610528945923,0.0
excel skills,72,1.6527777777777777,75.29166666666667,0.9444444179534912,0.7083333134651184,72,1.0,0.0,1.0,0.0
colbert bump,70,1.3142857142857143,113.77142857142857,0.6428571343421936,0.8428571224212646,70,1.0,0.0,1.0,0.0
This function displays your report in a nice excel-ready CSV format.
In [8]: print connector.parse_csv_as_dicts(convert_numbers=True)
[{'Avg. Time on Site': 314.06179775280901, 'Per Visit Goal Value': 0.0, 'Bounce Rate': 0.29494380950927734, 'Keyword': 'juice analytics', 'Visits': 356.0, 'Pages/Visit': 5.9353932584269664, 'Subscribe': 1.0, 'Solutions': 0.16292135417461395, '% New Visits': 0.38764044642448425, 'Goal Conversion Rate': 1.1629213094711304}, {'Avg. Time on Site': 98.077464788732399, 'Per Visit Goal Value': 0.0, 'Bounce Rate': 0.69014084339141846, 'Keyword': 'excel training', 'Visits': 142.0, 'Pages/Visit': 1.971830985915493, 'Subscribe': 1.0, 'Solutions': 0.021126760169863701, '% New Visits': 0.90845072269439697, 'Goal Conversion Rate': 1.0211267471313477}, {'Avg. Time on Site': 95.0, 'Per Visit Goal Value': 0.0, 'Bounce Rate': 0.77922075986862183, 'Keyword': 'excel charts', 'Visits': 77.0, 'Pages/Visit': 1.7922077922077921, 'Subscribe': 1.0, 'Solutions': 0.038961037993431091, '% New Visits': 0.90909093618392944, 'Goal Conversion Rate': 1.0389610528945923}, {'Avg. Time on Site': 75.291666666666671, 'Per Visit Goal Value': 0.0, 'Bounce Rate': 0.70833331346511841, 'Keyword': 'excel skills', 'Visits': 72.0, 'Pages/Visit': 1.6527777777777777, 'Subscribe': 1.0, 'Solutions': 0.0, '% New Visits': 0.94444441795349121, 'Goal Conversion Rate': 1.0}, {'Avg. Time on Site': 113.77142857142857, 'Per Visit Goal Value': 0.0, 'Bounce Rate': 0.84285712242126465, 'Keyword': 'colbert bump', 'Visits': 70.0, 'Pages/Visit': 1.3142857142857143, 'Subscribe': 1.0, 'Solutions': 0.0, '% New Visits': 0.6428571343421936, 'Goal Conversion Rate': 1.0}]
This function goes the extra step and converts the CSV into a dictionary for easier programmatic use. By default, all entries will be returned as python strings. Setting convert_numbers to True, as we did here, will additionally parse the dictionary to turn all numbers into float values.
In [9]: print connector.list_reports()
('ReferringSourcesReport', 'SearchEnginesReport', 'AllSourcesReport', 'KeywordsReport', 'CampaignsReport', 'AdVersionsReport', 'TopContentReport', 'ContentByTitleReport', 'ContentDrilldownReport', 'EntrancesReport', 'ExitsReport', 'GeoMapReport', 'LanguagesReport', 'HostnamesReport', 'SpeedsReport')
This gets a list of all the reports that I have successfully tested thus far. All site-specific reports should work. A couple site-section specific reports should be included in the next update of pyGAPI.
Google is great and will release a real API soon, but until then you can download pyGAPI.
13 comments | Show all comments only the last 5 are shown
sandro turriate said:
the api looks super friendly yet powerful, I'm so glad someone finally made these reports available programatically, awesome stuff man!
Son Nguyen said:
I wonder if this violate Google Analytics' TOS and how long before Google changes something that things break apart.
Sal said:
There is certainly a risk that something could/would break. Google, however, is a company that takes the high road in terms of programming and in doing what is best for the web. The code behind the Google Analytics website is very elegant, while pyGAPI does not do screen scraping for any of the real work. The data is pulled through the data exporting system. I would say that it is unlikely that the API would break without a major overhaul of the entire GA system.
I can't specifically speak towards the TOS, but pyGAPI is doing the equivalent work of an underpaid temp who simply logs in and downloads all the requested reports. The poor temp is just getting a break. Read the TOS and use pyGAPI at your own risk.
Chris Gemignani said:
Son Nguyen,
It is far more likely that Google will provide a supported API that would supercede this. That would be the Googley thing to do.
Similar APIs have been produced around Gmail without interference. If things break, we, the community will fix it.
Tom said:
Using the example above shouldn't it be connector.list_sites() to get a list of all the site id's. Also for me this only returning the first site.
The report list, connector.report_list(), seems not to be comprehensive here's a better one:
Google Analytics Reports
-Visitors-
VisitorsOverviewReport
GeoMapReport
VisitorTypesReport
LanguagesReport
-Visitor Trending-
VisitsReport
UniqueVisitorsReport
PageviewsReport
AveragePageviewsReport
TimeOnSiteReport
BounceRateReport
-Visitor Loyalty-
LoyaltyReport
RecencyReport
LengthOfVisitReport
DepthOfVisitReport
-Browser Capabilities-
BrowsersReport
PlatformsReport
OsBrowsersReport
ColorsReport
ResolutionsReport
FlashReport
JavaReport
-Network Properties-
NetworksReport
HostnamesReport
SpeedsReport
UserDefinedReport
-Traffic Sources-
TrafficSourcesReport
DirectSourcesReport
ReferringSourcesReport
SearchEnginesReport
AllSourcesReport
KeywordsReport
-Adwords-
AdwordsReport
KeywordPositionReport
OfflineAudioReport
CampaignsReport
-Content-
ContentReport
TopContentReport
ContentByTitleReport
ContentDrilldownReport
EntrancesReport
ExitsReport
Sal said:
Thanks for catching the errors Tom!
You are correct on all three counts.
I fixed the upload so that it correctly displays list_sites() if you have more than one site in your list, and i fixed the typo here in the blog.
I'll peek through the list of reports to make that more exhaustive as well.
Matt Webb said:
This is awesome work. Do you think this python script could work in conjunction with superkaramba on Linux?
Rodrigo said:
This is great. I put this together with a Samurize desktop to display Analytics data on my desktop.
Thanks!
Ludovic said:
Very nice work. Very useful to, let's say get your most visited pages without having to maintain parallel accounting. May I ask you to licence it to an OSS licence and put it on Google Code ? Would be great.
Sebastian said:
Hello,
it work well! Great.
How can i pull the "keyword" or "country" report for a specific URL?
(use segmention)
Thanks
Thierry said:
Awesome work !
Random said:
There is now an actual analytics API:
http://code.google.com/apis/analytics/docs/gdata/gdataDeveloperGuide.html
Sal said:
I wrote a Python API wrapper that I call 'degapi' for the new analytics API to replace this old code. I have yet to put up a post and link about it, but it can be found here: http://suryasev.github.com/python-degapi/
There is an automatic python code generator for this API at http://vascodegapi.juiceanalytics.com
Add a comment
Keyword Trends in Google Analytics With Greasemonkey
By Sal Uryasev
April 23, 2008
Find more about:
webanalytics
google
analytics
hack
greasemonkey
There is a new post that re-releases the script as a Firefox Plugin. Find it here.
After the warm reception for the first version of our Enhanced Google Analytics, we decided to add some new functionality. (Nothing like a few kinds words to keep us in the giving mood.) The first script created a couple new tables in the Google Analytics interface that highlight recent changes in referral visits. It uses Greasemonkey, an add-on for Firefox that allows a user to insert javascript directly into a webpage.
Our update gives you even more ability to understand the data in Google Analytics:
- At the suggestion of Avinash Kaushik, the new script works for keyword data, helping you see how organic search traffic is changing. An increase in a keyword may indicate a general change in user interests and/or improved performance on search results.
- My coworker Pete Skomoroch also suggested that I add the ability to see declines in referrals and new keyword searches.
- With the help of Paul Irish, the script is now better able to interface with the date widget on the Google Analytics site.
(Click the above button for a simulation.)

When you click the button, your browser will download some historical data behind the scenes, and display a nice summary of the best and worst performing keywords/referring domains.
Installation Instructions:
Firefox 2.0+
Greasemonkey
googleanalyticsdownloade.user.js
If you don't already have Firefox, install it. Install Greasemonkey, and do the required Firefox restart. You should see a handsome monkey peeking at you from the bottom right hand corner of your browser. Open the script file in your firefox browser, and Greasemonkey should give you an option to install the script.
Afterwards, log into Google Analytics, and navigate to your Referring Sources or Keywords Tab. Click the button.
Configuring the script:
We spent some time trying to find convenient default settings here at Juice Analytics, so the script should work straight out of the box. Some users, however, may find it convenient to alter some of these configurations. To do so, in Firefox, go to Tools=>Greasemonkey=>Manage User Scripts..., select Google Analytics Downloader, and then click Edit in the lower left corner of the window. This should open up the script file in a text editor. If your computer does not have a default text editor configured, you may have to choose one. 'c:\windows\notepad' is a good bet for Windows machines.
This is what you should see:

The bracket labeled 'keywords?' controls defaults for the Keywords page, and correspondingly, 'referring_sources?' controls the Referring Sources page.
To change the settings, simply change the corresponding variable to your preferred default. Make sure to refresh your Google Analytics webpage, if you have it open, so the new settings are loaded.
Now for the nitty gritty configuration details:
- display_limit: This controls the maximum entries that each table will contain. This may be useful for large, sprawling sites.
- growth_tolerance: This is the percentage growth parameter. Changing it to .10, for example, will catch everything that has grown by 10%, as opposed to the default 50% and 20%, respectively.
- minimum_number_elements: This is a significance benchmark that can be used to limit what is displayed upon the screen. By default, only keywords with at least 10 elements are displayed upon the screen. Referring Sites does not have a minimum by default, but one can be set if desired.
- limit: Limit is more of an internal parameter that determines how many entries should be downloaded from Google in order to get the results that are visible here on the page. Lower the limit to increase speed. If the limit is set to a very high number, you will get the largest result set, but you will have to sit around for a while for the results to load. Since the results are downloaded ordered by volume, raising the limit from the default numbers will not actually give more significant results. You will simply get more of the smaller results, such as keywords with only 1 hit.
- look_back: This is a very important parameter. The script uses the date displayed upon your Google Analytics page to determine the full range that you want to consider in your results, but 'look_back' determines how many of those days are used for the significance test. So, say the range you have displayed in Google is March 23 - April 22 and your look_back is 7 days. The script will compare the average referrals for a given keyword from April 16-22 to the average from March 23-April 15, and will return the keyword only if the recent average is 20% higher than the rest of the time period. Thus, if you want to increase the total range of the data, change the dates on the actual webpage. Change 'look_back' only if you want to change the period of significance.
Happy analyzing!
43 comments | Show all comments only the last 5 are shown
Avinash Kaushik said:
Sal: Thanks so much for this enhancement to the first script, given all the attention on Search I think this is absolutely super valuable. I am on a recommendation overdrive on this (I have a two slides on Juice in my official presentations! :).
Thank you again, this is excellent.
-Avinash.
SM said:
Thanks for pulling these scripts together, very useful.
Patrick H. said:
This is great, thanks a lot for posting it. Very good addition to GA.
James said:
Why didn't google analytics do this months ago?
This is great!
I wear many hats at my company. As entertaining as it may be, I don't have the time to play in excel.
Thank you!
Brian said:
Great work guys! This is killer!
New to the blog, but you've won a reader. ;)
Sascha said:
Thats perfect man!
Please more of this awesome features :)
Best Regards from germany
Tim said:
Great tool! But sadly it doesnt work anymore in FF 3. Do you have an update?
Regards, Tim
spudart said:
Yes, I love this tool. Unfortunately ever since Firefox automatically updated itself from 2.0.0.14 to 2.0.0.15, it doesn't work anymore in 2.0.0.15. An update would make my day.
Sal Uryasev said:
Silly Firefox. Thanks for pointing it out!
The script should work if you reinstall it now.
Steve said:
I'm having trouble getting any results to return when running the script. In one week we have about 9,000 different keywords sending traffic to our site - any thoughts on configuring some of the settings to retrieve results? I just get the "loading..." button showing for minutes on end. Maybe increasing the growth_tolerance or lowering the limit?
michelle said:
I don't understand how to get grease monkey to work - i followed all the install directions fine... now that its installed though I don't know what to do... can anyone help me? not sure I'm even in the right place :/
Teri said:
Your instructions state "Open the script file in your firefox browser." This is where I am getting lost. How and where do I open it? The link takes me to another page, and I can't download the file. Help, I am stuck.
Teri said:
Never mind - I found the install link on the scripts page. I have installed greasemokey and the script, re-started Firefox, gone to Google Analytics, and I am not getting the "Who sent me unusual traffic" button. Please advise.
Sal Uryasev said:
Are you guys going to the Keyword/Referring Sites sections of Google Analytics? The buttons are hanging out on those specific pages.
Laurence said:
Thanks so much for this script! One question, I was browsing through the code but haven't done much web coding in a while. If I wanted to adapt this for bounce rate (to see the outliers for bounce rate) what would I want to change?
I feel like I would just need to change which element in the array you are comparing and then sort by that element but I'm sure it's probably more complex than that. If you at least point me in the right direction, I can mess around with it some more.
Bjoern said:
Hi,
the script has been very useful for us, but with last week's small Google Analytics Rebrush it fails to work for me on Firefox2 and Greasemonkey. Is this something you can confirm or are we (two colleagues) a singular problem?
Chris said:
Hi There,
This has been working for me very well but lately , it does not seem to go past the loading stages when I click the button?
I dont see any new results...
What do you think has happend?
I use the latest version of greasemonkey and firefox?
Any ideas?
Chris Avery
Bleue said:
I have the same problem: loading...
but nothing happens.
Chris said:
Hmm, I think this is a new problem, I am looking for a solution, if I fine one I will post it here.
Chris Avery
Sal Uryasev said:
I updated the script. Google Analytics changed their URL around slightly, adding "#lts=1221579205724", which messed with the script.
Uninstalling and reinstalling the script would probably be the easiest way to install the update.
Chris said:
Hey,
Thanks for taking the time to do that , we all appreciate it
Cheers
Bjoern said:
Hi Sal,
thanks for updating this so quickly. I'll check it out. It really is a great script.
@ All: Sal and the Juice Analytics Team might appreciate it, if users became "fans" of the script at the download location at userscripts.org - so that there is some recognition.
Katie said:
Thank you so much for your clear explanation of how to intgrate this awesome script into GA. Anytime you can dumb it down a little for us non-web developers/coders it is much appreciated. Marketing Mavens everywhere thank you!
don said:
Sal,
Thanks to you and yours for sharing this script. I installed it into my GA account and saw it working in both reports. Now, when I try to run the script in the keywords report, nothing happens. The button changes to say "loading" but no data gets presented. In the referring sites report, the script appears to be working. I am not a programmer or developer. Is there anything I can try to get the script working again for keywords?
Sal Uryasev said:
Hey Don,
Is it possible that you installed the script initially before September?. Google did a redesign of the site that necessitated a fixup of the script around that time.
don said:
Sal,
Thanks for responding to my plea for help. I don't think it was back that far when I installed the script. Unfortunately I forgot to make a note in my configuration tracking document of the day I installed the script. It sounds like you might suggest that I download the script and reinstall it? If so, is there anything I need to do to uninstall it first?
Sal Uryasev said:
It is worth a try.
If you go to Tools => Greasemonkey => Manage User Scripts..., you can select uninstall there.
Milttea said:
HI
After installing Greasemonkey I navigate to Tools => Greasemonkey => Manage User Script. I do not see Google Analytics Downloader. Please advise
don said:
Sal,
I tried the uninstall, reinstall method and still get the same results. The button changes to "loading" but nothing happens. It continues to work in the referring sites report. Any thoughts on what else I can check?
Gavin Doolan said:
Hi
This is a great script, one way to improve it though would be to have a way to visual the data in terms of a graph.
Currently you can try the Google Analytics graph options, but it doesn't really provide any insightful data.
Nice script and thanks.
Cheers,
Gavin.
GL said:
Along with Milttea above Google Analytics Downloader does not appear.
don said:
FYI, I figured out that my problem was being caused by a Firefox extension, SEO for Firefox. Disabling that and reinstalling Greasemonkey and the script have solved my issue.
Darren said:
I just downloaded this yesterday and I seem to have the same "Loading" hang that others have mentioned. I've tried disabling the other user scripts I use (only a couple) but that doesn't seem to help.
The potential looks great - especially where our top keywords are always the same and we want to see what's generating any new/incremental buzz.
Thanks
Clive said:
Great stuff. One of two recommendations put forward by Avinash so thank him for that and thank 'you' for making a very very useful tool.
dot said:
I've been using this for a while and I love it, but recently it stopped working, and just sits there on "loading". I tried downloading the new version. I tried uninstalling and reinstalling greasemonkey. No luck. I don't have any other scripts running, and I'm running firefox 3.0.6. anyone have any ideas? this script is so useful it's addictive, and now I'm lost without it!
thanks,
dot
dabbled.org
Sal Uryasev said:
Hey dot,
Thanks for letting me know. I'll take a look at it. Occasionally google changes the way they run things, and the script needs to be updated. I'm actually really close to re-releasing this script as a firefox plugin, so that I can utilize the automatic updates, and then no-one has to re-download or notice it being broken in the future.
dot said:
Thanks! that would be cool..
Sal Uryasev said:
There is an version of the new and improved Firefox addon that encapsulates this functionality. It is faster, sleeker, and it will have automatic updates. You can find it here: https://addons.mozilla.org/en-US/firefox/addon/11120
It is still experimental so users who want to download it have to log into addons.mozilla.org. I need a few user reviews before I can nominate the addon to be fully public (and easy to install). So, leave reviews if you happen to wander by there.
I'll publish a blog post once I can get the addon into the public part of the addons site.
Dave H said:
Hi,
Installed fine both using Greasemonkey/FF Add-In, however when I push the button I can see the table briefly appear but almost instantly disappear. Instead, multiple sort asc/desc triangles appear in the main Site Usage table. Does it interfere with other plug-ins?
(Also it only works when you access GA directly (the orange version) as opposed to via Adwords (the green version) - but that's not the main problem!)
Sal Uryasev said:
Hey Dave,
My particular plugin is very non-intrusive. While it should not interfere with any other plugins that I know about, if you have something that is quite intrusive, there is always a chance. It does sound as if you have something extra installed that kicks off after the script runs... maybe some kinds of special scripts to neaten up webpages?
steve said:
Hey Dave,
Thanks for the plugin, but I can't get it to work.:( I'm on firefox 3.0.1 and I can see it installed and I restarted like it asked. Then I go to google analytics page and no blue button.
Can you help me? I realize this is in Beat and I should expect this.
norad73 said:
The button shows 3 tables but they are empty... I tried changing the date period but they are still empty... any ideas?
Lee said:
The question is why the heck doesn't GA do this already? Been beating my head trying to find out how to do this with GA but you've provided the only solution.
I'd really love to be able to change the ranges -- so be able to compare against the same time last year, 30 days and then the default 7 days. Any help on how to do this?
Thanks for an awesome and life saving job.
Add a comment
Enhancing Google Analytics Using Greasemonkey
By Sal Uryasev
April 11, 2008
Find more about:
webanalytics
google
analytics
hack
greasemonkey
There is a new post that re-releases the script as a Firefox Plugin. Find it here.
My boss Zach has a problem. Every four hours the craving strikes him. No matter where he is, he pulls out his shiny Macbook Pro and navigates to Juice's Google Analytics site. He pulls up the list of referrers to our site and meticulously searches for new domains. He has an freakish ability to pick out IP addresses that have never linked to us before. Even so, there had to be a better way.
I wondered whether Greasemonkey might be able to help. Greasemonkey is an extension for Firefox that allows users to install custom javascript when you visit a specific website. These scripts can add a delete button for Gmail, automatically display lyrics to your YouTube music video, or do pretty much anything else you would want to enhance the functionality of a website.
After poking around the subtleties of the Google Analytics interface, I came up with a little script that can identify the new referrals that Zach so desperately craves. When navigating to the "Referring Sites" section of Google Analytics, the script add the following button to the interface.

Pushing the button downloads all the referrer data for the date displayed in the Google Analytics range, as well as a similar set of data for the range up to, but not including, the last three days. The difference between the two data sources is used to calculate all of the results. The specific number of days can be changed by editing the first line of the script. Greasemonkey then displays the results in two tables above the original Referrer table. (Greasemonkey works entirely within your browser shell, so your data should be quite secure.)

The first table shows any sites that have displayed more than a 50% increase in visits over the last 3 days as compared to the rest of the time range. The second shows all new recent sites that do not appear at all more than 3 days ago. This can be quite useful to anyone, who, like Zach, absolutely needs to know about any new and exciting inbound links.
Installation Instructions:
Firefox 2.0+
Greasemonkey
googleanalyticsdownloade.user.js
If you don't already have Firefox, install it. Install Greasemonkey, and do the required Firefox restart. You should see a handsome monkey peeking at you from the bottom right hand corner of your browser. Open the script file in your firefox browser, and Greasemonkey should give you an option to install the script.
Afterwards, log into Google Analytics, and navigate to your Referring Sources Tab. Click the button.
21 comments | Show all comments only the last 5 are shown
John Henson said:
Sal,
Some really nice JavaScript. I really love to see stuff like this done with GA.
Would you mind if re-use your export logic (with credit)?
Thanks,
John
Sal said:
John,
Thanks for the kind comment. Feel free to reuse it at your leisure.
-Sal
Avinash Kaushik said:
Awesome enhancement!
I am also afflicted with the same disease as Zach, and now I have a easy cure!
Absolutely love it.
Thanks,
Avinash.
Paul Irish said:
Just took a look at the code to scrape the new date without changing the URL...
var DI = new unsafeWindow.DateInputView;
var startdate = DI.parseDate(DI.primaryBeginField.value)
var enddate = DI.parseDate(DI.primaryEndField.value)
I didn't test this within the context of the script, but it hands back some nice Date objects that you can play with... hopefully it simplifies things for you.
:)
Sal said:
Thanks Paul!
I added some increased date functionality built upon your suggestion.
Al said:
Great tool, nice features!
Thank you.
Al
Tim Leighton-Boyce said:
This is extremely useful, thank you. This certainly helps gain some clearer insight by cutting through the usual suspects and highlighting the items of interest.
Rich said:
This is so cool - thanks to Avinash for pointing it out and of course the juice team for building this functionality!
Matthew said:
I'm not a programmer but like the Greasemonkey script. If I wanted to change the number of days it looks back to 30 rather than 3, do I just make the following change? "look_back = 30"
Chris Gemignani said:
Matthew,
We've got a new script coming probably tomorrow with a few new features. We'll address customization in that post. It really isn't hard. :-)
Tim said:
Hi,
I read about the script and desperately want to have it. I just couldn't make it work. I installed Greasemonkey, installed the script and restarted the browser. Than I logged into my Google Analytics Account and looked at the referring sites report - no new buttom to click on.
Have I done something wrong? A little more detailed instructions would be helpful as I really want to use that script!
Thanks in advance,
Tim
Dave said:
I installed the script and it looks great! I'm always digging in to see who has sent us new traffic, so thanks! I'd like to make the "look_back" date different as well, but I don't know where to find the script to make that change and save it. Any help on where to find the installed script to edit it?
Josh Chambers said:
Thanks a lot! I'll be writing a blog post about this for sure. Great script.
Neerav said:
thanks for this excellent tool
Is there somewhere I can subscribe to keep up to date with new versions?
Sal Uryasev said:
Matthew/Dave: The new post (linked at the top of the page) should have the details you need.
Tim: It sounds as if you did everything correctly. Is the script visible if you go to Tools=>Greasemonkey=>Manage User Scripts?
Neerav: There really is no great way to subscribe to just this script. I will link in the relevant places if anything changes though.
Neerav said:
Hi Sal
Enabling the "subscribe to comments" plugin could help. That way you can post a comment saying there's a new version and everyone who commented gets that comment as an email
Tim said:
Hi Sal,
thank you so much - I just reinstalled it and now it works! This is really fantastic!
Thanks,
Tim
Nate Sidmore said:
Awesome tool Sal, (thanks to Avinash for the tip).
I did run into a problem with the Firefox pop-up message "Warning: Unresponsive script". However that problem can be solved by lengthening the time allowed for scripts to run. For more details go to http://lifehacker.com/software/firefox/put-off-firefox-15s-unresponsive-script-dialogue-162574.php
However I was bummed when after setting the time allowance to 10 minutes, and clicking the "Who Sent Me Unusual Traffic" button in GA, the script ran for 9 min 38 sec before returning results. Any tips on getting quicker returns?
Chris Gemignani said:
Nate:
Thanks for the encouragement. If you check our "Keyword Trends" Greasemonkey script (linked at the start of this post), we write about how to change the parameters in the script to make things run faster.
Edwin said:
How can I only bring up the report for non-paid keywords? Selecting it and then returning the results, still brings up cpc words as well.
Shankar said:
This is a nice tips, I 'll use it fro my site http://www.onlinegk.com






35 comments | Show all comments only the last 5 are shown
yadab das said:
The PyGTrends.py API looks really fascinating to me. I have almost converted the Code to Java and will publish this week with a Swing Interface. Any better suggestions on that?
Archie said:
Hi Yadab! Could you please share the Java code you have written? I am also working on it. Please contact me by email.
aavaliani (at) gmx.net
Gautham Ramachandran said:
The PyGTrends.py API is really awesome. I have a question though. Does Google frown upon iterative pings to Google Trends to pull Relative traffic. To make it more specific, if I have 2,000 keywords and I code iterative pulls from Google trends, do I stand a chance of getting banned?
Gautham.
Sal Uryasev said:
Yes and No.
I believe that Google tends to be generous regarding use of their services. They want people to get the maximum utility out of the products, but they don't want their generosity to be abused. The login requirement for downloading the Google Trends data is probably there just for that reason. The cap is probably quite large, but there certainly is one. I wouldn't build a webservice (without having users use their own account). You may have more luck if you lump many keywords per call, and spread out your data gathering over longer periods of time.
James Solo said:
This Python script is great and provides an excellent solution. However, I have never used Python before so I was hoping someone could email (james.solo |AT| mathworks.com) me step by step instructions on how to modify this script to work with keywords of my choice ( I have 40 total ) and to grab data from 2004 to-date using the "CSV with relative scaling" data file.
Many thanks,
James
Arjun said:
I have been using the pyGTrends module and have encountered problems when using keywords with more than one word. For instance, "air express" was one of the keywords. It has a search history--when I manually download the data from Google Trends, the historical data shows up fine. However, when I use the pyGTrends module, the data comes out as all zeroes.
The same problem occurs with all keywords/phrases that contain more than one word. Is pyGTrends compatible with only one-word keywords, and if not, how do I fix this problem?
If someone could email me, arjunrmodi at gmail dot com, that would be great. Thanks.
A said:
Great module.
Sal Uryasev said:
Arjun: It should work now. Thanks for pointing it out!
CurtD said:
Hi,
I am wondering if you Google Gurus know of how to attack a project I am trying to accomplish. I want to be able to run a script, query, software that can give me the answer to something like the following.."I want to know what search terms have been searched 1500 a day on average and have less than 200,000 results". The 1,500 and 200,000 are variables that can be changed. It would be cool to be able to have a drop down menu to say "last 30 days" "last 90 days" etc. I have posted this project on Elance and Guru and I have mostly gotten "never heard of it, can't be done." Some talked of building a library through API and then running queries on it. Does this query exist or how can I go about getting it done?
Sal Uryasev said:
Hey Curt,
The problem lies with the fact that Google Trends does not have any absolute numbers. You can only get results as they are relative to other results.
It appears to be an intentional omission. The only way to get at that data could be to get hired as a Google employee.
Clemens said:
Hi Sal,
This is really a great tool. I am facing the same issue as Arjun. I am not able to extract keyword combinations for phrases with e.g. two words. Single keywords work fine...
You mentioned, that this issue has been resolved. I downloaded the latest version from github, can you give me some advise how to extract keyword combinations, or how to adjust the code respectively.
Thanks in advance, this is really a great work.
Johan said:
Hi,
The code has (obviously) changed with later versions which makes the instructions above harder to follow. Where to enter the keywords, in this line?
def download_report(self, keywords, date='all', geo='all', geor='all', graph = 'all_csv', sort=0, scale=0, sa='N'):
I have never used Python before...
atleta said:
The lib does not work correctly at the moment. It messes up the headers, the header for the first column (e.g. 'Week') is not inserted by the code as it generates the headers from the first line, while the all tables ('sections') have one more column whose name you actually use to identify the section. Thus one solution is to insert the requested section name as the name of the first column.
Sal Uryasev said:
Thanks for the bugfix atleta. I implemented your idea.
Sal Uryasev said:
Clemens,
You can just enter two-word strings wherever you would enter one-word strings: 'google analytics' instead of 'banana'
Johan,
The 'self' is a pythonic reference to the object itself.
There is a bit more of a description here:
http://www.ibiblio.org/g2swap/byteofpython/read/self.html
Gautamm said:
I really loved this and it works great.
I have one question though, whenever I put in a single query it splits it into alphabets. How does one circumvent this?
Sal Uryasev said:
Hey Guatamm,
The first parameter to download_report needed to be a list or tuple, so something like ('bread') would have had to be ('bread',) or ['bread']. Python can be slightly annoying, where it can treat parentheses as simply parentheses, unless there is a comma to make it a tuple. I added in a small fix so that is no longer necessary, if you want to re-pull the code from git.
redneckjedi said:
Sal,
This is great, and it was working great for me. However, after a few calls in an iterative script, I started getting a 503 Service Unavailable message. I'm guessing they figured out I was running a script. Any idea how to get around this?
Sal Uryasev said:
Hey Redneckjedi,
I like your name.
My best guess is to try drastically slowing down your iterative script with sleep(x). Making multiple repeat calls is probably not built into Google's design, so my guess is that you're hitting some sort of rate limit. Google tends to be generous with this sort of stuff, so by spacing out the calls, it may help quite a bit.
redneckjedi said:
Thanks Sal. That's what I've been trying, and it works now that I'm on a different computer with a different IP. Any idea on how long I'll have to wait before I can run it on my original machine?
Sal Uryasev said:
I have no idea... sorry. If you find out though, this might be a good piece of information to stub into the post above.
atleta said:
redneckjedi: I've also run into this. No idea for how long they will ban your IP but you could use a proxy in the meanwhile.
Sal Uryasev said:
It looks like it is just a daily limit. There is a forum discussion about it here: http://www.thirtydaychallenge.com/forums/general-chat/5364-google-trends-limit-wtf.html
pstobbs said:
Hi Sal
Great tool- very useful indeed, many thanks.
I'm new to Python but pulled together a little code to cycle through different keyword inputs. It seemed to be working fine but then I ran into the issue mentioned by redneckjedi unfortunaltely. I had included a sleep(10) but clearly that wasn't enough and I am now getting Http Error 503: Service Unavailable. When I access the site manually I am greeted with a captcha so I can still run the queries manually just not through the script- has anyone else come across this? I hope the block will stop soon and I will then add a much longer sleep. I only have about 20 or so keyword sets which should be ok I think?
Anyone any ideas how long the ban lasts?
cheers
Piers
Below is my rudimentary loop which I'm sure could be improved upon
#----- import pygtrends
import sys
sys.path.append('c:\\python_code')
from pyGTrends import pyGTrends
connector = pyGTrends('username','password')
#----- initiate output file
import time
final_output = str()
#-----load csv inputs file
import csv
gtrends_input = csv.reader(open("google_trends_inputs.csv","rb"))
gtrends_input_list = []
gtrends_input_list.extend(gtrends_input)
#---- define loop
loop_range = range(1,len(gtrends_input_list))
for x in loop_range:
#---- parameters
tmp_row = gtrends_input_list[x]
tmp_terms = tmp_row[:5]
tmp_geo = tmp_row[5]
#---- save parameters to output
final_output = final_output + tmp_geo + '\n'
#---- get data
connector.download_report((tmp_terms),geo=tmp_geo,scale=1)
#---- append output
tmp_output = connector.csv() + '\n'
final_output = final_output + tmp_output
time.sleep(100)
print 'done',x, tmp_terms, tmp_geo, tmp_symbol
#---- end loop
#---- output
outfile = open('output.csv','w')
outfile.write(final_output)
outfile.close()
Dheeru said:
My IP was blocked for a day after submitting less than 100 queries.
Bas said:
resp = self.opener.open(self.url_ServiceLoginBoxAuth,params).read();
print " resp ::"+resp;
Im getting the response from google that says "Your browser's cookie functionality is turned off. Please turn it on". ie, not able to login to google account for trends.
Please help me out.
Sal Uryasev said:
Alright.
It turns out that Google changed their Login algorithm. Where in the past I could construct a unique 11-digit ID out of thin air, the new algorithm matched the 11-digit ID against one written into a cookie.
The reason why jcp20 were failing some of the time is probably due to Google rolling out their new login algorithm, and some of the calls would hit an older server, while others would fail on a new server. The data pulls should work consistently at this point.
I have also simplified the convenience functions, with the expectation that the user uses the csv.reader or the csv.DictReader features of the standard library.
Recommended example:
from csv import DictReader
r = pyGTrends(username, password)
r.download_report(('pants', 'skirt'))
d = DictReader(r.csv().split('\n'))
Josh said:
This is a great idea--I've been waiting for google to roll out their long promised API...
Has anybody but Dheeru experienced the limitations of this API. I would like to set up a datasource to integrate cleanly into a cakePHP app, but it doesn't seem worth the while if google cuts off access to an account/ip(?) after a small number of queries.
Anybody have any specifics on this?
SivaThumma said:
No doubt, This code is an excellent start.,
But As-it-is I tried your code, I am getting
Exception("Could not find requested section") [line number:101]
Can anyone reply on what should I do exactly ?
I would Thank anyone who takes kindness to mail me at sivatumma@hotmail.com.
Saravanan said:
Hi Sal,
Thank you so much for the wonderful code - really looks promising.
A quick question..I had used the following script
from pyGTrends import pyGTrends
from csv import DictReader
r = pyGTrends('my email', 'passwd')
r.download_report(('spain', 'wine'))
print DictReader(r.csv().split('\n'))
As a result of it, one "pyGTrends.pyc" got created and output was printing as "<csv.DictReader instance at 0x8b8998>". I didn't see any error display btw..
How do i get the CSV output of the report? Could you please assist with me on this?
Thanks
David said:
I have the same questions as the above.
also, is there a document for using this API.
Thanks
Aloysius Adrian said:
I want to ask about section in csv function. I was trying to get a certain csv, but the interpreter stops at line 115 that produces error message : "KeyError: 'main'"
I want to ask about the other parameter. I can not print the csv because of the section parameter there.
Thanks.
Sal Uryasev said:
The DictReader module is a Python convenience module for reading data into a dictionary. Its use is optional.
One thing that can be done if there are any issues is to print connector.raw_data, and that would display the direct result from Google Analytics. Google sometimes displays additional username/login related problems that the module may not have accounted for.
David said:
From yesterday, This script can't do print connect.csv(). It always returns "Could not find requested section" error. Do you know where the problem is?
I also download the original script in case I modify anything but the error remains.
Thanks
Aloysius Adrian said:
@David
I, also, can not get the connect.csv()
I contacted Mr. Uryasev, and he suggested that I try print connect.raw_data
I did that, but the result/output was : "You must be signed in to export data from Google Trends"
I wonder if you also get that kind of output..
said:
Add a comment