Juice Analytics

The PyGTrends.py API looks really fascinating to me. I have almost converted the Code to Java and will publish this week with a Swing Interface. Any better suggestions on that?

Hi Yadab! Could you please share the Java code you have written? I am also working on it. Please contact me by email.

aavaliani (at) gmx.net

The PyGTrends.py API is really awesome. I have a question though. Does Google frown upon iterative pings to Google Trends to pull Relative traffic. To make it more specific, if I have 2,000 keywords and I code iterative pulls from Google trends, do I stand a chance of getting banned?

Gautham.

Yes and No.
I believe that Google tends to be generous regarding use of their services. They want people to get the maximum utility out of the products, but they don’t want their generosity to be abused. The login requirement for downloading the Google Trends data is probably there just for that reason. The cap is probably quite large, but there certainly is one. I wouldn’t build a webservice (without having users use their own account). You may have more luck if you lump many keywords per call, and spread out your data gathering over longer periods of time.

This Python script is great and provides an excellent solution. However, I have never used Python before so I was hoping someone could email (james.solo |AT| mathworks.com) me step by step instructions on how to modify this script to work with keywords of my choice ( I have 40 total ) and to grab data from 2004 to-date using the “CSV with relative scaling” data file.

Many thanks,
James

I have been using the pyGTrends module and have encountered problems when using keywords with more than one word. For instance, “air express” was one of the keywords. It has a search history–when I manually download the data from Google Trends, the historical data shows up fine. However, when I use the pyGTrends module, the data comes out as all zeroes.

The same problem occurs with all keywords/phrases that contain more than one word. Is pyGTrends compatible with only one-word keywords, and if not, how do I fix this problem?

If someone could email me, arjunrmodi at gmail dot com, that would be great. Thanks.

Great module.

Arjun: It should work now. Thanks for pointing it out!

Hi,

I am wondering if you Google Gurus know of how to attack a project I am trying to accomplish. I want to be able to run a script, query, software that can give me the answer to something like the following..”I want to know what search terms have been searched 1500 a day on average and have less than 200,000 results”. The 1,500 and 200,000 are variables that can be changed. It would be cool to be able to have a drop down menu to say “last 30 days” “last 90 days” etc. I have posted this project on Elance and Guru and I have mostly gotten “never heard of it, can’t be done.” Some talked of building a library through API and then running queries on it. Does this query exist or how can I go about getting it done?

Hey Curt,

The problem lies with the fact that Google Trends does not have any absolute numbers. You can only get results as they are relative to other results.

It appears to be an intentional omission. The only way to get at that data could be to get hired as a Google employee.

Hi Sal,

This is really a great tool. I am facing the same issue as Arjun. I am not able to extract keyword combinations for phrases with e.g. two words. Single keywords work fine…
You mentioned, that this issue has been resolved. I downloaded the latest version from github, can you give me some advise how to extract keyword combinations, or how to adjust the code respectively.

Thanks in advance, this is really a great work.

Hi,
The code has (obviously) changed with later versions which makes the instructions above harder to follow. Where to enter the keywords, in this line?

def download_report(self, keywords, date=’all’, geo=’all’, geor=’all’, graph = ‘all_csv’, sort=0, scale=0, sa=’N'):

I have never used Python before…

The lib does not work correctly at the moment. It messes up the headers, the header for the first column (e.g. ‘Week’) is not inserted by the code as it generates the headers from the first line, while the all tables (‘sections’) have one more column whose name you actually use to identify the section. Thus one solution is to insert the requested section name as the name of the first column.

Thanks for the bugfix atleta. I implemented your idea.

Clemens,
You can just enter two-word strings wherever you would enter one-word strings: ‘google analytics’ instead of ‘banana’

Johan,
The ‘self’ is a pythonic reference to the object itself.
There is a bit more of a description here:
http://www.ibiblio.org/g2swap/byteofpython/read/self.html

I really loved this and it works great.

I have one question though, whenever I put in a single query it splits it into alphabets. How does one circumvent this?

Hey Guatamm,

The first parameter to download_report needed to be a list or tuple, so something like (‘bread’) would have had to be (‘bread’,) or ['bread']. Python can be slightly annoying, where it can treat parentheses as simply parentheses, unless there is a comma to make it a tuple. I added in a small fix so that is no longer necessary, if you want to re-pull the code from git.

Sal,

This is great, and it was working great for me. However, after a few calls in an iterative script, I started getting a 503 Service Unavailable message. I’m guessing they figured out I was running a script. Any idea how to get around this?

Hey Redneckjedi,

I like your name.

My best guess is to try drastically slowing down your iterative script with sleep(x). Making multiple repeat calls is probably not built into Google’s design, so my guess is that you’re hitting some sort of rate limit. Google tends to be generous with this sort of stuff, so by spacing out the calls, it may help quite a bit.

Thanks Sal. That’s what I’ve been trying, and it works now that I’m on a different computer with a different IP. Any idea on how long I’ll have to wait before I can run it on my original machine?

I have no idea… sorry. If you find out though, this might be a good piece of information to stub into the post above.

redneckjedi: I’ve also run into this. No idea for how long they will ban your IP but you could use a proxy in the meanwhile.

It looks like it is just a daily limit. There is a forum discussion about it here: http://www.thirtydaychallenge.com/forums/general-chat/5364-google-trends-limit-wtf.html

Hi Sal
Great tool- very useful indeed, many thanks.

I’m new to Python but pulled together a little code to cycle through different keyword inputs. It seemed to be working fine but then I ran into the issue mentioned by redneckjedi unfortunaltely. I had included a sleep(10) but clearly that wasn’t enough and I am now getting Http Error 503: Service Unavailable. When I access the site manually I am greeted with a captcha so I can still run the queries manually just not through the script- has anyone else come across this? I hope the block will stop soon and I will then add a much longer sleep. I only have about 20 or so keyword sets which should be ok I think?

Anyone any ideas how long the ban lasts?
cheers
Piers

Below is my rudimentary loop which I’m sure could be improved upon

#—– import pygtrends
import sys
sys.path.append(‘c:\\python_code’)
from pyGTrends import pyGTrends
connector = pyGTrends(‘username’,'password’)

#—– initiate output file
import time
final_output = str()

#—–load csv inputs file
import csv
gtrends_input = csv.reader(open(“google_trends_inputs.csv”,”rb”))
gtrends_input_list = []
gtrends_input_list.extend(gtrends_input)

#—- define loop
loop_range = range(1,len(gtrends_input_list))

for x in loop_range:

#—- parameters
tmp_row = gtrends_input_list[x]
tmp_terms = tmp_row[:5]
tmp_geo = tmp_row[5]

#—- save parameters to output
final_output = final_output + tmp_geo + ‘\n’

#—- get data
connector.download_report((tmp_terms),geo=tmp_geo,scale=1)

#—- append output
tmp_output = connector.csv() + ‘\n’
final_output = final_output + tmp_output
time.sleep(100)
print ‘done’,x, tmp_terms, tmp_geo, tmp_symbol

#—- end loop

#—- output

outfile = open(‘output.csv’,'w’)
outfile.write(final_output)
outfile.close()

My IP was blocked for a day after submitting less than 100 queries.

resp = self.opener.open(self.url_ServiceLoginBoxAuth,params).read();

print ” resp ::”+resp;

Im getting the response from google that says “Your browser’s cookie functionality is turned off. Please turn it on”. ie, not able to login to google account for trends.

Please help me out.

Alright.
It turns out that Google changed their Login algorithm. Where in the past I could construct a unique 11-digit ID out of thin air, the new algorithm matched the 11-digit ID against one written into a cookie.

The reason why jcp20 were failing some of the time is probably due to Google rolling out their new login algorithm, and some of the calls would hit an older server, while others would fail on a new server. The data pulls should work consistently at this point.

I have also simplified the convenience functions, with the expectation that the user uses the csv.reader or the csv.DictReader features of the standard library.

Recommended example:

from csv import DictReader
r = pyGTrends(username, password)
r.download_report((‘pants’, ‘skirt’))
d = DictReader(r.csv().split(‘\n’))

This is a great idea–I’ve been waiting for google to roll out their long promised API…

Has anybody but Dheeru experienced the limitations of this API. I would like to set up a datasource to integrate cleanly into a cakePHP app, but it doesn’t seem worth the while if google cuts off access to an account/ip(?) after a small number of queries.

Anybody have any specifics on this?

No doubt, This code is an excellent start.,
But As-it-is I tried your code, I am getting

Exception(“Could not find requested section”) [line number:101]

Can anyone reply on what should I do exactly ?

I would Thank anyone who takes kindness to mail me at sivatumma@hotmail.com.

Hi Sal,

Thank you so much for the wonderful code – really looks promising.

A quick question..I had used the following script

from pyGTrends import pyGTrends
from csv import DictReader
r = pyGTrends(‘my email’, ‘passwd’)
r.download_report((‘spain’, ‘wine’))
print DictReader(r.csv().split(‘\n’))

As a result of it, one “pyGTrends.pyc” got created and output was printing as ““. I didn’t see any error display btw..

How do i get the CSV output of the report? Could you please assist with me on this?

Thanks

I have the same questions as the above.
also, is there a document for using this API.
Thanks

I want to ask about section in csv function. I was trying to get a certain csv, but the interpreter stops at line 115 that produces error message : “KeyError: ‘main’”

I want to ask about the other parameter. I can not print the csv because of the section parameter there.

Thanks.

The DictReader module is a Python convenience module for reading data into a dictionary. Its use is optional.

One thing that can be done if there are any issues is to print connector.raw_data, and that would display the direct result from Google Analytics. Google sometimes displays additional username/login related problems that the module may not have accounted for.

From yesterday, This script can’t do print connect.csv(). It always returns “Could not find requested section” error. Do you know where the problem is?
I also download the original script in case I modify anything but the error remains.
Thanks

@David
I, also, can not get the connect.csv()

I contacted Mr. Uryasev, and he suggested that I try print connect.raw_data

I did that, but the result/output was : “You must be signed in to export data from Google Trends”

I wonder if you also get that kind of output..

I have the same problem as David and Aloysius. If anyone has overcome this, please let me know.

How do you select multiple years? e.g. 2008 2009 I’ve tried 2008-2009 and 1/2008 12m but no luck…

Thanks so much for this. I’ve just launched a site that uses this API to build a U.S. map tracking “flea” search activity, indicating flea prevalence. Have a look: http://banfieldfleafighter.com.

To do this, I set up a cron to poll Google Trends once every morning and write a small CSV file. This file gets parsed by PHP and fed to Flash with every visit to the site.

Aside from the very occasional “Cannot parse GALX out of login page” error, it works like a charm.

Hi Sal,

Great works! It’s a very useful lib.

Many thanks to you

You are asking for the Holly Grial. I also want that!

Is this still working?
When I try the example I get a chunk of hex in connector.raw_data
Thanks

Just tested, and it still seems to work.

i like it your post very interesting post

Thanks ! usefull