Getting the following error on Python AppEngine:
HTTPException: Deadline exceeded while waiting for HTTP response from URL: https://www.googleapis.com/books/v1/users/115429583296661000087/bookshelves/1001/volumes?maxResults=12&startIndex=0
URL is .json format, and im grabbing it via the following code on my application:
request = urllib2.Request(bookShelfUrl, None, {'Content-Type': 'application/json'})
bookShelfJsonRaw = urllib2.urlopen(request)
bookShelfJsonObject = json.load(bookShelfJsonRaw)
works fine when testing in localhost, only gives an error in production. it also worked fine in production up until today when it mysteriously started returning that error.
any thoughts?
I did not know was the issue solved by google or it start working right after i modify the code and specify timout parameter instead of deafults for http instantiation:
httplib2.Http(timeout=15)
The Google Books gData API was timing out because I wasn't passing in the country code. For whatever reason it worked fine without the country on a local test server but whenever I pushed it to production it would timeout. After adding country=US the json feed worked fine.
bookShelfUrl = 'https://www.googleapis.com/books/v1/users/' + \
bookShelfId + \
'/bookshelves/1001/volumes?' + \
'maxResults=' + str(maxResults) + \
'&startIndex=' + str(startIndex) + \
'&country=US'
Related
I'm experiencing an issue with urlfetch. Seems that for the URL used in the code, follow_redirects=True works only locally.
Here's the code I used for testing:
def redirect_test():
url = 'http://0214.by/job.php?busy=ПОСТОЯННАЯ'
response = urlfetch.fetch(url, follow_redirects=True)
if 'location' in response.headers:
ret = 'redirect ignored. location: %s<br>' % response.headers['location']
url = urlparse.urljoin(url, response.headers['location'])
response = urlfetch.fetch(url, follow_redirects=False)
if 'location' in response.headers:
ret += 'redirect followed manually and another redirect response received'
else:
ret += 'redirect followed manually and no more redirects received'
else:
ret = 'redirect followed automatically'
return ret
The problem is that when I'm running this code locally I'm always getting the "Redirect followed automatically" message. When deployed on google app engine, the "Redirect followed manually and no more redirects received" message is always shown.
The URL returns a redirect to https page.
It makes me think the redirect is followed as expected only when running locally for some reason. I would appreciate any ideas on what this could be caused by.
I've just followed a tutorial for creating a simple RESTful api using the MEAN stack from the heroku webpage. What I did was just cloning the repo that contains the sample code, added my mongodb_uri from mLab and then run the app locally (npm start).
It works perfectly, but when I try to make an invalid entry (not providing name and last name to a contact) the express server crashes and the entry is made on my database (which is inconsistent).
I've opened an issue on the github repo but I got no answers, I think that there must be something wrong with the error handling but I don't know what it might be.
Here it is what I get when the server crashes:
ERROR: Invalid user input
/Users/nanop/Desktop/mean-contactlist/node_modules/mongodb/lib/utils.js:98
process.nextTick(function() { throw err; });
^
Error: Can't set headers after they are sent.
at ServerResponse.OutgoingMessage.setHeader (_http_outgoing.js:344:11)
at ServerResponse.header (/Users/nanop/Desktop/mean-contactlist/node_modules/express/lib/response.js:719:10)
at ServerResponse.send (/Users/nanop/Desktop/mean-contactlist/node_modules/express/lib/response.js:164:12)
at ServerResponse.json (/Users/nanop/Desktop/mean-contactlist/node_modules/express/lib/response.js:250:15)
at /Users/nanop/Desktop/mean-contactlist/server.js:72:23
at /Users/nanop/Desktop/mean-contactlist/node_modules/mongodb/lib/collection.js:421:18
at handleCallback (/Users/nanop/Desktop/mean-contactlist/node_modules/mongodb/lib/utils.js:96:12)
at /Users/nanop/Desktop/mean-contactlist/node_modules/mongodb/lib/collection.js:726:5
at /Users/nanop/Desktop/mean-contactlist/node_modules/mongodb-core/lib/connection/pool.js:428:18
at nextTickCallbackWith0Args (node.js:433:9)
And finally this is the handleError method defined which I think it's ok:
function handleError(res, reason, message, code) {
console.log("ERROR: " + reason);
res.status(code || 500).json({"error": message});
}
This is the repo I refer to: https://github.com/chrisckchang/mean-contactlist
The code I am using is:
var frisby = require('frisby');
frisby.create('Get paf data')
.get('https://services.....')
.expectStatus(200)
.expectHeaderContains('content-type', 'application/json')
.toss();
The error I get while running it from CLI using
jasmine-node EndpointTest_spec.js
The error I get is:
Error: Expected 500 to equal 200
Error: Header 'content-type' not present in HTTP response
So do I need to first load my website and call services through my application and then run frisby ?
But then it defeats the purpose of just making a quick check for all the endpoints used in application without running it.
You are calling request with https:// which may be secure server so use
{ strictSSL: false} in your get method .get('https://services.....',{ strictSSL: false}). It will solve your problem.
var frisby = require('frisby');
frisby.create('Get paf data')
.get('https://services.....',{ strictSSL: false})
.expectStatus(200)
.expectHeaderContains('content-type', 'application/json')
.toss();
I have two AppEngine modules, a default module running Python and "java" module running Java. I'm accessing the Java module from the default module using urlfetch. According to the AppEngine docs (cloud.google.com/appengine/docs/java/appidentity), I can verify in the Java module that the request originates from a module in the same app by checking the X-Appengine-Inbound-Appid header.
However, this header is not being set (in a production deployment). I use urlfetch in the Python module as follows:
hostname = modules.get_hostname(module="java")
hostname = hostname.replace('.', '-dot-', 2)
url = "http://%s/%s" % (hostname, "_ah/api/...")
result = urlfetch.fetch(url=url, follow_redirects=False, method=urlfetch.GET)
Note that I'm using the notation:
<version>-dot-<module>-dot-<app>.appspot.com
rather than the notation:
<version>.<module>.<app>.appspot.com
which for some reason results in a 404 response.
In the Java module I'm running a servlet filter which looks at all the request headers as follows:
Enumeration<String> headerNames = httpRequest.getHeaderNames();
while (headerNames.hasMoreElements()) {
String headerName = headerNames.nextElement();
String headerValue = httpRequest.getHeader(headerName);
mLog.info("Header: " + headerName + " = " + headerValue);
}
AppEngine does set some headers, e.g. X-AppEngine-Country. But the X-Appengine-Inbound-Appid header is not set.
Why am I'm not seeing the documented behaviour? Any suggestions would be much appreciated.
Have a look at what I've been answered on Google groups, which led to an issue opened on the public issue tracker.
As suggested in the answer I received you can follow, for any update, the issue over there.
Using phantomjs page.evaluate to extract "resultStats" (div id) from http://www.google.com/search/?q=site:%s works on my local server but not on production server.
NOTE: I'm using the latest phantomjs 1.9.7, however I experienced the same issue with the previous version 1.9.6
NOTE: Phantomjs page.render (on Google home page as well as any other domain name) is working on both servers and creates nice screenshots.
On my production server (Debian stable 7.3 #linode.com) the PHP code below for a top level domain name as the "$url" returns:
TypeError: 'null' is not an object (evaluating 'document.getElementById('resultStats').textContent') phantomjs://webpage.evaluate():2 phantomjs://webpage.evaluate():3 phantomjs://webpage.evaluate():3 null
On my local server (debian testing) the PHP code below for the same "$url" returns:
About 43 results
This happens with any domain name/url I use as the argument - I've tested it on dozens.
What might cause this to occur in my remote production server and not my local server?
gsiteindex.js
var page = require('webpage').create(), site;
var site = phantom.args[0];
page.open("https://www.google.com/search?q=site:" + site, function (status) {
var result = page.evaluate(function () {
return document.getElementById('resultStats').textContent;
});
console.info(result);
phantom.exit();
});
.php
$phantomjs = "phantomjs";
$script = "gsiteindex.js";
$site = $url;
$command = "$phantomjs $script $site";
$googlestring = shell_exec($command);
echo $googlestring;
die();
Well, nrabinowitz was right. I tested it more on my own server using proxies, most timed out, some returned the above error, and a couple returned correct results (well I assume they were correct based on the location the IP address of the proxy - because the figures were a little different than using my ISPs public IP address (calif., USA)).
So it's simply a matter of google blocking certain types of requests from certain IP addresses.
Thanks again for the comment.
Incleude header with user-agent e.g.
header = {'user-asgent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64;
rv:68.0) Gecko/20100101 Firefox/68.0'}
Withuot user agent you get googles gefault style page without resultStats a also had this issue and adding header helped
Default google search page looks like this
enter image description here