I have two AppEngine modules, a default module running Python and "java" module running Java. I'm accessing the Java module from the default module using urlfetch. According to the AppEngine docs (cloud.google.com/appengine/docs/java/appidentity), I can verify in the Java module that the request originates from a module in the same app by checking the X-Appengine-Inbound-Appid header.
However, this header is not being set (in a production deployment). I use urlfetch in the Python module as follows:
hostname = modules.get_hostname(module="java")
hostname = hostname.replace('.', '-dot-', 2)
url = "http://%s/%s" % (hostname, "_ah/api/...")
result = urlfetch.fetch(url=url, follow_redirects=False, method=urlfetch.GET)
Note that I'm using the notation:
<version>-dot-<module>-dot-<app>.appspot.com
rather than the notation:
<version>.<module>.<app>.appspot.com
which for some reason results in a 404 response.
In the Java module I'm running a servlet filter which looks at all the request headers as follows:
Enumeration<String> headerNames = httpRequest.getHeaderNames();
while (headerNames.hasMoreElements()) {
String headerName = headerNames.nextElement();
String headerValue = httpRequest.getHeader(headerName);
mLog.info("Header: " + headerName + " = " + headerValue);
}
AppEngine does set some headers, e.g. X-AppEngine-Country. But the X-Appengine-Inbound-Appid header is not set.
Why am I'm not seeing the documented behaviour? Any suggestions would be much appreciated.
Have a look at what I've been answered on Google groups, which led to an issue opened on the public issue tracker.
As suggested in the answer I received you can follow, for any update, the issue over there.
Related
I am trying to run a simple python 2 server code with AppEngine and Datastore. When I run dev_appserver.py app.yaml, the program immediately exits (without an error) after the following outputs:
/home/username/google-cloud-sdk/lib/third_party/google/auth/crypt/_cryptography_rsa.py:22: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in the next release.
import cryptography.exceptions
INFO 2022-12-20 11:59:41,931 devappserver2.py:239] Using Cloud Datastore Emulator.
We are gradually rolling out the emulator as the default datastore implementation of dev_appserver.
If broken, you can temporarily disable it by --support_datastore_emulator=False
Read the documentation: https://cloud.google.com/appengine/docs/standard/python/tools/migrate-cloud-datastore-emulator
INFO 2022-12-20 11:59:41,936 devappserver2.py:316] Skipping SDK update check.
INFO 2022-12-20 11:59:42,332 datastore_emulator.py:156] Starting Cloud Datastore emulator at: http://localhost:22325
INFO 2022-12-20 11:59:42,981 datastore_emulator.py:162] Cloud Datastore emulator responded after 0.648865 seconds
INFO 2022-12-20 11:59:42,982 <string>:384] Starting API server at: http://localhost:38915
Ideally, it should have continued by runnning the server on port 8000. Also, it works with option --support_datastore_emulator=False.
This is the code:
import webapp2
import datetime
from google.appengine.ext import db, deferred, ndb
import uuid
from base64 import b64decode, b64encode
import logging
class Email(ndb.Model):
email = ndb.StringProperty()
class DB(webapp2.RequestHandler):
def post(self):
try:
mail = Email()
mail.email = 'Test'
mail.put()
except Exception as e:
print(e)
self.response.headers['Content-Type'] = 'application/json; charset=utf-8'
return self.response.out.write(e)
def get(self):
try:
e1 = Email.query()
logging.critical('count is: %s' % e1.count)
e1k = e1.get(keys_only=True)
logging.critical('count 2 is: %s' % e1k.count)
e1 = e1.get()
key = unicode(e1.key.urlsafe())
logging.critical('This is a critical message: %s' % key)
logging.critical('This is a critical message: %s' % e1k)
e2 = ndb.Key(urlsafe=key).get()
self.response.headers['Content-Type'] = 'application/json; charset=utf-8'
return self.response.out.write(str(e2.email))
except Exception as e:
print(e)
self.response.headers['Content-Type'] = 'application/json; charset=utf-8'
return self.response.out.write(e)
app = webapp2.WSGIApplication([
('/', DB)
], debug=True)
How can I find the reason this is not working?
Edit: I figured that the dev server works and writes to a datastore even with support_datastore_emulator=False option. I am confused by this option. I also don't know where the database is stored currently.
It should be count() and not count i.e.
logging.critical('count is: %s' % e1.count())
A get returns only 1 record and so it doesn't make sense to do a count after calling a get. Besides, the count operation is a method of the query instance not the results. This means the following code is incorrect
e1k = e1.get(keys_only=True)
logging.critical('count 2 is: %s' % e1k.count)
You should replace it with
elk = e1.fetch(keys_only=True) # fetch gives an array
logging.critical('count 2 is: %s' % len(e1k))
When you first run your App, it will execute the GET part of your code and because this is the first time your App is being run, you have no record in Datastore. This means e1 = e1.get() will return None and key = unicode(e1.key.urlsafe()) will lead to an error.
You have to modify your code to first check you have a value for e1 or e2 before you attempt to use the keys.
I ran your code with dev_appserver.py and it displayed these errors for me in the logs. But I ran it with an older version of gcloud SDK (Google Cloud SDK 367.0.0). I don't know why yours exited without displaying any errors. Maybe it's due to the version...??
Separately - Don't know why you're importing db. Google moved on to ndb long ago and you don't use db in your code
The default datastore (for the older generation runtimes like Python 2) is in .config (hidden folder) > gcloud > emulators > datastore
You can also specify your own location by using the flag --datastore_path. See documentation
I created the stub classes using CXF wsdl2java tool.
I am using Apache CXF library, with JCIFS. I validated the WSDL file itself through couple tools, it is good. Here is the code. It looks like some setting I must do.
//JCIFS Authentication related code
jcifs.Config.setProperty("jcifs.smb.client.domain", "NTS");
jcifs.Config.setProperty("jcifs.netbios.wins", "ecmchat.mark.gov");
jcifs.Config.setProperty("jcifs.smb.client.soTimeout", "300000"); // 5 minutes
jcifs.Config.setProperty("jcifs.netbios.cachePolicy", "1200"); // 20 minutes
jcifs.Config.setProperty("jcifs.smb.client.username", "user");
jcifs.Config.setProperty("jcifs.smb.client.password", "password");
//Register the jcifs URL handler to enable NTLM
jcifs.Config.registerSmbURLHandler();
//WSDL and Client settings
URL wsdlURL = BF.WSDL_LOCATION;
if (args.length > 0 && args[0] != null && !"".equals(args[0])) {
File wsdlFile = new File(args[0]);
try {
if (wsdlFile.exists()) {
wsdlURL = wsdlFile.toURI().toURL();
} else {
wsdlURL = new URL(args[0]);
}
} catch (MalformedURLException e) {
e.printStackTrace();
}
}
BF ss = new BF(wsdlURL, SERVICE_NAME);
BFSoap port = ss.getBFSoap12();
Client client = ClientProxy.getClient(port);
HTTPConduit http = (HTTPConduit) client.getConduit();
HTTPClientPolicy httpClientPolicy = new HTTPClientPolicy();
httpClientPolicy.setConnectionTimeout(36000);
httpClientPolicy.setAllowChunking(false);
httpClientPolicy.setReceiveTimeout(32000);
http.setClient(httpClientPolicy);
// Calling the method
System.out.println("Invoking testMethod...");
String _testMethod__return = port.testMethod();
System.out.println("testMethod.result=" + _testMethod__return);
I am getting the following exception
Caused by: com.ctc.wstx.exc.WstxParsingException: Unexpected close tag </span>; expected </br>.
at [row,col,system-id]: [59,22,"https://ecmchat.mark.gov/BF/BF.asmx"]
at com.ctc.wstx.sr.StreamScanner.constructWfcException(StreamScanner.java:621)
at com.ctc.wstx.sr.StreamScanner.throwParseError(StreamScanner.java:491)
at com.ctc.wstx.sr.StreamScanner.throwParseError(StreamScanner.java:475)
at com.ctc.wstx.sr.BasicStreamReader.reportWrongEndElem(BasicStreamReader.java:3365)
at com.ctc.wstx.sr.BasicStreamReader.readEndElem(BasicStreamReader.java:3292)
at com.ctc.wstx.sr.BasicStreamReader.nextFromTree(BasicStreamReader.java:2911)
at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1123)
at org.apache.cxf.staxutils.StaxUtils.readDocElements(StaxUtils.java:1361)
at org.apache.cxf.staxutils.StaxUtils.readDocElements(StaxUtils.java:1255)
at org.apache.cxf.staxutils.StaxUtils.read(StaxUtils.java:1183)
at org.apache.cxf.wsdl11.WSDLManagerImpl.loadDefinition(WSDLManagerImpl.java:235)
... 9 more
If I comment out the JCIFS NTLM authentication code, I get a HTTP 401 error. Therefore, I believe, at least it is passing some kind of authorization step.
And, if I use local WSDL in place of remote URL WSDL, then I get a different error like "method not implemented" on the call to the method. May be this is due to me not using the local WSDL correctly. I do not even know if we can use the local WSDL reference for remote service.
Then, I created a SoapUI dummy service with this WSDL, and the same code (but without the JCIFS authentication code) works good, and successfully calls the methods.
It appears to me that I must add some more appropriate settings in the configuration related code.
Am I right, and are you aware of any, for NTLM authentication and Apache CXF?
But parsing error is confusing???
I do not know if this is related.
My original WSDL URL that I gave was this.
https://ecmchat.mark.gov/BF/BF.asmx
I added a ?wsdl like below
https://ecmchat.mark.gov/BF/BF.asmx?wsdl
Then I am getting a different error.
I wonder why it is working if I access my local SoapUI version of the same WSDL service, but not for the remote one.
Invoking testMethod...
Jan 07, 2020 10:47:25 AM org.apache.cxf.phase.PhaseInterceptorChain doDefaultLogging
WARNING: Interceptor for {https://ecmchat.mark.gov}BF#{https://ecmchat.mark.gov}testMethod has thrown exception, unwinding now
java.lang.UnsupportedOperationException: Method not implemented.
at java.net.URLStreamHandler.openConnection(URLStreamHandler.java:96)
at java.net.URL.openConnection(URL.java:1028)
at org.apache.cxf.transport.https.HttpsURLConnectionFactory.createConnection(HttpsURLConnectionFactory.java:92)
at org.apache.cxf.transport.http.URLConnectionHTTPConduit.createConnection(URLConnectionHTTPConduit.java:121)
at org.apache.cxf.transport.http.URLConnectionHTTPConduit.setupConnection(URLConnectionHTTPConduit.java:125)
at org.apache.cxf.transport.http.HTTPConduit.prepare(HTTPConduit.java:505)
at org.apache.cxf.interceptor.MessageSenderInterceptor.handleMessage(MessageSenderInterceptor.java:47)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
at org.apache.cxf.endpoint.ClientImpl.doInvoke(ClientImpl.java:530)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:441)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:356)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:314)
at org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:96)
at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:140)
at com.sun.proxy.$Proxy33.testMethod(Unknown Source)
at edison.learn.BFSoap_BFSoap12_Client.main(BFSoap_BFSoap12_Client.java:90)
Exception in thread "main" javax.xml.ws.soap.SOAPFaultException: Method not implemented.
at org.apache.cxf.jaxws.JaxWsClientProxy.mapException(JaxWsClientProxy.java:195)
at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:145)
at com.sun.proxy.$Proxy33.testMethod(Unknown Source)
at edison.learn.BFSoap_BFSoap12_Client.main(BFSoap_BFSoap12_Client.java:90)
Caused by: java.lang.UnsupportedOperationException: Method not implemented.
at java.net.URLStreamHandler.openConnection(URLStreamHandler.java:96)
at java.net.URL.openConnection(URL.java:1028)
at org.apache.cxf.transport.https.HttpsURLConnectionFactory.createConnection(HttpsURLConnectionFactory.java:92)
at org.apache.cxf.transport.http.URLConnectionHTTPConduit.createConnection(URLConnectionHTTPConduit.java:121)
at org.apache.cxf.transport.http.URLConnectionHTTPConduit.setupConnection(URLConnectionHTTPConduit.java:125)
at org.apache.cxf.transport.http.HTTPConduit.prepare(HTTPConduit.java:505)
at org.apache.cxf.interceptor.MessageSenderInterceptor.handleMessage(MessageSenderInterceptor.java:47)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
at org.apache.cxf.endpoint.ClientImpl.doInvoke(ClientImpl.java:530)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:441)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:356)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:314)
at org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:96)
at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:140)
... 2 more
An error has occurred: Validation failed for [userAgent] with value
[]: The property userAgent is required and cannot be NULL, the empty
string, or the default [userAgent]
How can I resolve this exception?
Code example:
require_once 'Google/Api/Ads/AdWords/Lib/AdWordsUser.php';
$user = new AdWordsUser();
$user->LogDefaults();
$targetingIdeaService = $user->GetService('TargetingIdeaService', 'v201406');
Google Adwords SDK version 201406 requires you to set userAgent to a non-empty string by which you can identify your API Request and Google Team can identify where from the Request comes if any problem arises. Put any valid name to userAgent in the auth.ini file.
Getting the following error on Python AppEngine:
HTTPException: Deadline exceeded while waiting for HTTP response from URL: https://www.googleapis.com/books/v1/users/115429583296661000087/bookshelves/1001/volumes?maxResults=12&startIndex=0
URL is .json format, and im grabbing it via the following code on my application:
request = urllib2.Request(bookShelfUrl, None, {'Content-Type': 'application/json'})
bookShelfJsonRaw = urllib2.urlopen(request)
bookShelfJsonObject = json.load(bookShelfJsonRaw)
works fine when testing in localhost, only gives an error in production. it also worked fine in production up until today when it mysteriously started returning that error.
any thoughts?
I did not know was the issue solved by google or it start working right after i modify the code and specify timout parameter instead of deafults for http instantiation:
httplib2.Http(timeout=15)
The Google Books gData API was timing out because I wasn't passing in the country code. For whatever reason it worked fine without the country on a local test server but whenever I pushed it to production it would timeout. After adding country=US the json feed worked fine.
bookShelfUrl = 'https://www.googleapis.com/books/v1/users/' + \
bookShelfId + \
'/bookshelves/1001/volumes?' + \
'maxResults=' + str(maxResults) + \
'&startIndex=' + str(startIndex) + \
'&country=US'
Can error_handler be set for a blueprint?
#blueprint.errorhandler(404)
def page_not_found(error):
return 'This page does not exist', 404
edit:
https://github.com/mitsuhiko/flask/blob/18413ed1bf08261acf6d40f8ba65a98ae586bb29/flask/blueprints.py
you can specify an app wide and a blueprint local error_handler
You can use Blueprint.app_errorhandler method like this:
bp = Blueprint('errors', __name__)
#bp.app_errorhandler(404)
def handle_404(err):
return render_template('404.html'), 404
#bp.app_errorhandler(500)
def handle_500(err):
return render_template('500.html'), 500
errorhandler is a method inherited from Flask, not Blueprint.
If you are using Blueprint, the equivalent is app_errorhandler.
The documentation suggests the following approach:
def app_errorhandler(self, code):
"""Like :meth:`Flask.errorhandler` but for a blueprint. This
handler is used for all requests, even if outside of the blueprint.
"""
Therefore, this should work:
from flask import Blueprint, render_template
USER = Blueprint('user', __name__)
#USER.app_errorhandler(404)
def page_not_found(e):
""" Return error 404 """
return render_template('404.html'), 404
On the other hand, while the approach below did not raise any error for me, it didn't work:
from flask import Blueprint, render_template
USER = Blueprint('user', __name__)
#USER.errorhandler(404)
def page_not_found(e):
""" Return error 404 """
return render_template('404.html'), 404
add error handling at application level using the request proxy object:
from flask import request,jsonify
#app.errorhandler(404)
#app.errorhandler(405)
def _handle_api_error(ex):
if request.path.startswith('/api/'):
return jsonify(ex)
else:
return ex
flask Documentation
I too couldn't get the top rated answer to work, but here's a workaround.
You can use a catch-all at the end of your Blueprint, not sure how robust/recommended it is, but it does work. You could also add different error messages for different methods too.
#blueprint.route('/<path:path>')
def page_not_found(path):
return "Custom failure message"
Surprised others didn't mention miguelgrinberg's excellent tutorial.
https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-vii-error-handling
I found the sentry framework for error handling (links below). Seems overly complex. not sure of the threshold where it becomes useful.
https://flask.palletsprojects.com/en/1.1.x/errorhandling/
https://docs.sentry.io/platforms/python/guides/flask/
I combined previous excellent answers with the official docs from Flask, section 'Returning API Errors as JSON', in order to provide a more general approach.
Here is a working PoC that you can copy and paste on your registered blueprint API route handler (e.g. app/api/routes.py):
#blueprint.app_errorhandler(HTTPException)
def handle_exception(e):
"""Return JSON instead of HTML for HTTP errors."""
# start with the correct headers and status code from the error
response = e.get_response()
# replace the body with JSON
response.data = json.dumps({
"code": e.code,
"name": e.name,
"description": e.description,
})
response.content_type = "application/json"
return response
Flask doesnt support blueprint level error handlers for 404 and 500 errors. A BluePrint is a leaky abstraction. Its better to use a new WSGI App for this, if you need separate error handlers, this makes more sense.
Also i would recommend not to use flask, it uses globals all over the places, which makes your code difficult to manage if it grows bigger.