I have been going through the google cloud NDB Async examles tutorials
https://cloud.google.com/appengine/docs/standard/python/ndb/async
You can specify a whole WSGIApplication as ndb.toplevel. This makes
sure that each of the WSGIApplication's handlers waits for all async
requests before returning. (It does not "toplevel" all the
WSGIApplication's handlers.)
app = ndb.toplevel(webapp2.WSGIApplication([('/', MyRequestHandler)]))
Is this same functionality compatible with Flask? For example my code
app = Flask(__name__)
app.config.update(DEBUG = not SERVER_ISPRODUCTION)
app = ndb.toplevel(app)
...
#app.route('/test')
def testBackfill():
Gives me error
Traceback (most recent call last):
File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 240, in Handle
handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 299, in _LoadHandler
handler, path, err = LoadObject(self._handler)
File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 85, in LoadObject
obj = __import__(path[0])
#app.route('/test')
AttributeError: 'function' object has no attribute 'route'
This error goes away when I move the toplevel back directly to the request handler.
I feel either flask doesn't work with this functionality, or I'm doing something wrong in how I'm using toplevel.
My intention is for each of my request handlers within my application to wait for all of my async Google DataStore calls to finish before exiting (I'm using yield statements and tasklets within my request handlers).
The error is kind of expected: app is no longer the Flask app you created. I see 2 options to try:
rename the top level app (you need to match the change in the app.yaml config file as well):
app = Flask(__name__)
app.config.update(DEBUG = not SERVER_ISPRODUCTION)
my_app = ndb.toplevel(app) # change .app -> .my_app app.yaml
rename the flask app and all its references:
flask_app = Flask(__name__)
flask_app.config.update(DEBUG = not SERVER_ISPRODUCTION)
app = ndb.toplevel(flask_app)
...
#flask_app.route('/test')
def testBackfill(): **strong text**
Note: I'm not a Flask user, this may fix the missing route attribute error but I don't know if it will ultimately make the top level stuff work. But I can't write this as a comment.
Related
I'm having trouble deploying a flask app to the Google app engine. It works fine when I run it locally but as soon as it's deployed to the app engine it fails. With two exceptions (I'm always getting both, I'm unsure which one comes first).
Traceback (most recent call last):
File "/opt/python3.7/lib/python3.7/logging/handlers.py", line 1008, in emit
smtp = smtplib.SMTP(self.mailhost, port, timeout=self.timeout)
File "/opt/python3.7/lib/python3.7/smtplib.py", line 251, in __init__
(code, msg) = self.connect(host, port)
File "/opt/python3.7/lib/python3.7/smtplib.py", line 338, in connect
(code, msg) = self.getreply()
File "/opt/python3.7/lib/python3.7/smtplib.py", line 391, in getreply
+ str(e))
smtplib.SMTPServerDisconnected: Connection unexpectedly closed: timed out
and
Traceback (most recent call last):
File "/opt/python3.7/lib/python3.7/smtplib.py", line 387, in getreply
line = self.file.readline(_MAXLINE + 1)
File "/opt/python3.7/lib/python3.7/socket.py", line 589, in readinto
return self._sock.recv_into(b)
socket.timeout: timed out
It consistently happens within the __init__ function of a form. And specifically in there when I try to append a validator to a validators list.
self.password.validators.append(InputRequired(message='Please enter password!'))
It is not specific to this one validator, the same thing happens with others too. I am at a complete loss as to why and how appending validators would interact with SMTP or sockets.
One thing I noticed is that I had an issue with the Email validator which also worked fine locally but required me to install the module 'email_validator' to work. When I did that it complained that it couldn't use append on a tuple when I tried to append a validator to - what was a list in my local app - but for some reason turned out to be a tuple when I ran the exact same code on the app engine. I removed the Email validator (and the import/requirements) entirely and now I'm at the problem I've described above. It may or may not be related to the Email validator issue.
Update: The error message does indeed seem to be related to the validators.append() call seeing as by setting up the validators at object creation and avoiding all further changes via validators.append() fixed the problem for me. This is the requirements.txt that goes with the application:
blinker>=1.4
Click>=7.0
Flask>=1.1.1
Flask-Login>=0.5.0
Flask-Mail>=0.9.1
Flask-SQLAlchemy>=2.4.1
Flask-WTF>=0.14.3
itsdangerous>=1.1.0
Jinja2>=2.11.1
MarkupSafe>=1.1.1
PyJWT>=1.7.1
PyMySQL>=0.9.3
python-dotenv>=0.11.0
SQLAlchemy>=1.3.13
Werkzeug>=1.0.0
WTForms>=2.2.1
Instead of a full minimal, reproducible example I can offer a minimized version of the form that was causing the issues when instanciated on the GAE:
class ExampleForm(FlaskForm):
name = StringField('Name')
original_name = HiddenField('Original Name')
def add_validators(self):
names = [a.name for a in Users.query.all()]
if self.original_name.data:
names.remove(self.original_name.data)
self.name.validators.append(InputRequired(message='Please enter name!'))
self.name.validators.append(NoneOf(names, message='Username already exists!'))
def __init__(self):
super().__init__(*args, **kwargs)
self.add_validators()
As I'm writing this it occurs to me that maybe I need to always initialize an empty validators list on creation to ensure new entries can be appended to it (like with a normal list) like this name = StringField('Name', validators=[]). I'd still be confused as to why it works locally, but if it fixes the problem that'd be what really counts for me.
When running the following minimal code to execute the GMail API watch() method. A 'Backend Error' occurs. This is true both when running on a client machine, and when run directly on Google Cloud as a Cloud Function.
from google.oauth2 import service_account
import googleapiclient.discovery
SCOPES = ["https://mail.google.com/"]
SERVICE_ACCOUNT_FILE = '<JSON KEY FILE REMOVED>'
TARGET='<GSUITE DOMAIN USER EMAIL ADDRESS>'
credentials = service_account.Credentials.from_service_account_file(SERVICE_ACCOUNT_FILE, scopes=SCOPES)
credentials_delegated = credentials.with_subject(TARGET)
service = googleapiclient.discovery.build('gmail', 'v1', credentials=credentials_delegated)
val = { 'topicName': '<TOPIC NAME COPIED FROM GCLOUD PUB/SUB>' }
watch_resp = service.users().watch(userId='me', body=val).execute()
print(watch_resp)
Other GMail APIs such as service.users().getProfile(userId='me').execute() work as expected. This error occurs after ensuring that the service account and the delegated user have the pub/sub publisher role and that the topic name is correct and already created. The exact output of the error is as follows:
Traceback (most recent call last):
File "test.py", line 12, in <module>
watch_resp = service.users().watch(userId='me', body=val).execute()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
return wrapped(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/googleapiclient/http.py", line 842, in execute
raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 500 when requesting https://www.googleapis.com/gmail/v1/users/me/watch?alt=json returned "Backend Error">
I am getting below error while accessing the api via explorer '_ah/api/explorer' or when api is called from the app.
Below is relevant piece from app.yaml and api implementation.
From other questions on stackoverflow it looks like this could happen if endpoint config is not deployed using gcloud service-management but event after uploading gatewayv1openapi.json I am getting similar error.
Is there something i need to change in api code or app.yaml? One more thing is that I have uploaded exact same codebase (only difference is other project id instead of ixp-test in app.yaml) to another old project and it works fine. Any newly created project is giving this error.
/_ah/api/gateway/v1/invoke?alt=json
no scheduler thread, scheduler.run() will be invoked by report(...) (/base/data/home/apps/s~ixp-test/20171013t215957.404774686885375478/server/lib/endpoints_management/control/client.py:235)
Traceback (most recent call last):
File "/base/data/home/apps/s~ixp-test/20171013t215957.404774686885375478/server/lib/endpoints_management/control/client.py", line 231, in start
self._thread.start()
File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/api/background_thread/background_thread.py", line 108, in start
start_new_background_thread(self.bootstrap, ())
File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/api/background_thread/background_thread.py", line 87, in start_new_background_thread
raise ERROR_MAPerror.application_error
FrontendsNotSupported
.
.
.
(/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/runtime/wsgi.py:263)
Traceback (most recent call last):
File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 240, in Handle
handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 299, in _LoadHandler
handler, path, err = LoadObject(self._handler)
File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 96, in LoadObject
__import(cumulative_path)
File "/base/data/home/apps/s~ixp-test/20171013t215957.404774686885375478/server/main.py", line 18, in
api = endpoints.api_server([GatewayApi])
File "/base/data/home/apps/s~ixp-test/20171013t215957.404774686885375478/server/lib/endpoints/apiserving.py", line 514, in api_server
controller)
File "/base/data/home/apps/s~ixp-test/20171013t215957.404774686885375478/server/lib/endpoints_management/control/wsgi.py", line 126, in add_all
a_service = loader.load()
File "/base/data/home/apps/s~ixp-test/20171013t215957.404774686885375478/server/lib/endpoints_management/control/service.py", line 111, in load
return self._load_func(**kw)
File "/base/data/home/apps/s~ixp-test/20171013t215957.404774686885375478/server/lib/endpoints_management/config/service_config.py", line 79, in fetch_service_config
_log_and_raise(Exception, message_template.format(status_code))
File "/base/data/home/apps/s~ixp-test/20171013t215957.404774686885375478/server/lib/endpoints_management/config/service_config.py", line 127, in _log_and_raise
raise exception_class(message)
Exception: Fetching service config failed (status code 403)
Below is the relevant config
app.yaml
runtime: python27
api_version: 1
threadsafe: false
automatic_scaling:
max_idle_instances: 0
env_variables:
ENDPOINTS_SERVICE_NAME: ixp-test.appspot.com
ENDPOINTS_SERVICE_VERSION: 2017-08-13r0
API
#endpoints.api(name='gateway', version='v1')
class GatewayApi(remote.Service):
#endpoints.method(
GatewayRequest,
GatewayResponse,
path='invoke',
http_method='POST',
name='invoke')
def invoke(self, request):
pass
#endpoints.method(
GatewayRequest,
GatewayResponse,
path='invokeSecure',
http_method='POST',
name='invokeSecure',
scopes=[endpoints.EMAIL_SCOPE])
def invoke_secure(self, request):
pass
This might well happen if the service name in the gatewayv1openapi.json doesn't match the service name in the app.yaml; can you double-check that?
Managed to resolve it. Issue was due to not updating
ENDPOINTS_SERVICE_VERSION: 2017-08-13r0
for new project. Value from old deployed project for which it was working caused the error.
from getting started tutorial
gcloud service-management deploy echov1openapi.json
Service Configuration [2017-02-13-r2] uploaded for service [example-project.appspot.com]
I am trying to use python-twitter api in GAE.
I need to import Oauth2 and httplib2.
Here is how I did
For OAuth2, I downloaded github.com/simplegeo/python-oauth2/tree/master/oauth2. For HTTPLib2, I dowloaded code.google.com/p/httplib2/wiki/Install and extracted folder python2/httplib2 to project root folder.
my views.py
import twitter
def index(request):
api = twitter.Api(consumer_key='XNAUYmsmono4gs3LP4T6Pw',consumer_secret='xxxxx',access_token_key='xxxxx',access_token_secret='iHzMkC6RRDipon1kYQtE5QOAYa1bVfYMhH7GFmMFjg',cache=None)
return render_to_response('fbtwitter/index.html')
I got the error paste.shehas.net/show/jbXyx2MSJrpjt7LR2Ksc
AttributeError
AttributeError: 'module' object has no attribute 'SignatureMethod_PLAINTEXT'
Traceback (most recent call last)
File "D:\PythonProj\fbtwitter\kay\lib\werkzeug\wsgi.py", line 471, in __call__
return app(environ, start_response)
File "D:\PythonProj\fbtwitter\kay\app.py", line 478, in __call__
response = self.get_response(request)
File "D:\PythonProj\fbtwitter\kay\app.py", line 405, in get_response
return self.handle_uncaught_exception(request, exc_info)
File "D:\PythonProj\fbtwitter\kay\app.py", line 371, in get_response
response = view_func(request, **values)
File "D:\PythonProj\fbtwitter\fbtwitter\views.py", line 39, in index
access_token_secret='iHzMkC6RRDipon1kYQtE5QOAYa1bVfYMhH7GFmMFjg',cache=None)
File "D:\PythonProj\fbtwitter\fbtwitter\twitter.py", line 2235, in __init__
self.SetCredentials(consumer_key, consumer_secret, access_token_key, access_token_secret)
File "D:\PythonProj\fbtwitter\fbtwitter\twitter.py", line 2264, in SetCredentials
self._signature_method_plaintext = oauth.SignatureMethod_PLAINTEXT()
AttributeError: 'module' object has no attribute 'SignatureMethod_PLAINTEXT'
It seems I did not import Oauth2 correctly when I tracked the error in twitter.py
self._signature_method_plaintext = oauth.SignatureMethod_PLAINTEXT()
I even go to twitter.py and add import oauth2 as oauth but it couldnt solve the problem
Can anybody help?
I fixed it. In twitter.py,
try:
from hashlib import md5
except ImportError:
from md5 import md5
import oauth
CHARACTER_LIMIT = 140
# A singleton representing a lazily instantiated FileCache.
DEFAULT_CACHE = object()
REQUEST_TOKEN_URL = 'https://api.twitter.com/oauth/request_token'
ACCESS_TOKEN_URL = 'https://api.twitter.com/oauth/access_token'
AUTHORIZATION_URL = 'https://api.twitter.com/oauth/authorize'
SIGNIN_URL = 'https://api.twitter.com/oauth/authenticate'
Need to change import oauth to import oauth2 as oauth
I'm using tweetpy with my GAE application and it works well.
https://github.com/tweepy/tweepy
You can find some sample codes of tweetpy on GAE in google search.
I was trying to use the app engine mail sending service on my site.
Currently, I've put something as simple as:
message = mail.EmailMessage(sender="Subhranath Chunder <subhranath#gmail.com>", \
subject="Your account has been approved")
message.to = "Subhranath Chunder <subhranathc#yahoo.com>"
message.body = "Just another message"
message.send()
But every time the code handler raises this error:
2011-08-20 04:44:36.988
ApplicationError: 1 Internal error
Traceback (most recent call last):
File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/webapp/_webapp25.py", line 703, in __call__
handler.post(*groups)
File "/base/data/home/apps/s~subhranath/0-2.352669327317975990/index.py", line 71, in post
message.send()
File "/base/python_runtime/python_lib/versions/1/google/appengine/api/mail.py", line 895, in send
raise e
ApplicationError: ApplicationError: 1 Internal error
My code as a different version, and not the default one. I'm not able to change this version as the default version, unless this error gets resolved. Any idea why this is happening?
The sender must either have privilegies or be the logged in user. So subhranath#gmail.com should have admin rights to your location or be the logged in user to run your code.
Most basic example would be just a HTTP get to send an email:
def get(self):
message = mail.EmailMessage(sender='subhranath_app_admin#gmail.com', subject='Email')
message.to='any...#gmail.com'
message.body = os.environ.get('HTTP_HOST')
message.send()
If you want to send an email as the logged in user you can make the sender a variable:
senderemail = users.get_current_user().email() if users.get_current_user() else 'subhranath#gmail.com'
I think the error message you posted was not very clear and if it persists I recommend to get more info about the error message using logging and exceptions.
Seems to be a app engine system glitch. The same code is now working, without any new deployment.