I was trying to use the app engine mail sending service on my site.
Currently, I've put something as simple as:
message = mail.EmailMessage(sender="Subhranath Chunder <subhranath#gmail.com>", \
subject="Your account has been approved")
message.to = "Subhranath Chunder <subhranathc#yahoo.com>"
message.body = "Just another message"
message.send()
But every time the code handler raises this error:
2011-08-20 04:44:36.988
ApplicationError: 1 Internal error
Traceback (most recent call last):
File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/webapp/_webapp25.py", line 703, in __call__
handler.post(*groups)
File "/base/data/home/apps/s~subhranath/0-2.352669327317975990/index.py", line 71, in post
message.send()
File "/base/python_runtime/python_lib/versions/1/google/appengine/api/mail.py", line 895, in send
raise e
ApplicationError: ApplicationError: 1 Internal error
My code as a different version, and not the default one. I'm not able to change this version as the default version, unless this error gets resolved. Any idea why this is happening?
The sender must either have privilegies or be the logged in user. So subhranath#gmail.com should have admin rights to your location or be the logged in user to run your code.
Most basic example would be just a HTTP get to send an email:
def get(self):
message = mail.EmailMessage(sender='subhranath_app_admin#gmail.com', subject='Email')
message.to='any...#gmail.com'
message.body = os.environ.get('HTTP_HOST')
message.send()
If you want to send an email as the logged in user you can make the sender a variable:
senderemail = users.get_current_user().email() if users.get_current_user() else 'subhranath#gmail.com'
I think the error message you posted was not very clear and if it persists I recommend to get more info about the error message using logging and exceptions.
Seems to be a app engine system glitch. The same code is now working, without any new deployment.
Related
I'm having trouble deploying a flask app to the Google app engine. It works fine when I run it locally but as soon as it's deployed to the app engine it fails. With two exceptions (I'm always getting both, I'm unsure which one comes first).
Traceback (most recent call last):
File "/opt/python3.7/lib/python3.7/logging/handlers.py", line 1008, in emit
smtp = smtplib.SMTP(self.mailhost, port, timeout=self.timeout)
File "/opt/python3.7/lib/python3.7/smtplib.py", line 251, in __init__
(code, msg) = self.connect(host, port)
File "/opt/python3.7/lib/python3.7/smtplib.py", line 338, in connect
(code, msg) = self.getreply()
File "/opt/python3.7/lib/python3.7/smtplib.py", line 391, in getreply
+ str(e))
smtplib.SMTPServerDisconnected: Connection unexpectedly closed: timed out
and
Traceback (most recent call last):
File "/opt/python3.7/lib/python3.7/smtplib.py", line 387, in getreply
line = self.file.readline(_MAXLINE + 1)
File "/opt/python3.7/lib/python3.7/socket.py", line 589, in readinto
return self._sock.recv_into(b)
socket.timeout: timed out
It consistently happens within the __init__ function of a form. And specifically in there when I try to append a validator to a validators list.
self.password.validators.append(InputRequired(message='Please enter password!'))
It is not specific to this one validator, the same thing happens with others too. I am at a complete loss as to why and how appending validators would interact with SMTP or sockets.
One thing I noticed is that I had an issue with the Email validator which also worked fine locally but required me to install the module 'email_validator' to work. When I did that it complained that it couldn't use append on a tuple when I tried to append a validator to - what was a list in my local app - but for some reason turned out to be a tuple when I ran the exact same code on the app engine. I removed the Email validator (and the import/requirements) entirely and now I'm at the problem I've described above. It may or may not be related to the Email validator issue.
Update: The error message does indeed seem to be related to the validators.append() call seeing as by setting up the validators at object creation and avoiding all further changes via validators.append() fixed the problem for me. This is the requirements.txt that goes with the application:
blinker>=1.4
Click>=7.0
Flask>=1.1.1
Flask-Login>=0.5.0
Flask-Mail>=0.9.1
Flask-SQLAlchemy>=2.4.1
Flask-WTF>=0.14.3
itsdangerous>=1.1.0
Jinja2>=2.11.1
MarkupSafe>=1.1.1
PyJWT>=1.7.1
PyMySQL>=0.9.3
python-dotenv>=0.11.0
SQLAlchemy>=1.3.13
Werkzeug>=1.0.0
WTForms>=2.2.1
Instead of a full minimal, reproducible example I can offer a minimized version of the form that was causing the issues when instanciated on the GAE:
class ExampleForm(FlaskForm):
name = StringField('Name')
original_name = HiddenField('Original Name')
def add_validators(self):
names = [a.name for a in Users.query.all()]
if self.original_name.data:
names.remove(self.original_name.data)
self.name.validators.append(InputRequired(message='Please enter name!'))
self.name.validators.append(NoneOf(names, message='Username already exists!'))
def __init__(self):
super().__init__(*args, **kwargs)
self.add_validators()
As I'm writing this it occurs to me that maybe I need to always initialize an empty validators list on creation to ensure new entries can be appended to it (like with a normal list) like this name = StringField('Name', validators=[]). I'd still be confused as to why it works locally, but if it fixes the problem that'd be what really counts for me.
I created a small program in Python for reading messages from a Pub/Sub subscription. I am using Python 3.7 and google-cloud-pubsub 1.1.0.
My code is very simple:
from google.cloud import pubsub_v1
from google.auth import jwt
import json
service_account_info = json.load(open("service-account-info.json"))
audience_sub = "https://pubsub.googleapis.com/google.pubsub.v1.Subscriber"
credentials_sub = jwt.Credentials.from_service_account_info(
service_account_info, audience=audience_sub
)
subscriber_ring = pubsub_v1.SubscriberClient(credentials=credentials_sub)
def callback1(message):
print("In callback!!")
print(message.data)
message.ack()
sub_path = "projects/my-project/subscriptions/my-sub"
future = subscriber_ring.subscribe(sub_path, callback=callback1)
future.result()
When the code reaches "future.result()", it hangs there forever and times out 10 minutes later with the error
pubsub 503 failed to connect to all addresses
I already verified that:
Pub/Sub is up and running
My service account has all the needed permissions. I even tried with my personal Google Cloud account (I am the project owner) with the same results.
There are unacked messages in the Topic
My network connection is OK
but I cannot make it work. Any ideas?
EDIT: I got some more info from the exception:
om_grpc_error(exc), exc)
File "<string>", line 3, in raise_from
google.api_core.exceptions.ServiceUnavailable: 503 failed to connect to all addresses
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/anaconda3/envs/loadtest/lib/python3.7/site-packages/google/cloud/pubsub_v1/publisher/_batch/thread.py", line 219, in _commit
response = self._client.api.publish(self._topic, self._messages)
File "/usr/local/anaconda3/envs/loadtest/lib/python3.7/site-packages/google/cloud/pubsub_v1/gapic/publisher_client.py", line 498, in publish
request, retry=retry, timeout=timeout, metadata=metadata
File "/usr/local/anaconda3/envs/loadtest/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py", line 143, in __call__
return wrapped_func(*args, **kwargs)
File "/usr/local/anaconda3/envs/loadtest/lib/python3.7/site-packages/google/api_core/retry.py", line 286, in retry_wrapped_func
on_error=on_error,
File "/usr/local/anaconda3/envs/loadtest/lib/python3.7/site-packages/google/api_core/retry.py", line 206, in retry_target
last_exc,
File "<string>", line 3, in raise_from
google.api_core.exceptions.RetryError: Deadline of 60.0s exceeded while calling functools.partial(<function _wrap_unary_errors.<locals>.error_remapped_callable at 0x7fa030891a70>,
, metadata=[('x-goog-request-params', 'topic=projects/my-project/subscriptions/my-sub'), ('x-goog-api-client', 'gl-python/3.7.6 grpc/1.26.0 gax/1.16.0 gapic/1.2.0')]), last exception: 503 failed to connect to all addresses
It is likely that there is a firewall rule in place or some network configuration that is disallowing/dropping connections to *.googleapis.com (or specifically pubsub.googleapis.com). You can see an example of this with another Google product.
When running the following minimal code to execute the GMail API watch() method. A 'Backend Error' occurs. This is true both when running on a client machine, and when run directly on Google Cloud as a Cloud Function.
from google.oauth2 import service_account
import googleapiclient.discovery
SCOPES = ["https://mail.google.com/"]
SERVICE_ACCOUNT_FILE = '<JSON KEY FILE REMOVED>'
TARGET='<GSUITE DOMAIN USER EMAIL ADDRESS>'
credentials = service_account.Credentials.from_service_account_file(SERVICE_ACCOUNT_FILE, scopes=SCOPES)
credentials_delegated = credentials.with_subject(TARGET)
service = googleapiclient.discovery.build('gmail', 'v1', credentials=credentials_delegated)
val = { 'topicName': '<TOPIC NAME COPIED FROM GCLOUD PUB/SUB>' }
watch_resp = service.users().watch(userId='me', body=val).execute()
print(watch_resp)
Other GMail APIs such as service.users().getProfile(userId='me').execute() work as expected. This error occurs after ensuring that the service account and the delegated user have the pub/sub publisher role and that the topic name is correct and already created. The exact output of the error is as follows:
Traceback (most recent call last):
File "test.py", line 12, in <module>
watch_resp = service.users().watch(userId='me', body=val).execute()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
return wrapped(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/googleapiclient/http.py", line 842, in execute
raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 500 when requesting https://www.googleapis.com/gmail/v1/users/me/watch?alt=json returned "Backend Error">
I have been going through the google cloud NDB Async examles tutorials
https://cloud.google.com/appengine/docs/standard/python/ndb/async
You can specify a whole WSGIApplication as ndb.toplevel. This makes
sure that each of the WSGIApplication's handlers waits for all async
requests before returning. (It does not "toplevel" all the
WSGIApplication's handlers.)
app = ndb.toplevel(webapp2.WSGIApplication([('/', MyRequestHandler)]))
Is this same functionality compatible with Flask? For example my code
app = Flask(__name__)
app.config.update(DEBUG = not SERVER_ISPRODUCTION)
app = ndb.toplevel(app)
...
#app.route('/test')
def testBackfill():
Gives me error
Traceback (most recent call last):
File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 240, in Handle
handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 299, in _LoadHandler
handler, path, err = LoadObject(self._handler)
File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 85, in LoadObject
obj = __import__(path[0])
#app.route('/test')
AttributeError: 'function' object has no attribute 'route'
This error goes away when I move the toplevel back directly to the request handler.
I feel either flask doesn't work with this functionality, or I'm doing something wrong in how I'm using toplevel.
My intention is for each of my request handlers within my application to wait for all of my async Google DataStore calls to finish before exiting (I'm using yield statements and tasklets within my request handlers).
The error is kind of expected: app is no longer the Flask app you created. I see 2 options to try:
rename the top level app (you need to match the change in the app.yaml config file as well):
app = Flask(__name__)
app.config.update(DEBUG = not SERVER_ISPRODUCTION)
my_app = ndb.toplevel(app) # change .app -> .my_app app.yaml
rename the flask app and all its references:
flask_app = Flask(__name__)
flask_app.config.update(DEBUG = not SERVER_ISPRODUCTION)
app = ndb.toplevel(flask_app)
...
#flask_app.route('/test')
def testBackfill(): **strong text**
Note: I'm not a Flask user, this may fix the missing route attribute error but I don't know if it will ultimately make the top level stuff work. But I can't write this as a comment.
I have built a pipeline on AppEngine that loads data from Cloud Storage to BigQuery. This works fine, ..until there is any error. How can I can loading exceptions by BigQuery from my AppEngine code?
The code in the pipeline looks like this:
#Run the job
credentials = AppAssertionCredentials(scope=SCOPE)
http = credentials.authorize(httplib2.Http())
bigquery_service = build("bigquery", "v2", http=http)
jobCollection = bigquery_service.jobs()
result = jobCollection.insert(projectId=PROJECT_ID,
body=build_job_data(table_name, cloud_storage_files))
#Get the status
while (not allDone and not runtime.is_shutting_down()):
try:
job = jobCollection.get(projectId=PROJECT_ID,
jobId=insertResponse).execute()
#Do something with job.get('status')
except:
exc_type, exc_value, exc_traceback = sys.exc_info()
logging.error(traceback.format_exception(exc_type, exc_value, exc_traceback))
time.sleep(30)
This gives me status error, or major connectivity errors, but what I am looking for is functional errors from BigQuery, like fields formats conversion errors, schema structure issues, or other issues BigQuery may have while trying to insert rows to tables.
If any "functional" error on BigQuery's side happens, this code will run successfully and complete normally, but no table will be written on BigQuery. Not easy to debug when this happens...
You can use the HTTP error code from the exception. BigQuery is a REST API, so the response codes that are returned match the description of HTTP error codes here.
Here is some code that handles retryable errors (connection, rate limit, etc), but re-raises when it is an error type that it doesn't expect.
except HttpError, err:
# If the error is a rate limit or connection error, wait and
# try again.
# 403: Forbidden: Both access denied and rate limits.
# 408: Timeout
# 500: Internal Service Error
# 503: Service Unavailable
if err.resp.status in [403, 408, 500, 503]:
print '%s: Retryable error %s, waiting' % (
self.thread_id, err.resp.status,)
time.sleep(5)
else: raise
If you want even better error handling, check out the BigqueryError class in the bq command line client (this used to be available on code.google.com, but with the recent switch to gCloud, it isn't any more. But if you have gcloud installed, the bq.py and bigquery_client.py files should be in the installation).
The key here is this part of the pasted code:
except:
exc_type, exc_value, exc_traceback = sys.exc_info()
logging.error(traceback.format_exception(exc_type, exc_value, exc_traceback))
time.sleep(30)
This "except" is catching every exception, logging it, and letting the process continue without any consideration for re-trying.
The question is, what would you like to do instead? At least the intention is there with the "#Do something" comment.
As a suggestion, consider App Engine's task queues to check the status, instead of a loop with a 30 second wait. When tasks get an exception, they are automatically retried - and you can tune that behavior.