Why does dev_appserver.py exit without throwing errors? - google-app-engine

I am trying to run a simple python 2 server code with AppEngine and Datastore. When I run dev_appserver.py app.yaml, the program immediately exits (without an error) after the following outputs:
/home/username/google-cloud-sdk/lib/third_party/google/auth/crypt/_cryptography_rsa.py:22: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in the next release.
import cryptography.exceptions
INFO 2022-12-20 11:59:41,931 devappserver2.py:239] Using Cloud Datastore Emulator.
We are gradually rolling out the emulator as the default datastore implementation of dev_appserver.
If broken, you can temporarily disable it by --support_datastore_emulator=False
Read the documentation: https://cloud.google.com/appengine/docs/standard/python/tools/migrate-cloud-datastore-emulator
INFO 2022-12-20 11:59:41,936 devappserver2.py:316] Skipping SDK update check.
INFO 2022-12-20 11:59:42,332 datastore_emulator.py:156] Starting Cloud Datastore emulator at: http://localhost:22325
INFO 2022-12-20 11:59:42,981 datastore_emulator.py:162] Cloud Datastore emulator responded after 0.648865 seconds
INFO 2022-12-20 11:59:42,982 <string>:384] Starting API server at: http://localhost:38915
Ideally, it should have continued by runnning the server on port 8000. Also, it works with option --support_datastore_emulator=False.
This is the code:
import webapp2
import datetime
from google.appengine.ext import db, deferred, ndb
import uuid
from base64 import b64decode, b64encode
import logging
class Email(ndb.Model):
email = ndb.StringProperty()
class DB(webapp2.RequestHandler):
def post(self):
try:
mail = Email()
mail.email = 'Test'
mail.put()
except Exception as e:
print(e)
self.response.headers['Content-Type'] = 'application/json; charset=utf-8'
return self.response.out.write(e)
def get(self):
try:
e1 = Email.query()
logging.critical('count is: %s' % e1.count)
e1k = e1.get(keys_only=True)
logging.critical('count 2 is: %s' % e1k.count)
e1 = e1.get()
key = unicode(e1.key.urlsafe())
logging.critical('This is a critical message: %s' % key)
logging.critical('This is a critical message: %s' % e1k)
e2 = ndb.Key(urlsafe=key).get()
self.response.headers['Content-Type'] = 'application/json; charset=utf-8'
return self.response.out.write(str(e2.email))
except Exception as e:
print(e)
self.response.headers['Content-Type'] = 'application/json; charset=utf-8'
return self.response.out.write(e)
app = webapp2.WSGIApplication([
('/', DB)
], debug=True)
How can I find the reason this is not working?
Edit: I figured that the dev server works and writes to a datastore even with support_datastore_emulator=False option. I am confused by this option. I also don't know where the database is stored currently.

It should be count() and not count i.e.
logging.critical('count is: %s' % e1.count())
A get returns only 1 record and so it doesn't make sense to do a count after calling a get. Besides, the count operation is a method of the query instance not the results. This means the following code is incorrect
e1k = e1.get(keys_only=True)
logging.critical('count 2 is: %s' % e1k.count)
You should replace it with
elk = e1.fetch(keys_only=True) # fetch gives an array
logging.critical('count 2 is: %s' % len(e1k))
When you first run your App, it will execute the GET part of your code and because this is the first time your App is being run, you have no record in Datastore. This means e1 = e1.get() will return None and key = unicode(e1.key.urlsafe()) will lead to an error.
You have to modify your code to first check you have a value for e1 or e2 before you attempt to use the keys.
I ran your code with dev_appserver.py and it displayed these errors for me in the logs. But I ran it with an older version of gcloud SDK (Google Cloud SDK 367.0.0). I don't know why yours exited without displaying any errors. Maybe it's due to the version...??
Separately - Don't know why you're importing db. Google moved on to ndb long ago and you don't use db in your code
The default datastore (for the older generation runtimes like Python 2) is in .config (hidden folder) > gcloud > emulators > datastore
You can also specify your own location by using the flag --datastore_path. See documentation

Related

How to log Stackdriver log messages correlated by trace id using stdout Go 1.11

I'm using Google App Engine Standard Environment with the Go 1.11 runtime. The documentation for Go 1.11 says "Write your application logs using stdout for output and stderr for errors". The migration from Go 1.9 guide also suggests not calling the Google Cloud Logging library directly but instead logging via stdout.
https://cloud.google.com/appengine/docs/standard/go111/writing-application-logs
With this in mind, I've written a small HTTP Service (code below) to experiment logging to Stackdriver using JSON output to stdout.
When I print plain text messages they appear as expected in the Logs Viewer panel under textPayload. When I pass a JSON string they appear under jsonPayload. So far, so good.
So, I added a severity field to the output string and Stackdriver Log Viewer successfully categorizes the message according to the levelled logging NOTICE, WARNING etc.
https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry
The docs say to set the trace identifier to correlate log entries with the originating request log. The trace ID is extracted from the X-Cloud-Trace-Context header set by the container.
Simulate it locally using curl -v -H 'X-Cloud-Trace-Context: 1ad1e4f50427b51eadc9b36064d40cc2/8196282844182683029;o=1' http://localhost:8080/
However, this does not cause the messages to be threaded by request, but instead the trace property appears in the jsonPayload object in the logs. (See below).
Notice that severity has been interpreted as expected and does not appear in the jsonPayload. I had expected the same to happen for trace, but instead it appears to be unprocessed.
How can I achieve nested messages within the original request log message? (This must be done using stdout on Go 1.11 as I do not wish to log directly with the Google Cloud logging package).
What exactly is GAE doing to parse the stdout stream from my running process? (In the setup guide for VMs on GCE there is something about installing an agent program to act as a conduit to Stackdriver logging- is this what GAE has installed?)
app.yaml file looks like this:
runtime: go111
handlers:
- url: /.*
script: auto
package main
import (
"fmt"
"log"
"net/http"
"os"
"strings"
)
var projectID = "glowing-market-234808"
func parseXCloudTraceContext(t string) (traceID, spanID, traceTrue string) {
slash := strings.Index(t, "/")
semi := strings.Index(t, ";")
equal := strings.Index(t, "=")
return t[0:slash], t[slash+1 : semi], t[equal+1:]
}
func sayHello(w http.ResponseWriter, r *http.Request) {
xTrace := r.Header.Get("X-Cloud-Trace-Context")
traceID, spanID, _ := parseXCloudTraceContext(xTrace)
trace := fmt.Sprintf("projects/%s/traces/%s", projectID, traceID)
warning := fmt.Sprintf(`{"trace":"%s","spanId":"%s", "severity":"WARNING","name":"Andy","age":45}`, trace, spanID)
fmt.Fprintf(os.Stdout, "%s\n", warning)
message := "Hello"
w.Write([]byte(message))
}
func main() {
http.HandleFunc("/", sayHello)
port := os.Getenv("PORT")
if port == "" {
port = "8080"
}
log.Fatal(http.ListenAndServe(fmt.Sprintf(":%s", port), nil))
}
Output shown in the Log viewer:
...,
jsonPayload: {
age: 45
name: "Andy"
spanId: "13076979521824861986"
trace: "projects/glowing-market-234808/traces/e880a38fb5f726216f94548a76a6e474"
},
severity: "WARNING",
...
I have solved this by adjusting the program to use logging.googleapis.com/trace in place of trace and logging.googleapis.com/spanId in place of spanId.
warning := fmt.Sprintf(`{"logging.googleapis.com/trace":"%s","logging.googleapis.com/spanId":"%s", "severity":"WARNING","name":"Andy","age":45}`, trace, spanID)
fmt.Fprintf(os.Stdout, "%s\n", warning)
It seems that GAE is using the logging agent google-fluentd (a modified version of the fluentd log data colletor.)
See this link for a full explanation.
https://cloud.google.com/logging/docs/agent/configuration#special-fields
[update] June 25th, 2019: I've written a Logrus plugin that will help to thread log entries by HTTP request. It's available under on GitHub https://github.com/andyfusniak/stackdriver-gae-logrus-plugin.
[update]] April 3rd, 2020: I've since switched to using Cloud Run and the Logrus plugin appears to work fine with this platform also.

Broken datetime on CloudSQL + Google App Engine

So I had a lot of problems with datetimes in Django on the App Engine, but then I started tracing the error and seems like there's more serious problems. Here's what I'm seeing.
Here's my database:
mysql> select * from polls_question;
+----+---------------+----------------------------+
| id | question_text | pub_date |
+----+---------------+----------------------------+
| 1 | test | 2016-02-08 15:24:44.000000 |
+----+---------------+----------------------------+
1 row in set (0.16 sec)
And here's the code trying to read this data:
import MySQLdb
import os
import webapp2
class IndexPage(webapp2.RequestHandler):
def get(self):
self.response.headers['Content-Type'] = 'text/plain'
env = os.getenv('SERVER_SOFTWARE')
if (env and env.startswith('Google App Engine/')):
# Connecting from App Engine
db = MySQLdb.connect(
unix_socket='/cloudsql/<removed>:<removed>',
user='root',db='gaetest')
else:
# Connecting from an external network.
# Make sure your network is whitelisted
db = MySQLdb.connect(
host='<removed>',
port=3306,
user='root', passwd='<removed>',db='<removed>')
cursor = db.cursor()
cursor.execute('SELECT * FROM polls_question')
for r in cursor.fetchall():
self.response.write('%s\n' % str(r))
db.close()
app = webapp2.WSGIApplication([
('/', IndexPage),
])
On my computer it gives me:
(1L, 'test', datetime.datetime(2016, 2, 8, 15, 24, 44))
and when I access the remote url I get:
(1L, 'test', None)
Not sure what more I can do, this example is as simplified as it can get. Does anyone have any clue what's going on? Too bad Google is typically impossible to get hold off.
So after having spent way too much time on this I found the problem, the database was created using django's migrate function, which lead to it creating a datetime(6) (high precision), these don't work when accessing the site remotely. Normal datetimes work. So changing those makes it work.
The cause of the problem was when I used a newer mysql-python lib locally which supported datetime(6).
It was a bug with App Engine library MySQLdb 1.2.4b4, which at time of writing is also "latest". Switching the app.yaml dependency for MySQLdb from "latest" or "1.2.4b4" to "1.2.5" or "1.2.4" should fix the issue.

X-Appengine-Inbound-Appid header not set

I have two AppEngine modules, a default module running Python and "java" module running Java. I'm accessing the Java module from the default module using urlfetch. According to the AppEngine docs (cloud.google.com/appengine/docs/java/appidentity), I can verify in the Java module that the request originates from a module in the same app by checking the X-Appengine-Inbound-Appid header.
However, this header is not being set (in a production deployment). I use urlfetch in the Python module as follows:
hostname = modules.get_hostname(module="java")
hostname = hostname.replace('.', '-dot-', 2)
url = "http://%s/%s" % (hostname, "_ah/api/...")
result = urlfetch.fetch(url=url, follow_redirects=False, method=urlfetch.GET)
Note that I'm using the notation:
<version>-dot-<module>-dot-<app>.appspot.com
rather than the notation:
<version>.<module>.<app>.appspot.com
which for some reason results in a 404 response.
In the Java module I'm running a servlet filter which looks at all the request headers as follows:
Enumeration<String> headerNames = httpRequest.getHeaderNames();
while (headerNames.hasMoreElements()) {
String headerName = headerNames.nextElement();
String headerValue = httpRequest.getHeader(headerName);
mLog.info("Header: " + headerName + " = " + headerValue);
}
AppEngine does set some headers, e.g. X-AppEngine-Country. But the X-Appengine-Inbound-Appid header is not set.
Why am I'm not seeing the documented behaviour? Any suggestions would be much appreciated.
Have a look at what I've been answered on Google groups, which led to an issue opened on the public issue tracker.
As suggested in the answer I received you can follow, for any update, the issue over there.

How to catch BigQuery loading errors from an AppEngine pipeline

I have built a pipeline on AppEngine that loads data from Cloud Storage to BigQuery. This works fine, ..until there is any error. How can I can loading exceptions by BigQuery from my AppEngine code?
The code in the pipeline looks like this:
#Run the job
credentials = AppAssertionCredentials(scope=SCOPE)
http = credentials.authorize(httplib2.Http())
bigquery_service = build("bigquery", "v2", http=http)
jobCollection = bigquery_service.jobs()
result = jobCollection.insert(projectId=PROJECT_ID,
body=build_job_data(table_name, cloud_storage_files))
#Get the status
while (not allDone and not runtime.is_shutting_down()):
try:
job = jobCollection.get(projectId=PROJECT_ID,
jobId=insertResponse).execute()
#Do something with job.get('status')
except:
exc_type, exc_value, exc_traceback = sys.exc_info()
logging.error(traceback.format_exception(exc_type, exc_value, exc_traceback))
time.sleep(30)
This gives me status error, or major connectivity errors, but what I am looking for is functional errors from BigQuery, like fields formats conversion errors, schema structure issues, or other issues BigQuery may have while trying to insert rows to tables.
If any "functional" error on BigQuery's side happens, this code will run successfully and complete normally, but no table will be written on BigQuery. Not easy to debug when this happens...
You can use the HTTP error code from the exception. BigQuery is a REST API, so the response codes that are returned match the description of HTTP error codes here.
Here is some code that handles retryable errors (connection, rate limit, etc), but re-raises when it is an error type that it doesn't expect.
except HttpError, err:
# If the error is a rate limit or connection error, wait and
# try again.
# 403: Forbidden: Both access denied and rate limits.
# 408: Timeout
# 500: Internal Service Error
# 503: Service Unavailable
if err.resp.status in [403, 408, 500, 503]:
print '%s: Retryable error %s, waiting' % (
self.thread_id, err.resp.status,)
time.sleep(5)
else: raise
If you want even better error handling, check out the BigqueryError class in the bq command line client (this used to be available on code.google.com, but with the recent switch to gCloud, it isn't any more. But if you have gcloud installed, the bq.py and bigquery_client.py files should be in the installation).
The key here is this part of the pasted code:
except:
exc_type, exc_value, exc_traceback = sys.exc_info()
logging.error(traceback.format_exception(exc_type, exc_value, exc_traceback))
time.sleep(30)
This "except" is catching every exception, logging it, and letting the process continue without any consideration for re-trying.
The question is, what would you like to do instead? At least the intention is there with the "#Do something" comment.
As a suggestion, consider App Engine's task queues to check the status, instead of a loop with a 30 second wait. When tasks get an exception, they are automatically retried - and you can tune that behavior.

Writing files to Dropbox account from GAE

I am trying to create files in a Dropbox.com folder from a GAE application.
I have done all the steps the register a Dropbox application and installed the Python SDK from Dropbox locally on my development machine. (see dropbox.com API).
It all works perfectly when I use the cli_client.py test script in the dropbox SDK on my local machine to access dropbox - can 'put' files etc.
I now want to start working in GAE environment, so things get a bit tricky.
Some help would be useful.
For those familiar with the Dropbox API code, I had the following issues thus far:
Issue 1
The rest.py Dropbox API module uses pkg_resources to get the certs installed in site-packages of a local machine installation.
I replaced
TRUSTED_CERT_FILE = pkg_resources.resource_filename(__name__, 'trusted-certs.crt')
with
TRUSTED_CERT_FILE = file('trusted-certs.crt')
and placed the cert file in my GAE application directory. Perhaps this is not quite right; see my authentication error code below.
Issue 2
The session.py Dropbox API module uses oauth module, so I changed the include to appengine oauth.
But raised an exception that GAE's oauth does not have OAuthConsumer method used by the Dropbox session.py module. So i downloaded oauth 1.0 and added to my application an now import this instead of GAE oauth.
Issue 3
GAE ssl module does not seem to have CERT_REQUIRED property.
This is a constant, so I changed
self.cert_reqs = ssl.CERT_REQUIRED
to
self.cert_reqs = 2
This is used when calling
ssl.wrap_socket(sock, cert_reqs=self.cert_reqs, ca_certs=self.ca_certs)
Authentication Error
But I still can't connect to Dropbox:
Status: 401
Reason: Unauthorized
Body: {"error": "Authentication failed"}
Headers: [('date', 'Sun, 19 Feb 2012 15:11:12 GMT'), ('transfer-encoding', 'chunked'), ('connection', 'keep-alive'), ('content-type', 'application/json'), ('server', 'dbws')]
Here's my patched version of Dropbox Python SDK 1.4 which works well for me with Python 2.7 GAE: dropbox_python_sdk_gae_patched.7z.base64. No extra third-party libraries needed, only those provided by GAE environment.
Only file uploading (put_file) is tested. Here're setup steps:
Unpack archive to the root folder of GAE application (if main app is in the root folder). You can decode BASE64 using Base64 Encoder/Decoder: base64.exe -d dropbox_python_sdk_gae_patched.7z.base64 dropbox_python_sdk_gae_patched.7z.
Setup APP_KEY, APP_SECRET, ACCESS_TYPE, ACCESS_TOKEN_KEY, ACCESS_TOKEN_SECRET. First three are configured at dropbox application creation time. Last two are obtained when granting application access to specific dropbox account, you can get them through cli_client.py (from DB Python SDK) from token_store.txt file.
Use in the code like this:
import dropbox
# ...
def DropboxUpload(path, data):
sess = dropbox.session.DropboxSession(APP_KEY, APP_SECRET, ACCESS_TYPE)
sess.set_token(ACCESS_TOKEN_KEY, ACCESS_TOKEN_SECRET)
cli = dropbox.client.DropboxClient(sess)
data_file = StringIO.StringIO(data)
return cli.put_file(path, data_file)
# ...
import json
class DropboxUploadHandlerExample(webapp2.RequestHandler):
def get(self):
url = "http://www.google.com/"
result = urlfetch.fetch(url)
self.response.headers['Content-Type'] = 'application/json'
self.response.out.write(json.dumps(DropboxUpload('/fetch_result.dat', result.content)))
I successfully uploaded from Google Appengine to Dropbox with my own patched version
of the Dropbox SDK: https://github.com/cklein/dropbox-client-python
The usage of urllib2 was replaced by huTools.http: https://github.com/hudora/huTools/
This is the code that is called in a request handler:
db_client = dropbox.get_dropbox_client(consumer_key='', consumer_secret='', access_token_key='', access_token_secret='')
fileobj = StringIO.StringIO(data)
path = '/some/path/filename'
resp = db_client.put_file(path, fileobj)
fileobj.close()
As of April 2016, none of the other suggestions work. (Dropbox API version 2, Python SDK version 6.2).
If you only need a few of the SDK functions, I found it easiest to just use the HTTP API directly:
def files_upload(f, path, mode='add', autorename=False, mute=False):
args = {
'path': path,
'mode': mode,
'autorename': autorename,
'mute': mute,
}
headers = {
'Authorization': 'Bearer {}'.format(ACCESS_TOKEN),
'Dropbox-API-Arg': json.dumps(args),
'Content-Type': 'application/octet-stream',
}
request = urllib2.Request('https://content.dropboxapi.com/2/files/upload', f, headers=headers)
r = urllib2.urlopen(request)
I have patched the Dropbox Python SDK version 2.2 to work on Google App Engine. Please find the relevant code here:
https://github.com/duncanhawthorne/gae-dropbox-python
The relevant code patch (copied from github) for rest.py is here:
import io
import pkg_resources
-import socket
+#import socket
import ssl
import sys
import urllib
+import urllib2
+def mock_urlopen(method,url,body,headers,preload_content):
+ request = urllib2.Request(url, body, headers=headers)
+ r = urllib2.urlopen(request)
+ return r
+
try:
import json
except ImportError:
## -23,7 +29,10 ##
SDK_VERSION = "2.2.0"
-TRUSTED_CERT_FILE = pkg_resources.resource_filename(__name__, 'trusted-certs.crt')
+try:
+ TRUSTED_CERT_FILE = pkg_resources.resource_filename(__name__, 'trusted-certs.crt')
+except:
+ TRUSTED_CERT_FILE = file('trusted-certs.crt')
class RESTResponse(io.IOBase):
## -125,6 +134,7 ## def flush(self):
pass
def create_connection(address):
+ return
host, port = address
err = None
for res in socket.getaddrinfo(host, port, 0, socket.SOCK_STREAM):
## -152,7 +162,7 ## def json_loadb(data):
class RESTClientObject(object):
- def __init__(self, max_reusable_connections=8, mock_urlopen=None):
+ def __init__(self, max_reusable_connections=8, mock_urlopen=mock_urlopen):
"""
Parameters
max_reusable_connections
## -206,7 +216,7 ## def request(self, method, url, post_params=None, body=None, headers=None, raw_re
raise ValueError("headers should not contain newlines (%s: %s)" %
(key, value))
- try:
+ if True:
# Grab a connection from the pool to make the request.
# We return it to the pool when caller close() the response
urlopen = self.mock_urlopen if self.mock_urlopen else self.pool_manager.urlopen
## -217,14 +227,14 ## def request(self, method, url, post_params=None, body=None, headers=None, raw_re
headers=headers,
preload_content=False
)
- r = RESTResponse(r) # wrap up the urllib3 response before proceeding
- except socket.error as e:
- raise RESTSocketError(url, e)
- except urllib3.exceptions.SSLError as e:
- raise RESTSocketError(url, "SSL certificate error: %s" % e)
+ #r = RESTResponse(r) # wrap up the urllib3 response before proceeding
+ #except socket.error as e:
+ # raise RESTSocketError(url, e)
+ #except urllib3.exceptions.SSLError as e:
+ # raise RESTSocketError(url, "SSL certificate error: %s" % e)
- if r.status not in (200, 206):
- raise ErrorResponse(r, r.read())
+ #if r.status not in (200, 206):
+ # raise ErrorResponse(r, r.read())
return self.process_response(r, raw_response)
## -321,10 +331,11 ## def PUT(cls, *n, **kw):
return cls.IMPL.PUT(*n, **kw)
-class RESTSocketError(socket.error):
+class RESTSocketError():
"""A light wrapper for ``socket.error`` that adds some more information."""
def __init__(self, host, e):
+ return
msg = "Error connecting to \"%s\": %s" % (host, str(e))
socket.error.__init__(self, msg)

Resources