"ExistenceError" in simple AppEngine + Google Cloud Storage application - google-app-engine

I have a simple AppEngine handler as follows:
class TestGS(webapp2.RequestHandler):
def get(self):
file_name = '/gs/ds_stats/testfile'
files.gs.create(file_name, mime_type='text/html')
with files.open(file_name, 'a') as file_handle:
file_handle.write("foo")
files.finalize(file_name)
However, when I call this handler, I get ExistenceError: ApplicationError: 105 at the line with files.open(....
This seems like a super simple scenario, and there's no indication at all as to why this is failing (especially since the files.gs.create right above it seems to have succeeded, though, is there any way to verify this?).
Looking through the source code, I see the following problems can cause this error:
if (e.application_error in
[file_service_pb.FileServiceErrors.EXISTENCE_ERROR,
file_service_pb.FileServiceErrors.EXISTENCE_ERROR_METADATA_NOT_FOUND,
file_service_pb.FileServiceErrors.EXISTENCE_ERROR_METADATA_FOUND,
file_service_pb.FileServiceErrors.EXISTENCE_ERROR_SHARDING_MISMATCH,
file_service_pb.FileServiceErrors.EXISTENCE_ERROR_OBJECT_NOT_FOUND,
file_service_pb.FileServiceErrors.EXISTENCE_ERROR_BUCKET_NOT_FOUND,
]):
raise ExistenceError()
That's a pretty large range of issues... Of course it doesn't tell me which one! And again, strange that the 'create' seems to work.

The problem turned out to be a lack of clarity in the documentation. files.gs.create returns a special 'writeable file path' which you need to feed in to open and finalize. A correct example looks like this:
class TestGS(webapp2.RequestHandler):
def get(self):
file_name = '/gs/ds_stats/testfile'
writable_file_name = files.gs.create(file_name, mime_type='text/html')
with files.open(writable_file_name, 'a') as file_handle:
file_handle.write("foo")
files.finalize(writable_file_name)

Related

Method (step) get a NotImplementedError in file A but in file B not

I implemented step:
#given('my step')
def step_impl(context):
somthing
In file fileA.py this method (step) get a NotImplementedError but if i moved this method to another file eg. fileB.py step is running properly.
I don't know why and how can i fix it
You provide not enough information to answer your question.
Provide the StackTrace or look at the StackTrace when you get the NotImplementedError.
HINT:
Could be a weird Unicode encoding problem in "fileA.py"
(Unicode has also invisible characters)

Is it possible to get the log point on GAE?

I'm wondering if it's possible to log the line/column number when i log an error. The GAE logging lib seems to store only the error message but I think it would be quite valuable to get the exact location of the error/log point like most logging libraries do.
What you request is definitely possible, you just need to be very explicit about it! I recommend Andrew Gerrand's excellent article at http://blog.golang.org/error-handling-and-go for general information on the issue, including notes specific to App Engine.
That article does not specifically address stack traces, but of course you could do those yourself, via http://golang.org/pkg/runtime/#Stack .
Let's be honest and admit that Go -- being by design more of a system-programming language than of an application-focused one -- doesn't do quite as much implicit, automatic hand-holding for you, as more app-oriented languages such as the other App Engine ones -- Java, Python, PHP... [*] but, Go does give you all the tools you need to do just as little, or as much, "supporting infrastructure", as you desire to have in support of your own apps!-)
[*] hey, you don't even get automatically by-default propagating exceptions, as you do for other languages -- nay, you're responsible for catching and dealing with errors yourself, oh the horror...!-)
You can use the runtime package to trace the caller(I assume that's basically what you want). There is no GAE specific issue and it has nothing to do with the error handling.
// The depth specifies how many stack frames above
// lives the source line to be identified in the log message.
func traceCaller(depth int)(string, int){
_, file, line, ok := runtime.Caller(2 + depth)
if !ok {
file = "???"
line = 1
} else {
slash := strings.LastIndex(file, "/")
if slash >= 0 {
file = file[slash+1:]
}
}
return file, line
}

Google Cloud Storage listbucket in App Engine behaviour

gcs.listbucket() method behaves in a weird fashion: sometimes it seems that the bucket is empty when it is not: calling this handler several times doen't always returns the same list: sometimes it behaves correctly and sometimes it returns an iterator with zero items.
class MainHandler(webapp2.RequestHandler):
def get(self):
bucket_name = '/my-non-empty-bucket'
bucket_images = gcs.listbucket(bucket_name)
self.response.write('<br> '.join([b.filename for b in bucket_images]))
It seems that listbucket is timed out sometimes, I don't understand.
my retry parameters:
my_default_retry_params = gcs.RetryParams(initial_delay=0.2,
max_delay=5.0,
backoff_factor=2,
max_retry_period=15,
urlfetch_timeout=10)
gcs.set_default_retry_params(my_default_retry_params)
The log in app engine seems perfectly fine, no errors.
Any suggestion about how to further understand this behaviour wuld be appreciated

App Engine + NoseGAE bizarre broken test

I'm using the NoseGAE to write local unit tests for my App Engine application, however something is suddenly going wrong with one of my tests. I have standard setUp and tearDown functions, but one test seemingly broke for a reason I can't discern. Even stranger, setUp and tearDown are NOT getting called each time. I added global variables to count setUp/tearDown calls, and on my 4th test (the now seemingly broken one), setUp has been called twice and tearDown has been called once. Further, one of the objects from the third test exists when I query it by id, but not in a general query for its type. Here's some code that gives the bizarre picture:
class GameTest(unittest.TestCase):
def setUp(self):
self.testapp = webtest.TestApp(application)
self.testbed = testbed.Testbed()
self.testbed.activate()
self.testbed.init_datastore_v3_stub(
consistency_policy=datastore_stub_util.PseudoRandomHRConsistencyPolicy(probability=1),
require_indexes=True,
root_path="%s/../../../" % os.path.dirname(__file__)
)
def tearDown(self):
self.testbed.deactivate()
self.testapp.cookies.clear()
def test1(self):
...
def test2(self):
...
def test3(self):
...
# I create a Game object with the id 123 in this particular test
Game(id=123).put()
...
def test4(self):
print "id lookup: ", Game.get_by_id(123)
print "query: ", Game.query().get()
self.assertIsNone(Game.get_by_id(123))
This is an abstraction of the tests, but illustrates the issue.
The 4th test fails because it asserts that an object with that id does not exist. When I print out the two statements:
id lookup: Game(key=Key('Game', 123))
query: None
The id lookup shows the object created in test3, but the query lookup is EMPTY. This makes absolutely no sense to me. Further, I am 100% sure the test was working earlier. Does anyone have any idea how this is even possible? Could I possibly have some local corrupted file causing an issue?
I somewhat "solved" this. This issue only reproduced when I had other test cases in other files that were failing. Once I solved those, all my tests passed. I still don't fully understand why other failing tests should cause these bizarre issues with the testbed, but to anyone else having this issue, try fixing your other test cases first and see if that doesn't cause it to go away.

serve different wsgiapplications depending on request domain on GAE with threadsafe:true

what im trying to do is to load different applications (webapp2.WSGIApplication) depending on the request domain.
for example www.domain_1.com should load the application in app1.main.application while www.domain_2.com should load app2.main.appplication.
of course im on the same GAE appid and im using namespaces to separate the apps data.
this works pretty good with 'threadsafe:false' and a runner.py file where a function determines which application to return
it seems that with 'threadsafe:true' the first request loads the wsgiapplication into the instance and further requests dont execute the 'application dispatching' logic any more so the request gets a response from the wrong app.
im using python2.7 and webapp2
what is the best way to do this?
edit:
a very simplified version of my runner.py
def main():
if domain == 'www.mydomain_1.com':
from app_1 import application
namespace = 'app_1'
elif domain == 'www.domain_2.com':
from app_2 import application
namespace = 'app_2'
namespace_manager.set_namespace(namespace)
return wsgiref.handlers.CGIHandler().run(application)
if __name__ == '__main__':
main()
and in app.yaml
- url: /.*
script: app-runner.py
Your runner script is a CGI script. The full behavior of a CGI script with multithreading turned on is not documented, and the way the docs are written I'm guessing this won't be supported fully. Instead, the docs say you must refer to the WSGI application object directly from app.yaml, using the module path to a global variable containing the object, when multithreading is turned on. (CGI scripts retain their old behavior in Python 2.7 with multithreading turned off.)
The behavior you're seeing is explained by your use of imports. Within a single instance, each import statement only has an effect the first time it is encountered. After that, the module is assumed to be imported and the import statement has no effect on subsequent requests. You can import both values into separate names, then call run() with the appropriate value.
But if you want to enable multithreading (and that's a good idea), your dispatcher should be a WSGI application itself, stored in a module global referred to by app.yaml. I don't know offhand how to dispatch a request to another WSGI application from within a WSGI application, but that might be a reasonable thing to do. Alternatively, you might consider using or building a layer above WSGI to do this dispatch.
made it happen by subclassing webapp2.WSGIApplication and overriding __call__() which is called before dispatching to a RequestHandler.
prefixing routes (and removing the prefix in the handlers initialize) and substructuring config to be able to use the instance memory.
class CustomWSGIApplication(webapp2.WSGIApplication):
def __call__(self, environ, start_response):
routes, settings, ns = get_app(environ)
namespace_manager.set_namespace(ns)
environ['PATH_INFO'] = '/%s%s' %(ns, environ.get('PATH_INFO'))
for route in routes:
r, h = route # returns a tuple with mapping and handler
newroute = ('/%s%s'%(ns, r), h,)
self.router.add(newroute)
if settings:
self.config[ns] = settings
self.debug = debug
with self.request_context_class(self, environ) as (request, response):
try:
if request.method not in self.allowed_methods:
# 501 Not Implemented.
raise exc.HTTPNotImplemented()
rv = self.router.dispatch(request, response)
if rv is not None:
response = rv
except Exception, e:
try:
# Try to handle it with a custom error handler.
rv = self.handle_exception(request, response, e)
if rv is not None:
response = rv
except HTTPException, e:
# Use the HTTP exception as response.
response = e
except Exception, e:
# Error wasn't handled so we have nothing else to do.
response = self._internal_error(e)
try:
return response(environ, start_response)
except Exception, e:
return self._internal_error(e)(environ, start_response)

Resources