BadValueError: Entity has uninitialized properties ___ after resetting indexes and clearing memcache - google-app-engine

I'm working on a Google App Engine application in python. I tried to switch a query I was running for one of my routes to query on only one property instead of 2, which caused an indexing error to appear whenever we tried running that query.
It was something along the lines of "No index matching the specified parameters could be found", but I don't have any screen shots at the moment. In order to try and rectify the situation, we ran appcfg.py vacuum_indices and deleted all indices related to the original search. We then uploaded a new index.yaml specifying the new index. Though we were able to see that the new indexes had indeed been created on the admin panel, and the old ones were gone, we were still getting the same error.
We're really unsure why this is happening, and are having trouble finding documentation online for these issues. Our next thought was that some previous state in memcache was causing the query to attempt to use it's old index. So we flushed memcache, and now we're getting this error:
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1535, in __call__
rv = self.handle_exception(request, response, e)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1529, in __call__
rv = self.router.dispatch(request, response)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1278, in default_dispatcher
return route.handler_adapter(request, response)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1102, in __call__
return handler.dispatch()
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 572, in dispatch
return self.handle_exception(e, self.app.debug)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 570, in dispatch
return method(*args, **kwargs)
File "/base/data/home/apps/s~dev-erpetcloud2/dev1.392600188150722624/routes/users.py", line 172, in post
res_dict = cp_user.to_dict()
File "/base/data/home/apps/s~dev-erpetcloud2/dev1.392600188150722624/routes/models/../models/cp_models.py", line 248, in to_dict
animal_dict = animal.to_dict()
File "/base/data/home/apps/s~dev-erpetcloud2/dev1.392600188150722624/routes/models/../models/cp_models.py", line 574, in to_dict
protocol, params = self.get_protocol_and_params()
File "/base/data/home/apps/s~dev-erpetcloud2/dev1.392600188150722624/routes/models/../models/cp_models.py", line 395, in get_protocol_and_params
record = self.protocol_state_key.get()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/key.py", line 572, in get
return self.get_async(**ctx_options).get_result()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 342, in get_result
self.check_success()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 389, in _help_tasklet_along
value = gen.send(val)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/context.py", line 765, in get
pbs = entity._to_pb(set_key=False).SerializePartialToString()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/model.py", line 3158, in _to_pb
self._check_initialized()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/model.py", line 3014, in _check_initialized
'Entity has uninitialized properties: %s' % ', '.join(baddies))
BadValueError: Entity has uninitialized properties: title
Looking through the datastore, the entity that this trace references definitely does have a 'title' property.
I've looked around a lot for errors that can arise from deleting indices and flushing memcache, and nothing useful has come up.
If someone could perhaps give me a bit of insight into what could be happening here and how these systems work (my mental model might be off), or point me in the right direction, that would be fantastic. Thanks!!

This error signifies that the property 'title' has been specified as a required property but you are trying to write an entity to datastore without initializing this property. This error occurs only at the time of put(). By any chance, did you make any changes in the entity definition, or a part of code which writes these entities to datastore ?
Edit: The error can also happen while trying to read an entity which has no value specified for a 'required' property.

Related

Google App Engine 'Connection' object has no attribute 'commit'

We are randomly getting the following error when using Google App Engine standard environment python2.7:
AttributeError: 'Connection' object has no attribute 'commit'
An example stack trace:
Traceback (most recent call last): File
"/base/alloc/tmpfs/dynamic_runtimes/python27g/fdc6d631da52d25b/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py",
line 1535, in call
rv = self.handle_exception(request, response, e) File "/base/alloc/tmpfs/dynamic_runtimes/python27g/fdc6d631da52d25b/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py",
line 1529, in call
rv = self.router.dispatch(request, response) File "/base/alloc/tmpfs/dynamic_runtimes/python27g/fdc6d631da52d25b/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py",
line 1278, in default_dispatcher
return route.handler_adapter(request, response) File "/base/alloc/tmpfs/dynamic_runtimes/python27g/fdc6d631da52d25b/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py",
line 1102, in call
return handler.dispatch() File "/base/alloc/tmpfs/dynamic_runtimes/python27g/fdc6d631da52d25b/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py",
line 572, in dispatch
return self.handle_exception(e, self.app.debug) File "/base/alloc/tmpfs/dynamic_runtimes/python27g/fdc6d631da52d25b/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py",
line 570, in dispatch
return method(*args, **kwargs) File "/base/data/home/apps/s~boutirapp/prod20210407-0912.434271348867830388/api/sixdegree.py",
line 172, in post
body = self.process(*args, **kwargs) File "/base/data/home/apps/s~boutirapp/prod20210407-0912.434271348867830388/pages/paydollar_datafeed.py",
line 89, in process
BTOrder.confirmOrder(Ref, user) File "/base/data/home/apps/s~boutirapp/prod20210407-0912.434271348867830388/entities/btorder.py",
line 2034, in confirmOrder
self = db.run_in_transaction_options(xg_on_retry_5, cls._confirmOrderTransaction, orderId, paypalTxId, paypalResp) File
"/base/alloc/tmpfs/dynamic_runtimes/python27g/fdc6d631da52d25b/python27/python27_lib/versions/1/google/appengine/api/datastore.py",
line 2641, in RunInTransactionOptions
function, *args, **kwargs) File "/base/alloc/tmpfs/dynamic_runtimes/python27g/fdc6d631da52d25b/python27/python27_lib/versions/1/google/appengine/api/datastore.py",
line 2716, in _RunInTransactionInternal
ok, result = _DoOneTry(function, args, kwargs) File "/base/alloc/tmpfs/dynamic_runtimes/python27g/fdc6d631da52d25b/python27/python27_lib/versions/1/google/appengine/api/datastore.py",
line 2770, in _DoOneTry
if _GetConnection().commit(): AttributeError: 'Connection' object has no attribute 'commit'
The error occurs randomly when we are committing a datastore transaction using the google.appengine.ext.db library. The same lines of code might work fine when we retry immediately after the error. It also doesn't look like an error of our code as it is a missing attribute from a library object.
Any help to tackle the error is appreciated.
Edit: the error started to occur on 8 March 2021. We didn't change any related code close to that day.

google appengine db to ndb migration error: 'model_from_protobuf'

I am in the process of migrating from from db to ndb and i am getting an error in that i do not understand. It is coming from the following line:
return [db.model_from_protobuf(entity_pb.EntityProto(x)) for x in data]
The full error is below:
File
"/base/alloc/tmpfs/dynamic_runtimes/python27g/174489198b39d9fb/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1529, in __call__
rv = self.router.dispatch(request, response)
File "/base/alloc/tmpfs/dynamic_runtimes/python27g/174489198b39d9fb/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1278, in default_dispatcher
return route.handler_adapter(request, response)
File "/base/alloc/tmpfs/dynamic_runtimes/python27g/174489198b39d9fb/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1102, in __call__
return handler.dispatch()
File "/base/data/home/apps/s~proj-/44.427741511160881878/handlers/handler.py", line 329, in dispatch
super(SiteHandler, self).dispatch()
File "/base/data/home/apps/s~proj-/44.427741511160881878/handlers/handler.py", line 130, in dispatch
webapp2.RequestHandler.dispatch(self)
File "/base/alloc/tmpfs/dynamic_runtimes/python27g/174489198b39d9fb/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 572, in dispatch
return self.handle_exception(e, self.app.debug)
File "/base/alloc/tmpfs/dynamic_runtimes/python27g/174489198b39d9fb/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 570, in dispatch
return method(*args, **kwargs)
File "/base/data/home/apps/s~proj-/44.427741511160881878/handlers/site_handlers.py", line 112, in get
self.get_not_logged_in()
File "/base/data/home/apps/s~proj-/44.427741511160881878/handlers/site_handlers.py", line 139, in get_not_logged_in
(missions_memcache_key, missions_expiration)])
File "/base/data/home/apps/s~proj/44.427741511160881878/utils.py", line 368, in cache_get_multi
missing_items[k] = deserialize_entities(missing_items[k])
File "/base/data/home/apps/s~proj-/44.427741511160881878/utils.py", line 277, in deserialize_entities
return [db.model_from_protobuf(entity_pb.EntityProto(x)) for x in data]
AttributeError: 'module' object has no attribute 'model_from_protobuf'
What can i change this to fix it? The documentation has not really been so helpful
https://cloud.google.com/appengine/docs/standard/python/ndb/db_to_ndb
The errors seems to come from the fact that you are still using db instead of ndb in that line.
According to the documentation, you should be able to decode from the protobuf using ndb.LocalStructuredProperty. I have found another question where OP uses the aforementioned method to read from protobuf.
I have also found a public repository where it uses pb_to_entity to do the transformation but I'm not fully confident about this because of the lack of comments. You should try both methods and see which one do you prefer.

google app engine NDB records counts from NDB model

How many records we can get from google app engine from single query so that we can display count to user and is we can increase timeout limit 3 seconds to 5 seconds
In my experience, ndb cannot pull more than 1000 records at a time. Here is an example of what happens if I try to use .count() on a table that contains ~500,000 records.
s~project-id> models.Transaction.query().count()
WARNING:root:suspended generator _count_async(query.py:1330) raised AssertionError()
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/google_appengine/google/appengine/ext/ndb/utils.py", line 160, in positional_wrapper
return wrapped(*args, **kwds)
File "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/google_appengine/google/appengine/ext/ndb/query.py", line 1287, in count
return self.count_async(limit, **q_options).get_result()
File "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/google_appengine/google/appengine/ext/ndb/tasklets.py", line 383, in get_result
self.check_success()
File "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/google_appengine/google/appengine/ext/ndb/tasklets.py", line 427, in _help_tasklet_along
value = gen.throw(exc.__class__, exc, tb)
File "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/google_appengine/google/appengine/ext/ndb/query.py", line 1330, in _count_async
batch = yield rpc
File "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/google_appengine/google/appengine/ext/ndb/tasklets.py", line 513, in _on_rpc_completion
result = rpc.get_result()
File "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 614, in get_result
return self.__get_result_hook(self)
File "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/google_appengine/google/appengine/datastore/datastore_query.py", line 2910, in __query_result_hook
self._batch_shared.conn.check_rpc_success(rpc)
File "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/google_appengine/google/appengine/datastore/datastore_rpc.py", line 1377, in check_rpc_success
rpc.check_success()
File "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 580, in check_success
self.__rpc.CheckSuccess()
File "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/google_appengine/google/appengine/api/apiproxy_rpc.py", line 157, in _WaitImpl
self.request, self.response)
File "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 308, in MakeSyncCall
handler(request, response)
File "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 362, in _Dynamic_Next
assert next_request.offset() == 0
AssertionError
To by pass this, you can do something like:
objs = []
q = None
more = True
while more:
_objs, q, more = models.Transaction.query().fetch_page(300, start_cursor=q)
objs.extend(_objs)
But even that will eventually hit memory/timeout limits.
Currently I use Google Dataflow to pre-compute these values and store the results in Datastore as the models DaySummaries & StatsPerUser
EDIT:
snakecharmerb is correct. I was able to use .count() in the production environment, but the more entities it has to count, the longer it seems to take. Here's a screenshot of my logs viewer where it took ~15 seconds to count ~330,000 records
When I tried adding a filter to that query which returned a count of ~4500, it took about a second to run instead.
EDIT #2:
Ok I had another app engine project with a kind with ~8,000,000 records. I tried to do .count() on that in my http request handler and the request timed-out after running for 60 seconds.

Compute Engine OAuth tokens are invalid to App Engine

I've noticed that service accounts on GCE instances now have a much longer token than before, and I suspect they are causing AppEngine applications to be unable to work with them, resulting in an InvalidOAuthParametersError.
I obtain a token from a Compute Engine instance like this:
# curl "http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token" -H "Metadata-Flavor: Google"
{"access_token":".......","expires_in":2544,"token_type":"Bearer"}
If I take the resulting 153 character token and POST it to an AppEngine Python application in an 'Authorisation: Bearer ...' header, it causes the app to fail. The app works fine with a shorter (73 character) token generated for a user account.
The Oauth bit of the App that throws the exception is:
SCOPE = 'https://www.googleapis.com/auth/userinfo.email'
email = oauth.get_current_user(SCOPE).email().lower()
The error from AppEngine is:
Traceback (most recent call last):
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1511, in __call__
rv = self.handle_exception(request, response, e)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1505, in __call__
rv = self.router.dispatch(request, response)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1253, in default_dispatcher
return route.handler_adapter(request, response)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1077, in __call__
return handler.dispatch()
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 547, in dispatch
return self.handle_exception(e, self.app.debug)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 545, in dispatch
return method(*args, **kwargs)
File "/...../v4.398557956806027726/oauth.py", line 13, in post
user_name = self._get_authd_user()
File "/...../v4.398557956806027726/oauth.py", line 36, in _get_authd_user
email = oauth.get_current_user(SCOPE).email().lower()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/oauth/oauth_api.py", line 109, in get_current_user
_maybe_call_get_oauth_user(_scope)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/oauth/oauth_api.py", line 220, in _maybe_call_get_oauth_user
_maybe_raise_exception()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/oauth/oauth_api.py", line 237, in _maybe_raise_exception
raise InvalidOAuthParametersError(error_detail)
InvalidOAuthParametersError
If I truncate the token to 73 characters, I see an InvalidOauthToken error (as you'd expect). If I use the full token at https://www.googleapis.com/oauth2/v3/tokeninfo?access_token=... then it returns correct information for the service account, including the correct scopes (userinfo.email).
I'm also 99% sure that the tokens were shorter last week, and that they worked fine in this scenario.
My question is: why does AppEngine think that these service account tokens are invalid? The error would seem to suggest that it is the length causing the issue, or perhaps I'm missing something.

GAE error (Error code 104) while creating a new blob

On GAE this line of code:
file_name = files.blobstore.create(mime_type='image/png')
drops google.appengine.runtime.DeadlineExceededError
Here is the full method code:
class UploadsHandler(JSONRequestHandler):
def upload_blob(self, content, filename):
file_name = files.blobstore.create(mime_type='image/png')
file_str_list = split_len(content, 65520)
with files.open(file_name, 'a') as f:
for line in file_str_list:
f.write(line)
files.finalize(file_name)
return files.blobstore.get_blob_key(file_name)
Logging message ends with:
A serious problem was encountered with the process that handled this request, causing it to exit. This is likely to cause a new process to be used for the next request to your application. If you see this message frequently, you may be throwing exceptions during the initialization of your application. (Error code 104)
Full error stack:
<class 'google.appengine.runtime.DeadlineExceededError'>:
Traceback (most recent call last):
File "/base/data/home/apps/s~mockup-cloud/1.352909931378411668/main.py", line 389, in main
util.run_wsgi_app(application)
File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/webapp/util.py", line 98, in run_wsgi_app
run_bare_wsgi_app(add_wsgi_middleware(application))
File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/webapp/util.py", line 116, in run_bare_wsgi_app
result = application(env, _start_response)
File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/webapp/_webapp25.py", line 703, in __call__
handler.post(*groups)
File "/base/data/home/apps/s~mockup-cloud/1.352909931378411668/main.py", line 339, in post
original_key = "%s" % self.upload_blob(src)
File "/base/data/home/apps/s~mockup-cloud/1.352909931378411668/main.py", line 268, in upload_blob
file_name = files.blobstore.create(mime_type='image/png')
File "/base/python_runtime/python_lib/versions/1/google/appengine/api/files/blobstore.py", line 68, in create
return files._create(_BLOBSTORE_FILESYSTEM, params=params)
File "/base/python_runtime/python_lib/versions/1/google/appengine/api/files/file.py", line 487, in _create
_make_call('Create', request, response)
File "/base/python_runtime/python_lib/versions/1/google/appengine/api/files/file.py", line 228, in _make_call
rpc.wait()
File "/base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 533, in wait
self.__rpc.Wait()
File "/base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_rpc.py", line 119, in Wait
rpc_completed = self._WaitImpl()
File "/base/python_runtime/python_lib/versions/1/google/appengine/runtime/apiproxy.py", line 131, in _WaitImpl
rpc_completed = _apphosting_runtime___python__apiproxy.Wait(self)
Blob is created while file upload. Other methods of the app work great. It looks like blobstore is not responding for under 30 sec.
Any ideas why this happens?
Thanks!
Seems like you're not the only one having this issue:
http://groups.google.com/group/google-appengine/browse_thread/thread/27e52484946cbdc1#
(posted today)
Seems like Google had some reconfigurations of their servers. Now everything's working fine as it was before.
A runtime.DeadlineExceededError occurs when your request handler takes too long to execute - the blobstore call just happened to be what was running when that happened. You need to profile your handler with appstats to see why it's so slow.

Resources