DeadlineExceededError despite using ndb.put_multi_async - google-app-engine

Question

I have the following POST method in my api. After sending a big JSON collection around 350 entries to the api, I get an DeadlineExceedError exception.
I already am utilising the ndb.multi_put_async. I don't know what else I could do to speed this up?
def post(self):
arguments = self.reqparser.parse_args()
json_records = arguments.get('records')
user = User.query(User.email == request.authorization.username).get()
if user:
records_put_list = []
events_put_list = []
for json_record in json_records:
record_id = json_record['record_id']
rec = Record.get_or_insert(record_id,
user=user.key,
record_date=record_date,
timestamp=record_timestamp)
for json_event in json_record['events']:
event = Event.get_or_insert(json_event['event_id'],
parent=rec.key,
user=user.key,
is_deleted=json_event['is_deleted'])
if event.timestamp < json_event['timestamp']:
event.user = user.key
event.record = rec.key
event.date_time = event_dt
event.timestamp = json_event['timestamp']
event.parent = rec.key
events_put_list.append(event)
ndb.put_multi_async(records_put_list)
ndb.put_multi_async(events_put_list)
return '', 201
else:
return '', 401
This is the exception. Any advice what I could do?
Traceback (most recent call last):
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 266, in Handle
result = handler(dict(self._environ), self._StartResponse)
File "/base/data/home/apps/s~feeltracker1/1-0-1-0.376950929493492366/lib/flask/app.py", line 1836, in __call__
return self.wsgi_app(environ, start_response)
File "/base/data/home/apps/s~feeltracker1/1-0-1-0.376950929493492366/lib/flask/app.py", line 1817, in wsgi_app
response = self.full_dispatch_request()
File "/base/data/home/apps/s~feeltracker1/1-0-1-0.376950929493492366/lib/flask/app.py", line 1475, in full_dispatch_request
rv = self.dispatch_request()
File "/base/data/home/apps/s~feeltracker1/1-0-1-0.376950929493492366/lib/flask/app.py", line 1461, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/base/data/home/apps/s~feeltracker1/1-0-1-0.376950929493492366/lib/flask_restful/__init__.py", line 397, in wrapper
resp = resource(*args, **kwargs)
File "/base/data/home/apps/s~feeltracker1/1-0-1-0.376950929493492366/lib/flask/views.py", line 84, in view
return self.dispatch_request(*args, **kwargs)
File "/base/data/home/apps/s~feeltracker1/1-0-1-0.376950929493492366/lib/flask_restful/__init__.py", line 487, in dispatch_request
resp = meth(*args, **kwargs)
File "/base/data/home/apps/s~feeltracker1/1-0-1-0.376950929493492366/application/http_basic_auth.py", line 36, in decorated
return f(*args, **kwargs)
File "/base/data/home/apps/s~feeltracker1/1-0-1-0.376950929493492366/application/rest_api_view.py", line 362, in post
is_deleted=json_event['is_deleted'])
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/model.py", line 3401, in _get_or_insert
return cls._get_or_insert_async(*args, **kwds).get_result()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 325, in get_result
self.check_success()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 320, in check_success
self.wait()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 304, in wait
if not ev.run1():
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/eventloop.py", line 235, in run1
delay = self.run0()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/eventloop.py", line 197, in run0
callback(*args, **kwds)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 474, in _on_future_completion
self._help_tasklet_along(ns, ds_conn, gen, val)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 371, in _help_tasklet_along
value = gen.send(val)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/context.py", line 1015, in transaction
ok = yield tconn.async_commit(options)
DeadlineExceededError

Answer 1

You are not using the right logic. Do not put the get_or_insert in the loop - that is never going to work for anything except the smallest number of records (and is generally an anti-pattern to avoid). Instead create the entities and append them to the list inside the loop. Once this is done, the put_multi will then take this list and handle all the puts in parallel. There is a question about how many items in a put_multi list is tolerable, but 350 is far from what I remember as a limit.

Answer 2

You can use a Task API to split this processing job into smaller batches (e.g. 100 entries per task - you can find the optimum number).