TransactionFailedError on GAE when no transaction - google-app-engine

Question

I got this error:
TransactionFailedError: too much contention on these datastore entities. please try again.
Even though I'm not doing any transactions. The line of my code that causes the error is
ndb.put_multi(entity_list) # entity_list is a list of 100 entities
This error doesn't happen often so it isn't a big deal, but I'm curious why I get this error. Any ideas?
Here is most of the traceback:
Traceback (most recent call last):
...
File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/ext/deferred/deferred.py", line 318, in post
self.run_from_request()
File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/ext/deferred/deferred.py", line 313, in run_from_request
run(self.request.body)
File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/ext/deferred/deferred.py", line 155, in run
return func(*args, **kwds)
File "/base/data/home/apps/s~opavote/2017-09-15.404125237783169549/tasks.py", line 70, in start_election
models.Voter.create(e.eid, chunk)
File "/base/data/home/apps/s~opavote/2017-09-15.404125237783169549/models.py", line 2426, in create
ndb.put_multi(voters + vbs)
File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/ext/ndb/model.py", line 3958, in put_multi
for future in put_multi_async(entities, **ctx_options)]
File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 383, in get_result
self.check_success()
File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 427, in _help_tasklet_along
value = gen.throw(exc.__class__, exc, tb)
File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/ext/ndb/context.py", line 824, in put
key = yield self._put_batcher.add(entity, options)
File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 427, in _help_tasklet_along
value = gen.throw(exc.__class__, exc, tb)
File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/ext/ndb/context.py", line 358, in _put_tasklet
keys = yield self._conn.async_put(options, datastore_entities)
File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 513, in _on_rpc_completion
result = rpc.get_result()
File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/datastore/datastore_rpc.py", line 928, in get_result
result = rpc.get_result()
File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 613, in get_result
return self.__get_result_hook(self)
File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/datastore/datastore_rpc.py", line 1893, in __put_hook
self.check_rpc_success(rpc)
File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/datastore/datastore_rpc.py", line 1385, in check_rpc_success
raise _ToDatastoreError(err)
TransactionFailedError: too much contention on these datastore entities. please try again.

Answer 1

Note that the error is actually received from the datastore itself, in the RPC response: self.check_rpc_success(rpc).
Which makes me suspect that on the datastore side, to ensure operation consistency/reliability across the redundant pieces of infra supporting it, every write operation is actually using the same/similar mechanisms as for transactional operations. The difference would be that those also have some transactional checks on the client side, before/after the RPC exchange and maybe explicit RPC transaction start/end triggers for the datastore.
From Life of a Datastore Write, a quote suggesting that some common mechanisms are being used regardless of the operations being transactional or not (emphasis mine):
If the commit phase has succeeded but the apply phase failed, the
datastore will roll forward to apply the changes to indexes under two
circumstances:
The next time you execute a read or write or start a transaction on this entity group, the datastore will first roll
forward and fully apply this committed but unapplied write, based on
the data in the log.
And one of the possible reasons for failures would be simply too many parallel accesses to the same entities, even if they're just read-only. See Contention problems in Google App Engine, though in that case they're for transactions on the client side.
Note that this is just a theory ;)

Answer 2

It might be worth re-reviewing transactions and entity groups, noting the various definitions and limits.
Putting "Every attempt to create, update, or delete an entity takes place in the context of a transaction," and, "There is a write throughput limit of about one transaction per second within a single entity group," probably speaks to what you're seeing, particularly if entity_list contains entities that would fall into the same entity group.