I am trying to solve the racing problem based on this to prevent duplicate user registrations. So if the account exists or the email has been used, no entity will be created.
#ndb.transactional
def get_or_insert2(account, email):
accountExists, emailExists = False, False
entity = Member.get_by_id(account)
if entity is not None:
accountExists = True
if Member.query(Member.email==email).fetch(1):
emailExists = True
if not accountExists and not emailExists:
entity = Member(id=account)
entity.put()
return (entity, accountExists, emailExists)
My questions:
I got an error message: BadRequestError: Only ancestor queries are allowed inside transactions. what was the problem?
Is the code correct? I mean, can it really solve the racing problem?
Thanks.
Transactions work on entity groups, and you can include up to 5 entity groups in a cross group transaction. An entity group is handled by a single server (or group, replicated), which means it is able to have consistent internal state when checking data or doing ancestor queries within the entity group.
Regular queries are global, on indexes with eventual consistency. You don't know when all changes from all nodes have been included in an index. You can't lock up the entire datastore to get consistent snapshot state for your transaction. This is a key difference from a regular RDBMS if you're used to consistent index for queries.
For 1), the problem is that you're doing a regular query inside a transaction, which doesn't work as explained above. The answer to 2) then becomes no, query can't solve racing problem, you need explicit gets.
You will need a Model for Member, Email and SSN. This is a quick untested example that hopefully gets you going:
class Member(ndb.Model):
email = ndb.KeyProperty()
ssn = ndb.KeyProperty()
# More user properties goes here...
class Email(ndb.Model):
member = ndb.KeyProperty()
class SSN(ndb.Model):
member = ndb.KeyProperty()
#ndb.tasklet
def get_or_insert2(account, email, ssn):
created = False
member_key = ndb.Key(Member, account)
email_key = ndb.Key(Email, email)
ssn_key = ndb.Key(SSN, ssn)
member_obj, email_obj, ssn_obj = yield ndb.get_multi_async([member_key, email_key, ssn_key])
if member_obj is None and email_obj is None and ssn_obj is None:
member_obj = Member(key=member_key, email=email_key, ssn=ssn_key))
email_obj = Email(key=email_key, member=member_key)
ssn_obj = SSN(key=ssn_key, member=member_key)
yield ndb.put_multi_async([member_obj, email_obj])
created = True
raise ndb.Return([created, member_obj, email_obj, ssn_obj])
outcome = ndb.transaction(lambda: get_or_insert2(account, email, ssn), xg=True)
I'm not sure if it works to combine #ndb.tasklet and #ndb.transactional(xg=True) decorators, and if so, which order, just try it out.
If you need to query User based on email or ssn, you could for example rename the KeyProperties to *_ref and make something like
#ndb.ComputedProperty
def email(self):
return self.email_ref.id()
While this ends up being more lines of code than you anticipated, it is conceptually simple and straight forward, and you can easily figure out what's going on when you get back to it later.
Related
An NDB model contains two properties: email and password. How to avoid adding to the database two records with the same email? NDB doesn't have UNIQUE option for a property, like relational databases do.
Checking that new email is not in the database before adding—won't satisfy me, because two parallel processes can both simultaneously do the checking and each add the same email.
I'm not sure that transactions can help here, I am under this impression after reading some of the manuals. Maybe the synchronous transactions? Does it mean one at a time?
Create the key of the entity by email, then use get_or_insert to check if exists.
Also read about keys , entities. and models
#ADD
key_a = ndb.Key(Person, email);
person = Person(key=key_a)
person.put()
#Insert unique
a = Person.get_or_insert(email)
or if you want to just check
#ADD
key_a = ndb.Key(Person, email);
person = Person(key=key_a)
person.put()
#Check if it's added
new_key_a =ndb.Key(Person, email);
a = new_key_a.get()
if a is not None:
return
Take care. Changing email will be really difficult (need to create new entry and copy all entries to new parent).
For that thing maybe you need to store the email, in another entity and have the User be the parent of that.
Another way is to use Transactions and check the email property. Transaction's work in the way: First that commits is the First that wins. A concept which means that if 2 users check for email only the first (lucky) one will succeed, thus your data will be consistent.
Maybe you are looking for the webapp2-authentication module, that can handle this for you. It can be imported like this import webapp2_extras.appengine.auth.models. Look here for a complete example.
I also ran into this problem, and the solution above didn't solve my problem:
making it a key was unacceptable in my case (i need the property to be changeable in the future)
using transactions on the email property doesn't work AFAIK (you can't do queries on non-key names inside transactions, so you can't check whether the e-mail already exists).
I ended up creating a separate model with no properties, and the unique property (email address) as the key name. In the main model, I store a reference to the email model (instead of storing the email as a string). Then, I can make 'change_email' a transaction that checks for uniqueness by looking up the email by key.
This is something that I've come across as well and I settled on a variation of #Remko's solution. My main issue with checking for an existing entity with the given email is a potential race condition like op stated. I added a separate model that uses an email address as the key and has a property that holds a token. By using get_or_insert, the returned entities token can be checked against the token passed in and if they match then the model was inserted.
import os
from google.appengine.ext import ndb
class UniqueEmail(ndb.Model):
token = ndb.StringProperty()
class User(ndb.Model):
email = ndb.KeyProperty(kind=UniqueEmail, required=True)
password = ndb.StringProperty(required=True)
def create_user(email, password):
token = os.urandom(24)
unique_email = UniqueEmail.get_or_insert(email,
token=token)
if token == unique_email.token:
# If the tokens match, that means a UniqueEmail entity
# was inserted by this process.
# Code to create User goes here.
# The tokens do not match, therefore the UniqueEmail entity
# was retrieved, so the email is already in use.
raise ValueError('That user already exists.')
I implemented a generic structure to control unique properties. This solution can be used for several kinds and properties. Besides, this solution is transparent for other developers, they use NDB methods put and delete as usual.
1) Kind UniqueCategory: a list of unique properties in order to group information. Example:
‘User.nickname’
2) Kind Unique: it contains the values of each unique property. The key is the own property value which you want to control of. I save the urlsafe of the main entity instead of the key or key.id() because is more practical and it doesn’t have problem with parent and it can be used for different kinds. Example:
parent: User.nickname
key: AVILLA
reference_urlsafe: ahdkZXZ-c3RhcnQtb3BlcmF0aW9uLWRldnINCxIEVXNlciIDMTIzDA (User key)
3) Kind User: for instance, I want to control unique values for email and nickname. I created a list called ‘uniqueness’ with the unique properties. I overwritten method put in transactional mode and I wrote the hook _post_delete_hook when one entity is deleted.
4) Exception ENotUniqueException: custom exception class raised when some value is duplicated.
5) Procedure check_uniqueness: check whether a value is duplicated.
6) Procedure delete_uniqueness: delete unique values when the main entity is deleted.
Any tips or improvement are welcome.
class UniqueCategory(ndb.Model):
# Key = [kind name].[property name]
class Unique(ndb.Model):
# Parent = UniqueCategory
# Key = property value
reference_urlsafe = ndb.StringProperty(required=True)
class ENotUniqueException(Exception):
def __init__(self, property_name):
super(ENotUniqueException, self).__init__('Property value {0} is duplicated'.format(property_name))
self. property_name = property_name
class User(ndb.Model):
# Key = Firebase UUID or automatically generated
firstName = ndb.StringProperty(required=True)
surname = ndb.StringProperty(required=True)
nickname = ndb.StringProperty(required=True)
email = ndb.StringProperty(required=True)
#ndb.transactional(xg=True)
def put(self):
result = super(User, self).put()
check_uniqueness (self)
return result
#classmethod
def _post_delete_hook(cls, key, future):
delete_uniqueness(key)
uniqueness = [nickname, email]
def check_uniqueness(entity):
def get_or_insert_unique_category(qualified_name):
unique_category_key = ndb.Key(UniqueCategory, qualified_name)
unique_category = unique_category_key.get()
if not unique_category:
unique_category = UniqueCategory(id=qualified_name)
unique_category.put()
return unique_category_key
def del_old_value(key, attribute_name, unique_category_key):
old_entity = key.get()
if old_entity:
old_value = getattr(old_entity, attribute_name)
if old_value != new_value:
unique_key = ndb.Key(Unique, old_value, parent=unique_category_key)
unique_key.delete()
# Main flow
for unique_attribute in entity.uniqueness:
attribute_name = unique_attribute._name
qualified_name = type(entity).__name__ + '.' + attribute_name
new_value = getattr(entity, attribute_name)
unique_category_key = get_or_insert_unique_category(qualified_name)
del_old_value(entity.key, attribute_name, unique_category_key)
unique = ndb.Key(Unique, new_value, parent=unique_category_key).get()
if unique is not None and unique.reference_urlsafe != entity.key.urlsafe():
raise ENotUniqueException(attribute_name)
else:
unique = Unique(parent=unique_category_key,
id=new_value,
reference_urlsafe=entity.key.urlsafe())
unique.put()
def delete_uniqueness(key):
list_of_keys = Unique.query(Unique.reference_urlsafe == key.urlsafe()).fetch(keys_only=True)
if list_of_keys:
ndb.delete_multi(list_of_keys)
Do I pay a penalty on query performance if I choose to query repeated property? For example:
class User(ndb.Model):
user_name = ndb.StringProperty()
login_providers = ndb.KeyProperty(repeated=true)
fbkey = ndb.Key("ProviderId", 1, "ProviderName", "FB")
for entry in User.query(User.login_providers == fbkey):
# Do something with entry.key
vs
class User(ndb.Model)
user_name = ndb.StringProperty()
class UserProvider(ndb.Model):
user_key = ndb.KeyProperty(kind=User)
login_provider = ndb.KeyProperty()
for entry in UserProvider.query(
UserProvider.user_key == auserkey,
UserProvider.login_provider == fbkey
):
# Do something with entry.user_key
Based on the documentation from GAE, it seems that Datastore take care of indexing and the first less verbose option would be using the index. However, I failed to find any documentation to confirm this.
Edit
The sole purpose of UserProvider in the second example is to create a one-to-many relationship between a user and it's login_provider. I wanted to understand if it worth the trouble of creating a second entity instead of querying on repeated property. Also, assume that all I need is the key from the User.
No. But you'll raise your write costs because each entry needs to be indexed, and write costs are based on the number of indexes updated.
I recently came across a number of articles pointing out to flatten the data for NoSQL databases. Coming from traditional SQL databases I realized I am replicating a SQL db bahaviour in GAE. So I started to refactor code where possible.
We have e.g. a social media site where users can become friends with each other.
class Friendship(ndb.Model):
from_friend = ndb.KeyProperty(kind=User)
to_friend = ndb.KeyProperty(kind=User)
Effectively the app creates a friendship instance between both users.
friendshipA = Friendship(from_friend = UserA, to_friend = userB)
friendshipB = Friendship(from_friend = UserB, to_friend = userA)
How could I now move this to the actual user model to flatten it. I thought maybe I could use a StructuredProperty. I know it is limited to 5000 entries, but that should be enough for friends.
class User(UserMixin, ndb.Model):
name = ndb.StringProperty()
friends = ndb.StructuredProperty(User, repeated=True)
So I came up with this, however User can't point to itself, so it seems. Because I get a NameError: name 'User' is not defined
Any idea how I could flatten it so that a single User instance would contain all its friends, with all their properties?
You can't create a StructuredProperty that references itself. Also, use of StructuredProperty to store a copy of User has additional problem of needing to perform a manual cascade update if a user ever modifies a property that is stored.
However, as KeyProperty accept String as kind, you can easily store the list of Users using KeyProperty as suggested by #dragonx. You can further optimise read by using ndb.get_multi to avoid multiple round-trip RPC calls when retrieving friends.
Here is a sample code:
class User(ndb.Model):
name = ndb.StringProperty()
friends = ndb.KeyProperty(kind="User", repeated=True)
userB = User(name="User B")
userB_key = userB.put()
userC = User(name="User C")
userC_key = userC.put()
userA = User(name="User A", friends=[userB_key, userC_key])
userA_key = userA.put()
# To retrieve all friends
for user in ndb.get_multi(userA.friends):
print "user: %s" % user.name
Use a KeyProperty that stores the key for the User instance.
I'm trying to wrap my head around eventuality consistency and 1 write per sec principles in GAE datastore. I have a scenario and two questions:
#python like pseudo-code
class User:
user_id = StringProperty
last_update_time = DateTimeProperty
class Comment:
user_id = StringProperty
comment = StringProperty
...
def AddCommentAndReturnAllComments(user_id):
user = db.GqlQuery("SELECT * FROM User where user_id = :1", user_id)
user.last_update_time = datetime.now()
user.put()
comment = Comment(parent=User(user_id))
comment.put()
comments = db.GqlQuery("SELECT * FROM Comment where user_id = :1", user_id)
return comments
Questions:
Will I get an exception here because I make two writes into the same EntityGroup within one second (user.put and comment.put)? Is there a simple way around it?
If I remove the parent=user(user_id), the two entities will no longer belong to the same EntityGroup. Does it mean that the list of comments returned from the function might not contain the last added comment?
Am I doing something inherently wrong?
I know that I got the entity referencing part wrong. It doesn't matter for the question (or does it?)
This seems to be a soft limit. In practice I see up to 5 writes/s allowed.
Yes and it also happens now, because you are not using ancestor query.
Nothing, except as mentioned in point 2.
I know this is a classical problem, but I still don't know how to do it. On Google App Engine, I have a member registration form which uses jQuery's validation to check if a username exists.
There of course is a concurrency problem: several users try to register, enter the same username, Validation finds the username available, and allow them to press "Add" at the approximately same time. Validation wouldn't detect this. In my application, username, email, and Personal ID should all be unique. How do I prevent the following code from having the concurrency problem:
member = Member()
member.username = self.request.get('username')
member.Pid = self.request.get('Pid')
member.email = self.request.get('email')
...
As the uniqueness constraint is on username, you have to use it as key in datastore and use transactions.
def txn():
key = ndb.Key(Member, username)
member = key.get()
if member is not None:
raise CustomAlreadyExistsException(member) # This will abort txn
member = Member(
id=username,
Pid=self.request.get('Pid'),
email=self.request.get('email'),
...)
member.put()
ndb.transaction(txn)
This makes sure only one person can register a username.
The jQuery helper would check if ndb.Key(Member, userid).get() gives a result or not. The GET is not transactional.
To improve usability client side in "reserving" a username after checking availability, you could use memcached as suggested by Daniel, but I'd call YAGNI, skip the complexity and rather let some people get validation error after submitting the form. Note that memcached is best effort and has no guarantees about anything.
If you need guaranteed uniqueness on multiple fields, you have to add Model classes for them and check in a cross group (XG) transaction.
class Pid(ndb.Model):
member = ndb.KeyProperty()
class Email(ndb.Model):
member = ndb.KeyProperty()
class Member(ndb.Model):
pid = ndb.KeyProperty()
email = ndb.KeyProperty()
#property
def pid_value(self):
return self.pid.id()
#property
def email_value(self):
return self.email.id()
def txn():
member_key = ndb.Key(Member, username)
pid_key = ndb.Key(PersonalId, self.request.get('Pid'))
email_key = ndb.Key(Email, self.request.get('email'))
member, pid, email = ndb.get_multi([member_key, pid_key, email_key])
if member is not None or pid is not None or email is not None:
raise CustomAlreadyExistsException(member, pid, email) # This will abort txn
# Create instances referencing each other
email = Email(key=email_key, member=member_key)
pid = Pid(key=pid_key, member=member_key)
member = Member(
key=member_key,
pid=pid_key,
email=email_key,
...)
ndb.put_multi([member, pid, email])
ndb.transaction(txn, xg=True)
This is a great use for memcache. Your Ajax validation function should put an entry into memcache to record that the username has been requested. It should also check both memcache and the datastore to ensure that the username is free. Similarly, the registration code should check memcache to ensure that the current user is the one who requested the username.
This nicely solves your concurrency problem, and the best thing is that entries in memcache expire by themselves, either on a timed basis or when the cache gets too full.
I agreed with tesdal.
If you still want to implement the memcache tric sugested by Daniel, you should do something like "memcache.add(usernameA, dummy value, short period);". So you know that usernameA is reserved for a short period and wont conflict with "memcache.add(usernameB, ..."