I have NDB model class like below:
class Contest(ndb.Model):
end_date = ndb.DateProperty(required=True, indexed=True)
ended = ndb.BooleanProperty(required=False, indexed=True)
...
I will have a daily cron job to mark contests with passed end_date with ended equal to True. I've written the following code to do it:
contests = Contest.query()
current_datetime = datetime.datetime.utcnow()
today = datetime.date(current_datetime.year, current_datetime.month, current_datetime.day)
contests = contests.filter(Contest.end_date < today)
contests = contests.filter(Contest.ended == False)
contests = contests.fetch(limit=10)
for contest in contests:
contest.ended = True
ndb.put_multi(contests)
But I don't like it, since I have to read all entities just to update one value. Is there any way to modify it to read keys_only?
The object data overwrites the existing entity. The entire object is sent to Datastore
https://cloud.google.com/datastore/docs/concepts/entities#Datastore_Updating_an_entity
So you cannot send only one field of an entity, it will "remove" all existing fields. To be more accurate - replace an entity with all fields with new version of entity that have only one field.
You have to load all entities you want to update, with all properties, not just keys, set new value of a property, and put back into database.
I think a Python property is a good solution here:
class Contest(ndb.Model):
end_date = ndb.DateProperty(required=True, indexed=True)
#property
def ended(self):
return self.end_date < date.today()
This way you don't ever need to update your entities. The value is automatically computed whenever you need it.
Related
I am using the google cloud datastore python client to write an entity into the datastore which contains an embedded entity. An example entity might look like:
data_type: 1
raw_bytes: <unindexed blob>
values: <indexed embedded entity>
I checked the data from the console and the data is getting saved correctly and the values are present.
Next, I need to run a query from a python app engine application. I have represented the above as the following entity in my app engine code:
class DataValues(ndb.Model):
param1 = ndb.BooleanProperty()
param2 = ndb.IntegerProperty()
param3 = ndb.IntegerProperty()
class MyEntity(ndb.Expando):
data_type = ndb.IntegerProperty(required=True)
raw_bytes = ndb.BlobProperty()
values = ndb.StructuredProperty(DataValues)
One of the filters in the query depends on a property in values. Sample query code is as below:
MyEntity.query().filter(MyEntity.data_type == 1).filter(MyEntity.values.param1 == True).get()
I have created the corresponding composite index in my index.yaml
The query runs successfully but the resulting entity contains the embedded entity values as None. All other property values are present.
What can be the issue here ?
Add properties of DataValues entity as properties of the MyEntity.
This is a bit of a guess, but since datastore attributes are kind of keyed by both their name (in this case values) and the name of the "field type/class" (i.e. StructuredProperty), this might fix your problem:
class EmbeddedProperty(ndb.StructuredProperty):
pass
class MyEntity(ndb.Expando):
data_type = ndb.IntegerProperty(required=True)
raw_bytes = ndb.BlobProperty()
values = EmbeddedProperty(DataValues)
Give it a shot and let me know if values starts coming back non-null.
I struggled with the same problem, wanting to convert the embedded entity into a Python dictionary. One possible solution, although not a very elegant one, is to use a GenericProperty:
class MyEntity(ndb.Model):
data_type = ndb.IntegerProperty(required=True)
raw_bytes = ndb.BlobProperty()
values = ndb.GenericProperty()
values will then be read as an "Expando" object: Expando(param1=False,...). You can access the individual values with values.param1, values.param2 etc. I would prefer having a custom model class, but this should do the job.
I had NDB class before and added new property receive_news just now:
class User(ndb.Model):
'''
Index
Key: user_id
'''
lang = ndb.StringProperty(required=None, indexed=True)
receive_news = ndb.BooleanProperty(required=None, indexed=True)
I would like to get list of users, who would like to receive my news (all users currently). I tried the following options:
keys = User.query(User.receive_news != True).fetch(keys_only=True)
keys = User.query(User.receive_news == None).fetch(keys_only=True)
both returns 0. How should I work properly with this new property?
Datastore indexes are only updated when entities are being written to the datastore, so only entities written after the new index was created will be added to it.
To have the pre-existing entities added to the index (so that they can be found by the query) you'll have to get and then re-write them. Maybe using something along these lines (you'll have to split it in separate requests/tasks if they are too many)
keys = []
for user in ndb.get_multi(User.query().fetch(keys_only=True)):
if user.receive_news is None:
user.receive_news = True
keys.append(user.key)
ndb.put_multi(keys)
Can I compare two properties of single kind in ndb of GAE?
class GameLog(ndb.Model):
duration = ndb.IntegerProperty(default=0)
time = ndb.IntegerProperty(default=0)
I need compare these two properties. How can I do this?
GameLog.duration > GameLog.time
To accomplish this kind of thing you need to save the (precomputed) result, so it's indexed and you can query for it.
To make this easier ndb gives you Computed Properties:
class GameLog(ndb.Model):
duration = ndb.IntegerProperty(default=0)
time = ndb.IntegerProperty(default=0)
timeout = ndb.ComputedProperty(lambda self: self.duration > self.time)
You don't need to maintain this property yourself, every time you put() the entity the value will get calculated and saved. Now you can do the query:
GameLog.query(GameLog.timeout == True)
I am building an order system on Google App Engine.
Suppose I have the following model of Order:
class Order(ndb.Model):
time: DateTimeProperty()
The date plus a serial number as a string is used as the id of one entity:
today = str(datetime.datetime.now())[:10].replace('-','')
# Fetch today's orders
orders = Order.query(...).order(-Order.time).fetch(1)
if len(orders)==0: # No entities: today's first order
orderNum = today + '00001'
else:
orderNum = str(int(orders[0].key.id())+1)
order = Order(id=orderNum)
order.date = datetime.datetime.now()
order.put()
Suppose there are multiple clerks and they may execute the program at the same time. An obvious problem is that they may obtain the same orderNum and actually write to the same entity.
How do I do to prevent such situation from happening?
You could use something similar to the code below.
#ndb.transactional(retries=3)
def create_order_if_not_exists(order):
exists = order.key.get()
if exists is not None:
raise Exception("Order already exists: ID=%s" % order.key.id())
order.put()
Or you can use Model's class method get_or_insert() that transactionally retrieves an existing entity or creates a new one. See details here.
I want to check if db.Model is changed or not. How to do it?
Some example year ago I designed such model and saved:
class DeleteMe(db.Model):
a = db.StringProperty()
key = db.Key().from_path('DeleteMe', 'a')
a = DeleteMe(key = key)
After year I was add some new and change default property:
class DeleteMe(db.Model):
a = db.StringProperty(default = 'zero')
b = db.StringProperty(default = 'cat')
key = db.Key().from_path('DeleteMe', 'a')
a = DeleteMe.get(key)
When I am reading model a == None and b == cat but in datastore b == None how can I check if b == None in datastore?
I used undocumented ea = a._entity.get('b') to check if ea != a.a what is alternetive documented method.
You can't normally check if b == None with db or ndb.
What you have suggested is probably the best approach. As you have found the underlying raw entity doesn't even have the b property so the default value kicks in. Additionally there is also no index for you can query.
Normally when handling schema migration you have to visit every entity and update it.
In some cases you can do this lazily (ie only update when you next fetch the entity) then your method would be used to establish the actual value or nonexistence of b and if the entity needs updating.
You could use a version property in the entity to help manage complex schema updates.
With a version property defined it wouldn't matter if b == "cat" because you would know if you need to write back/update the object based on the version of the entities schema.