This weird case happened twice already in the last 2 days.
I used Datastore Admin to remove all entities, no more than 100, to later re–upload db using remote_api_shell but after the request the Datastore Write Operations reached the limit:
This is the first and the only operation I did since last 24h reset.
Also the error is reported in remote_api_shell when I try to put new entities.
Any advice welcome.
Edit:
Here the models, nothing huge...
class Brand(BaseModel):
'''
Brand class
`Marca` in Etax db
'''
name = db.StringProperty()
abbr = db.StringProperty()
def __repr__(self):
return ('<Brand {0} instance at {1}>'
.format(self.abbr.encode('utf-8'), hex(id(self))))
class Model(BaseModel):
'''
Model class
`Gamma` in Etax db
'''
name = db.StringProperty()
code = db.IntegerProperty()
brand = db.ReferenceProperty(Brand, collection_name='models')
def __repr__(self):
return ('<Model {0} instance at {1}>'
.format(self.code, hex(id(self))))
class TrimLevel(BaseModel):
'''
Trim Level class
`Modello` in Etax db
'''
name = db.StringProperty()
etax_code = db.IntegerProperty()
start_production_date = db.DateProperty()
end_production_date = db.DateProperty()
retail_buy_prices = db.ListProperty(int)
retail_sell_prices = db.ListProperty(int)
list_prices = db.ListProperty(int)
model = db.ReferenceProperty(Model, collection_name='trim_levels')
fuel_supply = db.StringProperty()
gear_shift = db.StringProperty()
gear_speeds = db.IntegerProperty()
doors = db.IntegerProperty()
seats = db.IntegerProperty()
kw = db.IntegerProperty()
def __repr__(self):
return ('<TrimLevel {0} instance at {1}>'
.format(self.etax_code, hex(id(self))))
If you look at billing docs, that a high-level delete takes several low-level write operations:
Entity Delete (per entity): 2 writes + 2 writes per indexed property value + 1 write per composite index value
So if 100 entity deletes used 50k write ops, it means that your every entity had 500 index entries.
This can happen when entity has large list properties or havs a compound index spanning multiple list properties (= exploding index)
Do you have any compound indexes defined? What properties does your entity have?
Related
I would like to create a site that helps users to remember meaning of certain words.
class Word(models.Model):
word = models.CharField(max_length=30, unique=True)
meaning = models.CharField(max_length=200)
memory_strength = models.FloatField()
user = models.ForeignKey(User)
I want each user to have individual (unique) value of memory_strength for every item of Word, while values of word and meaning would be the same for each and every user. How can I achieve that?
class Word(models.Model):
word = models.CharField(max_length=30, unique=True)
meaning = models.CharField(max_length=200)
class Memory(models.Model):
memory_strength = models.FloatField()
user = models.ForeignKey(User)
word = models.ForeignKey(Word)
I have a GAE database entity that looks like this:
class Notification(db.Model):
alert = db.StringProperty()
type = db.StringProperty()
status = db.StringProperty(default="unread", choices=set(["unread", "read"]))
created = db.DateTimeProperty(auto_now_add=True)
modified = db.DateTimeProperty(auto_now=True)
entity = db.StringProperty()
record = db.ReferenceProperty(model.RecordModel)
actor = db.ReferenceProperty(model.Profile)
account = db.ReferenceProperty(model.Account)
... and I create an entity like so:
notify = model2.Notification(account=account)
notify.alert = message
notify.type = "reminder"
notify.actor = actor
notify.record = record
notify.put()
This call raises an error *'Notification' object has no attribute '_key'*
nquery = db.Query(model2.Notification).filter('account =', self.session.account).order('-created')
for n in nquery:
try:
_dict = {}
_dict['serverID'] = str(n.key()) #- raises error!
try:
nquery = Notification.all().filter('account =', self.session.account).order('-created')
I think I've figured it out! The "entity" property in my Notification class is causing some sort of naming conflict in python appengine. Changing the name removes the "error object has no attribute '_key'" error. Go figure!
I have a simple ndb model as follows:
class TaskList(ndb.Model):
title = ndb.StringProperty()
description = ndb.TextProperty()
priority = ndb.IntegerProperty()
class UserProfile(ndb.Model):
user = ndb.UserProperty()
tasks = ndb.KeyProperty(TaskList)
As known, a TaskList object will have an Entity Kind Entity Key and an ID.
Given an ID say 7.
I can very well get the object with ID 7 as follows:
task = ndb.Key(TaskList, 7).get()
But how do i get a user who has the task ID 7?
I tried:
tsk = ndb.Key(TaskList, 7).get()
user = UserProfile.query(UserProfile.tasks == tsk.key)
It works, but is there a better method?
You're nearly there - but there's no need to fetch the task, just to use its key property again:
task_key = ndb.Key(TaskList, 7)
user = UserProfile.query(UserProfile.tasks == task_key)
or equivalently:
user = UserProfile.query(UserProfile.tasks == ndb.Key(TaskList, 7))
I have a model with a lot of properties and when I try to write it I get a warning on the logs that the request is taking too much CPU times (over 1000cpu_ms and over 8000api_cpu_ms).
The following test program demonstrates the issue:
class TestModel(db.Model):
user = db.UserProperty()
p1 = db.StringProperty()
p2 = db.StringProperty()
p3 = db.StringProperty()
p4 = db.StringProperty()
p5 = db.StringProperty() # 475cpu_ms with 1..5 only
p6 = db.StringProperty()
p7 = db.StringProperty()
p8 = db.StringProperty()
p9 = db.StringProperty()
p10 = db.StringProperty() # 760cpu_ms with 1..10 only
p11 = db.StringProperty()
p12 = db.StringProperty()
p13 = db.ListProperty(unicode)
p14 = db.ListProperty(int)
p15 = db.ListProperty(int)
p16 = db.BooleanProperty()
p17 = db.DateTimeProperty() #over 1000cpu_ms with all these
class DebugitHandler(webapp.RequestHandler):
def get(self):
instance = TestModel()
instance = instance.put()
self.response.out.write("Done.")
All the CPU time is spent on that one .put() call.
Other than splitting the model into more than one (which, probably won't help as I would still have the same number of properties) is there a way to get around this problem? a way to put 1 instance with 20 properties in the datastore without taking over 1 second of CPU time?
As stated in the quota documentation:
As the number of properties associated with a given entity increases,
so does the CPU time required to read and write that entity
If you don't need to sort or filter on each of these properties, you could add the indexed = False declaration to some of them; this should save some CPU time..
class TestModel(db.Model):
user = db.UserProperty(indexed = False)
p1 = db.StringProperty(indexed = False)
p2 = db.StringProperty(indexed = False)
p3 = db.StringProperty(indexed = False)
....
So i have two model classes:
class Dog(db.model):
dogName = StringProperty()
dogBreed = StringProperty()
class Cat(db.model):
catName = StringProperty()
catBreed = StringProperty()
and then i have a third model class to hold all the pictures
class Images(db.model):
imageReference = ReferenceProperty(*Animal*, collection_name = 'allImages')
imageURL = StringProperty()
Animal is either a Dog or a Cat. Obviously this does not compile.
Now my question is: Is there a way I can put Cat pictures in with Dog pictures? Or do I need to create more models like this:
class DogImages(db.model):
imageReference = ReferenceProperty(Dog, collection_name = 'allImages')
imageURL = StringProperty()
class CatImages(db.model):
imageReference = ReferenceProperty(Cat, collection_name = 'allImages')
imageURL = StringProperty()
You could use PolyModel:
class Animal(polymodel.PolyModel):
name = db.StringProperty()
breed = db.StringProperty()
class Dog(Animal):
pass
class Cat(Animal):
pass
Now you can have a ReferenceProperty that references Animals, and either Dogs or Cats will be permitted.
However, you don't have any properties that are specific to each type of animal - why not just have a regular Animal model, add a property indicating what species it is, and skip the separate models entirely?