I have a python program in Google App Engine
When finding an object in the datastore when I have the key as a string, how can I do a direct read. Below is my code which is performing a loop, not good.....
class Opportunity(db.Model):
customer = db.ReferenceProperty(Customer,collection_name='opportunitys')
BNusername = db.StringProperty()
opportunity_no = db.StringProperty()
# etc etc etc.....
#BnPresets holds the object key as a string
opportunitys = Opportunity.all()
opportunitys.filter('BNusername =',BnPresets.myusername)
for oprec in opportunitys:
if str(oprec.key()) == BnPresets.recordkey:
opportunity = oprec
# I have the object here and can process etc etc
You can instantiate db.Key from string by passing it directly to the constructor:
opportunity_key = db.Key(BnPresets.recordkey)
Once you have that, simply db.get to obtain the entity identified by this key:
opportunity = db.get(opportunity_key)
I guess (by looking at the query you use) that you also want to verify the username of the object you got:
if opportunity.BNusername == BnPresets.myusername
process_opportunity(opportunity)
That should be pretty much it. The bottom line is that you should use the key first - as it uniquely identifies your entity - rather than querying for some other property and iterating through results.
Related
I'm having trouble using get_by_id in google app engine python ndb.
Python Code
Attempt 1
resource = Content.get_by_id(6093630880088064)
resource is None
Attempt 2
resource = Content.get_by_id(6093630880088064, parent = 5249205949956096)
BadValueError: Expected Key instance, got 5249205949956096L
Attempt 3
key_parent = ndb.Key('Subject', '5249205949956096')
resource = Content.get_by_id(6093630880088064, parent = key_parent)
resource is None
Datastore
Entity Kind Content
Entity Key ahBkZXZ-YnJhaW5ib290ZWNocigLEgdTdWJqZWN0GICAgICAxKkJDAsSB0NvbnRlbnQYgICAgIDE6QoM
ID 6093630880088064
Parent ahBkZXZ-YnJhaW5ib290ZWNochQLEgdTdWJqZWN0GICAgICAxKkJDA
Subject:id=5249205949956096
Any suggestions would be greatly appreciated. My goal is that resource will be an object where I can do something like resource.name to retrieve the name property.
This problem occur when you have the key as string, the simplest solution is:
Model.get_by_id(int(id))
This problem is most likely to appear is when you read the id from url and the like:
class MyHandler(webapp2.RequestHandler):
def get(self, id):
instance = Model.get_by_id(int(id))
Solved :)
key = int(key)
key_parent = ndb.Key(Subject, 5249205949956096)
resource = Content.get_by_id(key, parent = key_parent)
Attempt 1:
You get None because no instance exists with this ID. Either the ID is wrong or it's a string not an integer like you’re passing to the function.
Attempt 2:
get_by_id actually expects the parent before the ID (if the parent is passed). If you wanna keep that order then use
resource = Content.get_by_id(id = 6093630880088064, parent = parent_key)
More importantly, the parent argument is the key of the parent not its ID.
Attempt 3:
In your code the ID of the parent is passed as a string and that of the content is passed as an integer. I'm guessing that the problem lies there. You pass either as strings of integers.
Hope that helps.
I want to scan all records to check if there is not errors inside data.
How can I disable BadValueError to no break scan on lack of required field?
Consider that I can not change StringProperty to not required and such properties can be tenths in real code - so such workaround is not useful?
class A(db.Model):
x = db.StringProperty(required = True)
for instance in A.all():
# check something
if something(instance):
instance.delete()
Can I use some function to read datastore.Entity directly to avoid such problems with not need validation?
The solution I found for this problem was to use a resilient query, it ignores any exception thrown by a query, you can try this:
def resilient_query(query):
query_iter = iter(query)
while True:
next_result = query_iter.next()
#check something
yield next_result
except Exception, e:
next_result.delete()
query = resilient_query(A.query())
If you use ndb, you can load all your models as an ndb.Expando, then modify the values. This doesn't appear to be possible in db because you cannot specify a kind for a Query in db that differs from your model class.
Even though your model is defined in db, you can still use ndb to fix your entities:
# Setup a new ndb connection with ndb.Expando as the default model.
conn = ndb.make_connection(default_model=ndb.Expando)
# Use this connection in our context.
ndb.set_context(ndb.make_context(conn=conn))
# Query for all A kinds
for a in ndb.Query(kind='A'):
if a.x is None:
a.x = 'A more appropriate value.'
# Re-put the broken entity.
a.put()
Also note that this (and other solutions listed) will be subject to whatever time limits you are restricted to (i.e. 60 seconds on an App Engine frontend). If you are dealing with large amounts of data you will most likely want to write a custom map reduce job to do this.
Try setting a default property option to some distinct value that does not exist otherwise.
class A(db.Model):
x = db.StringProperty(required = True, default = <distinct value>)
Then load properties and check for this value.
you can override the _check_initialized(self) method of ndb.Model in your own Model subclass and replace the default logic with your own logic (or skip altogether as needed).
It might be the most dumb question and my apologies for the same but I am confused
I have the following entity:
class Profile(ndb.Model):
name = ndb.StringProperty()
identifier = ndb.StringProperty()
pic = ndb.BlobKeyProperty() # stores the key to the profile picture blob
I want to delete the "pic" property value of the above entity so that it should look as fresh as if "pic" was never assigned any value. I do not intend to delete the complete entity. Is the below approach correct:
qry = Profile.query(Profile.identifier==identifier)
result_record_list = qry.fetch()
if result_record_list:
result_record_list[0].pic.delete() # or result_record_list[0].pic = none # or undefined or null
I am deleting the actual blob referred by this blob key separately
assign None to it and put it back to the datastore.
result_record_list[0].pic = None
result_record_list[0].put()
The datastore is an OO schemaless databse. So you can add and remove properties from the the Kind (ndb.Model) without the need of a schema update.
If you also want to cleanup the entities look at this anwser from Guido
I have a latency problem in my application due to the datastore doing additional queries for referenced entities. I have received good advice on how to handle this for single value properties by the use of the get_value_for_datastore() function. However my application also have one-to many relationships as shown in the code below, and I have not found a way to prefetch these entities. The result is an unacceptable latency when trying to show a table of 200 documents and their associated documentFiles (>6000ms).
(There will probably never be more than 10.000 Documents or DocumentFiles)
Is there a way to solve this?
models.py
class Document(db.Expando):
title = db.StringProperty()
lastEditedBy = db.ReferenceProperty(DocUser, collection_name = 'documentLastEditedBy')
...
class DocUser(db.Model):
user = db.UserProperty()
name = db.StringProperty()
hasWriteAccess= db.BooleanProperty(default = False)
isAdmin = db.BooleanProperty(default = False)
accessGroups = db.ListProperty(db.Key)
...
class DocumentFile(db.Model):
description= db.StringProperty()
blob = blobstore.BlobReferenceProperty()
created = db.DateTimeProperty() # needs to be stored here in relation to upload / download of everything
document = db.ReferenceProperty(Document, collection_name = 'files')
#property
def link(self):
return '%s' % (self.key().id(),self.blob.filename)
...
main.py
docUsers = DocUser.all()
docUsersNameDict = dict([(i.key(), i.name) for i in docUsers])
documents = Document.all()
for d idocuments:
out += '<td>%s</td>' % d.title
docUserKey = Document.lastEditedBy.get_value_for_datastore(d)
out +='<td>%s</td>' % docUsersNameDict.get(docUserKey)
out += '<td>'
# Creates a new query for each document, resulting in unacceptable latency
for file in d.files:
out += file.link + '<br>'
out += '</td>'
Denormalize and store the link in your Document, so that getting the link will be fast.
You will need to be careful that when you update a DocumentFile, you need to update the associated Document. This operates under the assumption that you read the link from the datastore far more often than you update it.
Denormalizing is often the fix for poor performance on App Engine.
Load your files asynchronously. Use get_value_for_datastore on d.files, which should return a collection of keys, which you can then do db.get_async(key) to return a future object. You will not be able to write out your result procedurally as you have done, but it should be trivial to assemble a partial request / dictionary for all documents, with a collection of pending future gets(), and then when you do your iteration to build the results, you can finalize the futures, which will have finished without blocking {~0ms latency}.
Basically, you need two iterations. The first iteration will go through and asynchronously request the files you need, and the second iteration will go through, finalize your gets, and build your response.
https://developers.google.com/appengine/docs/python/datastore/async
Was wondering if I'm unconsciously using the Put method in my last line of code ( Please have a look). Thanks.
class User(db.Model):
name = db.StringProperty()
total_points = db.IntegerProperty()
points_activity_1 = db.IntegerProperty(default=100)
points_activity_2 = db.IntegerProperty(default=200)
def calculate_total_points(self):
self.total_points = self.points_activity_1 + self.points_activity_2
#initialize a user ( this is obviously a Put method )
User(key_name="key1",name="person1").put()
#get user by keyname
user = User.get_by_key_name("key1")
# QUESTION: is this also a Put method? It worked and updated my user entity's total points.
User.calculate_total_points(user)
While that method will certainly update the copy of the object that is in-memory, I do not see any reason to believe that the change will be persisted to the the datastore. Datastore write operations are costly, so they are not going to happen implicitly.
After running this code, use the datastore viewer to look at the copy of the object in the datastore. I think that you may find that it does not have the changed total_point value.