I'm struggling with a KeyProperty query, and can't see what's wrong.
My model is
class MyList(ndb.Model):
user = ndb.KeyProperty(indexed=True)
status = ndb.BooleanProperty(default=True)
items = ndb.StructuredProperty(MyRef, repeated=True, indexed=False)
I create an instance of MyList with the appropriate data and can run the following properly
cls = MyList
lists = cls.query().fetch()
Returns
[MyList(key=Key('MyList', 12), status=True, items=..., user=Key('User', 11))]
But it fails when I try to filter by user, i.e. finding lists where the user equals a particular entity; even when using the one I've just used for insert, or from the previous query result.
key = lists[0].user
lists = cls.query(cls.user=key).fetch()
Returns
[]
But works fine with status=True as the filter, and I can't see what's missing?
I should add it happens in a unit testing environment with the following v3_stub
self.policy = datastore_stub_util.PseudoRandomHRConsistencyPolicy(probability=0)
self.testbed.init_datastore_v3_stub(
require_indexes=True,
root_path="%s/../"%(os.path.dirname(__file__)),
consistency_policy=self.policy
)
user=Key('User', 11) is a key to a different class: User. Not MyList
Perhaps you meant:
user = ndb.KeyProperty(kind='User', indexed=True)
Your code looks fine, but I have noticed some data integrity issues when developing locally with NDB. I copied your model and code, and I also got the empty list at first, but then after a few more attempts, the data is there.
Try it a few times?
edit: possibly related?
google app engine ndb: put() and then query(), there is always one less item
Related
I am trying to write an app engine application for my university. What I am trying to achieve right now, is to create a method which takes in a Course name, and returns a list of all the CourseYears (think of that as being like a link table e.g. if Maths is the course, and it has Year 1, year 2 and Year 3; MathsYear1, MathsYear2 and MathsYear3 would be the names of the CourseYears).
This is the code for the module (WARING: super dirty code below!):
#ApiMethod(name = "courseYears")
public ArrayList<CourseYear> courseYears(#Named("name") String name){
DatastoreService datastore = DatastoreServiceFactory.getDatastoreService();
Query.Filter keyFilter = new Query.FilterPredicate("name", Query.FilterOperator.EQUAL, name);
Query query = new Query("Course").setFilter(keyFilter);
PreparedQuery preparedQuery = datastore.prepare(query);
List<Entity> resultList = preparedQuery.asList(FetchOptions.Builder.withLimit(1));
Course course = ofy().load().type(Course.class).id(resultList.get(0).getKey().getId()).now();
ArrayList<String> courseYearNames = course.getAllCourseYearNames();
System.out.println(course.getName());
ArrayList<CourseYear> courseYears = new ArrayList<CourseYear>();
for(String courseYearName: courseYearNames){
Query.Filter courseNameFilter = new Query.FilterPredicate("name", Query.FilterOperator.EQUAL, courseYearName);
Query query2 = new Query("CourseYear").setFilter(courseNameFilter);
List<Entity> resL = preparedQuery.asList(FetchOptions.Builder.withLimit(1));
System.out.println("test");
CourseYear courseYear = ofy().load().type(CourseYear.class).id(resL.get(0).getKey().getId()).now();
courseYears.add(courseYear);
}
return courseYears;
}
It basically takes a Course name in, applies a filter on all courses to get the corresponding Course object, and then calls getAllCourseYearNames() on the course to get an array list containing all its CourseYears' names. (I would have loved to do this using Keys, but parameterised Objectify keys don't seem to be supported in this version of App Engine).
I then try and get the CourseYears by looping through the arraylist of names and applying the filter for each name. I print "test" each time to see how many times it is looping. Like I said, a super dirty way of doing it.
When I try passing a few course names as a parameters, it loops the correct number of times only once or twice, and after that does not loop at all (doesn't print "test"). I could understand if it never looped, but not doing it correctly once or twice and then never again. It doesn't successfully return a list of CourseYears when it does work, but rather the relevant number of NULLs - I don't know if this is relevant. I believe it successfully retrieves the course every time, as I print the name of the course after loading and it never fails to do this.
If anyone has ANY suggestions for why this may be happening, I would be incredibly grateful to hear them!
Thanks
query2 is never used in your code. You reuse preparedQuery from your previous query, which runs on a different entity kind.
I want to scan all records to check if there is not errors inside data.
How can I disable BadValueError to no break scan on lack of required field?
Consider that I can not change StringProperty to not required and such properties can be tenths in real code - so such workaround is not useful?
class A(db.Model):
x = db.StringProperty(required = True)
for instance in A.all():
# check something
if something(instance):
instance.delete()
Can I use some function to read datastore.Entity directly to avoid such problems with not need validation?
The solution I found for this problem was to use a resilient query, it ignores any exception thrown by a query, you can try this:
def resilient_query(query):
query_iter = iter(query)
while True:
next_result = query_iter.next()
#check something
yield next_result
except Exception, e:
next_result.delete()
query = resilient_query(A.query())
If you use ndb, you can load all your models as an ndb.Expando, then modify the values. This doesn't appear to be possible in db because you cannot specify a kind for a Query in db that differs from your model class.
Even though your model is defined in db, you can still use ndb to fix your entities:
# Setup a new ndb connection with ndb.Expando as the default model.
conn = ndb.make_connection(default_model=ndb.Expando)
# Use this connection in our context.
ndb.set_context(ndb.make_context(conn=conn))
# Query for all A kinds
for a in ndb.Query(kind='A'):
if a.x is None:
a.x = 'A more appropriate value.'
# Re-put the broken entity.
a.put()
Also note that this (and other solutions listed) will be subject to whatever time limits you are restricted to (i.e. 60 seconds on an App Engine frontend). If you are dealing with large amounts of data you will most likely want to write a custom map reduce job to do this.
Try setting a default property option to some distinct value that does not exist otherwise.
class A(db.Model):
x = db.StringProperty(required = True, default = <distinct value>)
Then load properties and check for this value.
you can override the _check_initialized(self) method of ndb.Model in your own Model subclass and replace the default logic with your own logic (or skip altogether as needed).
I have several entities that I am searching across that include dates, and the Search API works great across all of them except for one thing - sorting.
Here's the data model for one of my entities (simplified of course):
class DepositReceipt(ndb.Expando):
#Sets creation date
creation_date = ndb.DateTimeProperty(auto_now_add=True)
And the code to create the search.Document where de is an instance of the entity:
document = search.Document(doc_id=de.key.urlsafe(),
fields=[search.TextField(name='deposit_key', value=de.key.urlsafe()),
search.DateField(name='created', value=de.creation_date),
search.TextField(name='settings', value=de.settings.urlsafe()),
])
This returns a valid document.
And finally the problem line. I took this snippet from the official GAE Search API tutorial and just changed the direction of the sort to DESCENDING and changed the search expression to created (the date property from the Document above).
expr_list = [search.SortExpression(
expression="created", default_value='',
direction=search.SortExpression.DESCENDING)]
I don't think this is important, but the rest of the search code looks like this:
sort_opts = search.SortOptions(expressions=expr_list)
query_options = search.QueryOptions(
cursor=query_cursor,
limit=_NUM_RESULTS,
sort_options=sort_opts)
query_obj = search.Query(query_string=query, options=query_options)
search_results = search.Index(name=index_name).search(query=query_obj)
In production, I get this error message:
InvalidRequest: Failed to parse search request "settings:ag5zfmdoaWRvbmF0aW9uc3IQCxIIU2V0dGluZ3MYmewDDA"; failed to parse date
Changing the expression="created" to anything else works perfectly fine. This also happens across my other entity types that use dates, so I have no idea what's going on. Advice?
I think default_value needs to be a valid date, rather than '' as you have it.
I'm trying to retrieve a user object from a foreign key reference but each time I try to do so nothing gets returned...
My table is set up like this:
FBUserID long,
UserID uniqueidentifier
so I have my repository try to get the User when it's provided the FBUserID:
public User getUserByFBuid(long uid)
{
User fbUser = null;
IEnumerable<FBuid> fbUids = _myEntitiesDB.FBuidSet.Where(user => user.FBUserID == uid);
fbUser = fbUids.FirstOrDefault().aspnet_Users;
return fbUser;
}
I've checked that the uid (FBUserID) passed in is correct, I've check that the UserID is matched up to the FBUserID. And I've also checked to make sure that fbUids.Count() > 0...
I've returned fbUids.FirstOrDefault().FBUserID and gotten the correct FBUserID, but any time I try to return the aspnet_Users or aspnet_Users.UserName, etc... I don't get anything returned. (I'm guessing it's getting an error for some reason)
I don't have debugging set up properly so that's probably why i'm having so much troubles... but so far all the checking I've done I've been doing return this.Json(/* stuff returned form the repository */) so that I can do an alert when it gets back to the javascript.
Anyone know why I would have troubles retrieving the user object from a foreign key relationship like that?
Or do you have any suggestions as to finding out what's wrong?
For now, with Entity Framework 1, you don't get automatic delayed loading, e.g. if you want to traverse from one entity to the next, you need to either do an .Include("OtherEntity") on your select to include those entities in the query, or you need to explicitly call .Load("OtherEntity") on your EntityContext to load that entity.
This was a design decision by the EF team not to support automagic deferred loading, since they considered it to be too dangerous; they wanted to make it clear and obvious to the user that he is also including / loading a second set of entities.
Due to high popular demand, the upcoming EF v4 (to be released with .NET 4.0 sometime towards the end of 2009) will support the automatic delayed loading - if you wish to use it. You need to explicitly enable it since it's off by default:
context.ContextOptions.DeferredLoadingEnabled = true;
See some articles on that new feature:
A Look at Lazy Loading in EF4
POCO Lazy Loading
Don't know if this is what you are asking but i do a join like so;
var votes = from av in dc.ArticleVotes
join v in dc.Votes on av.voteId equals v.id
where av.articleId == articleId
select v;
Did this help or am I off base?
Was wondering if I'm unconsciously using the Put method in my last line of code ( Please have a look). Thanks.
class User(db.Model):
name = db.StringProperty()
total_points = db.IntegerProperty()
points_activity_1 = db.IntegerProperty(default=100)
points_activity_2 = db.IntegerProperty(default=200)
def calculate_total_points(self):
self.total_points = self.points_activity_1 + self.points_activity_2
#initialize a user ( this is obviously a Put method )
User(key_name="key1",name="person1").put()
#get user by keyname
user = User.get_by_key_name("key1")
# QUESTION: is this also a Put method? It worked and updated my user entity's total points.
User.calculate_total_points(user)
While that method will certainly update the copy of the object that is in-memory, I do not see any reason to believe that the change will be persisted to the the datastore. Datastore write operations are costly, so they are not going to happen implicitly.
After running this code, use the datastore viewer to look at the copy of the object in the datastore. I think that you may find that it does not have the changed total_point value.