I have the following model:
class Product(ndb.Model):
name = ndb.StringProperty()
bidTime = ndb.DateTimeProperty()
price = ndb.IntegerProperty()
...
I'd likd to use the following query:
productRanks = Product.query(Product.bidTime>=startDate,
Product.bidTime<endDate).order(-Product.price).fetch()
where startDate and endDate are datetime objects. But I got the following error message:
The first sort property must be the same as the property to which the inequality filter is applied
If I add Product.bidTime in the order then there will be no error:
.order(Product.bidTime, -Product.price)
However, the sorted result would be wrong (according to date, not price). So, what is the problem?
There is no problem as far as appengine is concerned. It is behaving as documented. From the docs
Note: Because of the way the App Engine Datastore executes queries, if
a query specifies inequality filters on a property and sort orders on
other properties, the property used in the inequality filters must be
ordered before the other properties.
See https://developers.google.com/appengine/docs/python/datastore/queries#Sort_Orders
You may need to sort in memory after you get your result set.
Related
I have a simple little "Observation" class:
from google.appengine.ext import ndb
class Observation(ndb.Model):
remote_id = ndb.StringProperty()
dimension_id = ndb.IntegerProperty()
metric = ndb.StringProperty()
timestamp_observed = ndb.StringProperty()
timestamp_received = ndb.DateTimeProperty(auto_now_add=True)
#classmethod
def query_book(cls):
return cls.query()
I can run projection queries against the Datastore to return only certain columns. E.g:
observations = Observation.query().fetch(projection=[Observation.dimension_id])
This works nicely, but I only want unique results. The documentation makes this sound easy:
# Functionally equivalent
Article.query(projection=[Article.author], group_by=[Article.author])
Article.query(projection=[Article.author], distinct=True)
But when I do this:
observations = Observation.query().fetch(projection=[Observation.dimension_id], group_by=[Observation.dimension_id])
observations = Observation.query().fetch(projection=[Observation.dimension_id], distinct=True)
I get errors for both variants.
TypeError: Unknown configuration option ('group_by')
TypeError: Unknown configuration option ('distinct')
This happens on localhost and in the prod environment too. What am I missing?
Silly me - all of these params need to sit within the query() function, not within fetch(). The projection elements actually works in fetch(), but you need to move both the projection and distinct arguments into query() to get it to work.
From Grouping:
Projection queries can use the distinct keyword to ensure that only
completely unique results will be returned in a result set. This will
only return the first result for entities which have the same values
for the properties that are being projected.
Article.query(projection=[Article.author], group_by=[Article.author])
Article.query(projection=[Article.author], distinct=True)
Both queries are equivalent and will produce each author's name only
once.
Hope this helps anyone else with a similar problem :)
I am trying to query if a certain date belongs to a specific range of dates. Source code example:
billing_period_found = BillingPeriod.query(
ndb.AND(
transaction.date > BillingPeriod.start_date,
transaction.date < BillingPeriod.end_date)
).get()
Data definition:
class Transaction(ndb.Model):
date = ndb.DateProperty(required=False)
class BillingPeriod(ndb.Model):
start_date = ndb.DateProperty(required=False)
end_date = ndb.DateProperty(required=False)
Getting the following error:
TypeError: can't compare datetime.date to DateProperty
The message error does make sense because datetime is different from DateProperty. However, as you can see, the definition for transaction.date is not datetime, so I am not getting where this attempt to convert datetime to date is coming from. Anyways - If I figure out how to convert datetime to DateProperty, I guess it would fix the problem.
Any ideas on how to solve this?
Thanks!
The App Engine datastore does not allow queries with inequalities on multiple properties (not a limitation of ndb, but of the underlying datastore). Selecting date-range entities that contain a certain date is a typical example of tasks that this makes it impossible to achieve in a single query.
Check out Optimizing a inequality query in ndb over two properties for an example of this question, and, in the answer, one suggestion that might work: query for (in your case) all BillingPeriod entities with end_date greater than the desired date, perhaps with a projection to just get their key and start_date; then, select out of those only those with start_date less than the desired date, in your own application (if you only want one of them, then a next over the iterator will stop as soon as it finds one).
Edit: the issue above is problem #1 with this code; once solved, problem #2 arises -- as clearly listed at https://cloud.google.com/appengine/docs/python/ndb/queries, the property is ndb queries is always on the left of the comparison operator. So, one can't do date < BillingPeriod.end_date, as that would have the property on the right; rather, one does BillingPeriod.end_date > date.
If I have a class like:
class carModel(ndb.Model):
type = ndb.StringProperty(required=True)
brand = ndb.StringProperty(required=True)
I want that a user can type a GQL query into a text field. (Same like at the Datastore viewer).
I'll get the string and insert it into my query - for example:
queryString = "carModel.brand=='bmw'"
qry = carModel.query(queryString)
result = query.get()
I get following error message:
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/query.py", line 1059, in filter
raise TypeError('Cannot filter a non-Node argument; received %r' % arg)
TypeError: Cannot filter a non-Node argument; received "carModel.brand=='bmw'"
I think that class has not been recognized from the string. Any suggestions for a solution where I can keep the users freedom to use any GQL queries on the model?
carModel.brand is actually a class and property reference not a string.
Either construct the query using GQL, or you have to assemble the query as per the Query class docs which means parsing your own set of arguments.
Be sure to use appropriate substitution rules or someone could query for things not intended and check for GQL that does not comply with what ever rules you have in place, or someone can query for anything. Remember also that indexes must exist for the queries submitted.
class MyEntity(db.Model):
timestamp = db.DateTimeProperty()
title = db.StringProperty()
number = db.FloatProperty()
db.GqlQuery("SELECT * FROM MyEntity WHERE title = 'mystring' AND timestamp >= date('2012-01-01') AND timestamp <= date('2012-12-31') ORDER BY timestamp DESC").fetch(1000)
This should fetch ~600 entities on app engine. On my dev server it behaves as expected, builds the index.yaml, I upload it, test on server but on app engine it does not return anything.
Index:
- kind: MyEntity
properties:
- name: title
- name: timestamp
direction: desc
I try splitting the query down on datastore viewer to see where the issue is and the timestamp constraints work as expected. The query returns nothing on WHERE title = 'mystring' when it should be returning a bunch of entities.
I vaguely remember fussy filtering where you had to call .filter("prop =",propValue) with the space between property and operator, but this is a GqlQuery so it's not that (and I tried that format with the GQL too).
Anyone know what my issue is?
One thing I can think of:
I added the list of MyEntity entities into the app via BulkLoader.py prior to the new index being created on my devserver & uploaded. Would that make a difference?
The last line you wrote is probably the problem.
Your entities in the actual real datastore are missing the index required for the query.
As far as I know, when you add a new index, App Engine is supposed to rebuild your indexes for you. This may take some time. You can check your admin page to check the state of your indexes and see if it's still building.
Turns out there's a slight bug in the bulkloader supplied with App Engine SDK - basically autogenerated config transforms strings as db.Text, which is no good if you want these fields indexed. The correct import_transform directive should be:
transform.none_if_empty(str)
This will instruct App Engine to index the uploaded field as a db.StringProperty().
I write these codes
Query q = pm.newQuery(User.class);
q.setFilter("textPosts != null"); // textPosts is a Text
List<User> users = (List<User>) q.execute();
resp.getWriter().println("user num: " + users.size());
And I get
user num: 0
I'm sure that the number should greater than 0.
I probably miss something important.
Thanks in advance!
Querying Unindexed properties:
A query with a filter or sort order on a property will never match an entity with that property unindexed
So your entity is not found because Text properties are never indexed.
This is also relevant (but in your case Text being unindexed prevails) - entities with non-existing properties are not found with queries:
The App Engine datastore distinguishes between an entity without a given property and an entity with a null value for a property. JDO does not support this distinction: every field of an object has a value, possibly null.
So if you have an Entity without a property, then it will not be found with a query, since a query requires that the property exists.
Maybe because Text will not be indexed for query purposes, as it specified in javadoc: http://code.google.com/intl/en/appengine/docs/java/javadoc/com/google/appengine/api/datastore/Text.html
Btw, i'm sure that it must throw exception at this case.