When using the endpoints-proto-datastore query_method on an EndpointsModel, is it possible to have an inequality filter passed as a query field?
For instance, say I have a model with a lastmodified field, and I want to query for all the records that have a lastmodified > date, where date is a value passed to the API by the client.
class MyModel(EndpointsModel):
attr1 = ndb.StringProperty()
lastmodified = ndb.DateTimeProperty()
From the documentation I have seen, it seems like the query_fields is strictly an equality filter.
Yes, but you'll need to use an EndpointsAliasProperty with a name different than the lastmodified field. For example, you could do something like was done in the Picturesque app used as a sample at Google I/O 2013.
First, define your model
from google.appengine.ext import ndb
from endpoints_proto_datastore.ndb import EndpointsModel
class MyModel(EndpointsModel):
attr1 = ndb.StringProperty()
lastmodified = ndb.DateTimeProperty()
then create an alias property which will be used to update the query
from google.appengine.ext import endpoints
from endpoints_proto_datastore.ndb import EndpointsAliasProperty
...
#EndpointsAliasProperty(name='modifiedSince', setter=ModifiedSinceSet)
def ModifiedSince(self):
raise endpoints.BadRequestException(
'modifiedSince value should never be accessed.')
This property won't ever be accessed, but its setter is relevant to make the query sort on what we want, so we need to define ModifiedSinceSet (it needs to come before the variable is reference, as in the sample):
import datetime
from endpoints_proto_datastore import utils
...
def ModifiedSinceSet(self, value):
try:
modified_since = utils.DatetimeValueFromString(value)
if not isinstance(modified_since, datetime.datetime):
raise TypeError('Not a datetime stamp.')
except TypeError:
raise endpoints.BadRequestException('Invalid timestamp for modifiedSince.')
self._endpoints_query_info._filters.add(
MyModel.lastmodified >= modified_since)
Here we convert the passed in value (from the request) to a datetime.datetime object and then use it to add a filter to self._endpoints_query_info._filters on the current EndpointsModel object.
This _endpoints_query_info is of course used in query_method to create the query that gets passed to your method.
Another alternative:
You could just parse the value and use introduce your own syntax. For example, accept strings like lastmodified>=TIMESTAMP and then parse the timestamp from the statement.
Related
I recently encountered a situation where one might want to run a datastore query which includes a kind, but the class of the corresponding model is not available (e.g. if it's defined in a module that hasn't been imported yet).
I couldn't find any out-of-the-box way to do this using the google.appengine.ext.db package, so I ended up using the google.appengine.api.datastore.Query class from the low-level datastore API.
This worked fine for my needs (my query only needed to count the number of results, without returning any model instances), but I was wondering if anyone knows of a better solution.
Another approach I've tried (which also worked) was subclassing db.GqlQuery to bypass its constructor. This might not be the cleanest solution, but if anyone is interested, here is the code:
import logging
from google.appengine.ext import db, gql
class ClasslessGqlQuery(db.GqlQuery):
"""
This subclass of :class:`db.GqlQuery` uses a modified version of ``db.GqlQuery``'s constructor to suppress any
:class:`db.KindError` that might be raised by ``db.class_for_kind(kindName)``.
This allows using the functionality :class:`db.GqlQuery` without requiring that a Model class for the query's kind
be available in the local environment, which could happen if a module defining that class hasn't been imported yet.
In that case, no validation of the Model's properties will be performed (will not check whether they're not indexed),
but otherwise, this class should work the same as :class:`db.GqlQuery`.
"""
def __init__(self, query_string, *args, **kwds):
"""
**NOTE**: this is a modified version of :class:`db.GqlQuery`'s constructor, suppressing any :class:`db.KindError`s
that might be raised by ``db.class_for_kind(kindName)``.
In that case, no validation of the Model's properties will be performed (will not check whether they're not indexed),
but otherwise, this class should work the same as :class:`db.GqlQuery`.
Args:
query_string: Properly formatted GQL query string.
*args: Positional arguments used to bind numeric references in the query.
**kwds: Dictionary-based arguments for named references.
Raises:
PropertyError if the query filters or sorts on a property that's not indexed.
"""
from google.appengine.ext import gql
app = kwds.pop('_app', None)
namespace = None
if isinstance(app, tuple):
if len(app) != 2:
raise db.BadArgumentError('_app must have 2 values if type is tuple.')
app, namespace = app
self._proto_query = gql.GQL(query_string, _app=app, namespace=namespace)
kind = self._proto_query._kind
model_class = None
try:
if kind is not None:
model_class = db.class_for_kind(kind)
except db.KindError, e:
logging.warning("%s on %s without a model class", self.__class__.__name__, kind, exc_info=True)
super(db.GqlQuery, self).__init__(model_class)
if model_class is not None:
for property, unused in (self._proto_query.filters().keys() +
self._proto_query.orderings()):
if property in model_class._unindexed_properties:
raise db.PropertyError('Property \'%s\' is not indexed' % property)
self.bind(*args, **kwds)
(also available as a gist)
You could create a temporary class just to do the query. If you use an Expando model, the properties of the class don't need to match what is actually in the datastore.
class KindName(ndb.Expando):
pass
You could then do:
KindName.query()
If you need to filter on specific properties, then I suspect you'll have to add them to the temporary class.
I currently have a model in NDB and I'd like to add a new property to it. Let's say I have the following:
class User(Model, BaseModel):
name = ndb.StringProperty(required=False)
email = ndb.StringProperty(required=False)
#property
def user_roles(self):
return UserRole.query(ancestor=self.key).fetch()
#property
def roles(self):
return [user_role.role for user_role in UserRole.query(ancestor=self.key).fetch()]
Now, let's say, I've added one additional property called market_id. For example,
class User(Model, BaseModel):
name = ndb.StringProperty(required=False)
email = ndb.StringProperty(required=False)
#property
def user_roles(self):
return UserRole.query(ancestor=self.key).fetch()
#property
def roles(self):
return [user_role.role for user_role in UserRole.query(ancestor=self.key).fetch()]
#property
def market_id(self):
""" fetches `id` for the resource `market` associated with `user` """
for each_role in UserRole.query(ancestor=self.key):
resource = each_role.role.get().resource
if resource.kind() == 'Market':
return resource.id()
return None
The problem here is, roles are fetched properly as expected for all the existing entities (since that property had been there since the beginning and also, an extra column can be observed in datastore called roles).
Since, I'm dealing with Python class property, I assume that migration is not required. But, how does column called roles already exist? And why newly added property called market_id does not? Does it require migration?
The change you're suggesting is not an actual ndb model change as you're not adding/deleting/modifying any of the model's datastore properties. Only ndb.Property class children are real ndb model properties that are stored when the entity is put() into the datastore.
The property you're adding is a Python class #property - nothing to do with what's in the datastore.
So for this particular case no migration is needed.
The update to the question makes this even more clear, I believe. The market_id #property is not a User datastore entity property. To get values for it you don't need to update the User entity, but you have to create/edit corresponding UserRole entities with their resource property point to a Market entity.
How does one query for computed properties?
class MyModel(EndpointsModel):
attr1 = ndb.IntegerProperty(default=0)
#EndpointsComputedProperty(property_type=messages.BooleanField)
def attr2(self):
return self.attr1 % 2 == 1
#endpoints.api(name='myapi', version='v1', description='My Little API')
class MyApi(remote.Service):
#MyModel.query_method(query_fields=('attr2'),
path='mymodels', name='mymodel.list')
def MyModelList(self, query):
return query
In this case query will always have a filter that will test for attr2 == False.
The cause seems to be that the filters are created from an entity created with FromMessage. As attr2 is a computed property, it cannot be set. As attr1 defaults to 0, attr2 is always False regardless of what is passed in.
You can query for computed properties by setting up a placeholder alias property similar to how inequality queries can be achieved, as described here: https://stackoverflow.com/a/17288934/4139124
More specifically, we set up an alias property with a different name from our existing attr1 and attr2 fields. We also define a setter that gets called whenever the alias property is assigned to, and use this setter to modify the query. Note that the setter must appear above the alias property:
import logging
from google.appengine.ext import ndb
from endpoints_proto_datastore.ndb import EndpointsModel
from endpoints_proto_datastore.ndb.properties import EndpointsAliasProperty
from endpoints_proto_datastore.ndb.properties import EndpointsComputedProperty
from protorpc import messages
class MyModel(EndpointsModel):
attr1 = ndb.IntegerProperty(default=0)
#EndpointsComputedProperty(property_type=messages.BooleanField)
def attr2(self):
return self.attr1 % 2 == 1
def Attr2AliasSetter(self, value):
self._endpoints_query_info._filters.add(MyModel.attr2 == value)
#EndpointsAliasProperty(name='attr2_alias',
property_type=messages.BooleanField,
setter=Attr2AliasSetter)
def Attr2Alias(self):
logging.error('attr2_alias should never be accessed')
We then update the query_method to accept the attr2_alias field rather than attr2:
#endpoints.api(name='myapi', version='v1', description='My Little API')
class MyApi(remote.Service):
#MyModel.query_method(query_fields=('attr2_alias'),
path='mymodels', name='mymodel.list')
def MyModelList(self, query):
return query
Then, for example, querying for myapi/mymodels?attr2_alias=true will return all MyModel entities with attr2 set to true.
Context
I have the following data structure:
class Birthday(ndb.Model):
day = ndb.IntegerProperty()
month = ndb.IntegerProperty()
year = ndb.IntegerProperty()
class User(ndb.Model):
name = ndb.StringProperty()
birthday = ndb.StructuredProperty(Birthday)
# ... other properties
Problem
When I try to use the populate() method on an instance of User, it gives an error: expecting a Birthday instance instead of a dictionary of params.
If I remove the birthday property, everything works fine: the User instance is populated with the dictionary of params.
Shouldn't the populate() method recognize structured properties and automatically populate them as well?
Any clues?
Thanks
PS: The populate method could also use a forgiving mode on which it ignores unknown properties for which there are references on the params dictionary.
>>Added comments
I'm using a generic REST Handler which is extended for accessing and changing several data types. The extension has to define a method getModel() that returns the model class to access/manipulate. The model class has to implement a few methods, namely create(cls, params).
The POST handler parses params (sent by AngularJS using $resouce -- link below) the following way:
# inside the generic REST Handler
params = json.loads(self.request.body, object_hook=self.datetime_decoder) # parse json params
...
self.getModel().create(params) # invokes the create method of the
The model class implements the create method the following way:
#classmethod
def create(cls, params = None):
obj = cls()
if params:
obj.update(**params)
obj.put()
return True, obj
return False, None
The contents of the JSON dict are:
{"name":"Ana Matos","email":"ana.matos#nvd.com","phone":"+35196983465671","birthday":{"day":1,"month":0,"year":1980},"gender":"FEMALE","groups":["2012/2013"],"serviceProviderId":206133}
JSON contens -- firefox screenshot
AngularJS $resource
Are you reporting a bug or requesting a feature? The populate() method requires its parameter types to match the declared type of the property, which in this case is a Birthday instance.
It would help if you showed the contents of the JSON dict that you are passing to populate() (and exactly how you are passing it).
Possibly the solution is as simple as getting the 'birthday' value from the JSON dict and using it to create a Birthday instance. But I would have to see your code to know for sure.
In my Google App Engine app I have model objects that need to be stored. These objects are parameterized by various policy objects. For example, my Event class has a Privacy policy object which determines who can see, update, etc. There are various subclasses of PrivacyPolicy that behave differently. The Event consults its PrivacyPolicy object at various points.
class PrivacyPolicy(db.Model):
def can_see(self, event, user):
pass
class OwnerOnlyPolicy(PrivacyPolicy):
def can_see(self, event, user):
return user == event.owner
class GroupOnlyPolicy(PrivacyPolicy):
def can_see(self, event, user):
for grp in event.owner.groups()
if grp.is_member(user):
return True
return False
class OnlyCertainUsersPolicy(PrivacyPolicy):
def __init__(self, others):
self.others = others
def can_see(self, event, user):
return user in others
I could make my Event class use a ReferenceProperty to the PrivacyPolicy:
class Event(db.Model):
privacy: db.ReferenceProperty(PrivacyPolicy)
#…
The reason I don’t like this is that the one-to-one relationship means that nobody every queries for the policy object, there is no need to maintain the back-reference from the policy to its Event object, and in no other way is PrivacyPolicy an independent db-level object. It is functionally equivalent to an IntegerProperty, in that it is part of the Event object’s state, it’s just an object instead of a number — specifically it’s an object that can have zero state or lots of state, unknown to the Event type.
I can’t find anyone talking about how to approach such a situation. Is there a tool/approach I don’t know about? Do I just suck it up and use a reference property and the hell with the overhead?
If the only other way to handle this is a custom Property type, any advice about how to approach it would be welcome. My first thought is to use a TextProperty to store the string rep of the policy object (policy), decode it when needed, caching the result, and having any change to the policy object invalidate the cache and update the string rep.
You're overcomplicating by trying to store this in the datastore. This belongs in code rather than in the datastore.
The least complicated way would be:
class Event(db.Model):
privacy = db.IntegerProperty()
def can_see(self, user):
if self.privacy == PRIVACY_OWNER_ONLY:
return user == event.owner
else if self.privacy == PRIVACY_GROUP:
for grp in self.owner.groups()
if grp.is_member(user):
return True
return False
Sometimes all it takes is to think of the right approach. The solution is to introduce a new kind of property that uses pickle to store and retrieve values, such as that described in https://groups.google.com/forum/?fromgroups#!topic/google-appengine/bwMD0ZfRnJg
I wanted something slightly more sophisticated, because pickle isn’t always the answer, and anyway documentation is nice, so here is my ObjectReference type:
import pickle
from google.appengine.ext import db
class ObjectProperty(db.Property):
def __init__(self, object_type=None, verbose_name=None, to_store=pickle.dumps, from_store=pickle.loads, **kwds):
"""Initializes this Property with all the given options
All args are passed to the superclass. The ones used specifically by this class are described here. For
all other args, see base class method documentation for details.
Args:
object_type: If not None, all values assigned to the property must be either instances of this type or None
to_store: A function to use to convert a property value to a storable str representation. The default is
to use pickle.dumps()
from_store: A function to use to convert a storable str representation to a property value. The default is
to use pickle.loads()
"""
if object_type and not isinstance(object_type, type):
raise TypeError('object_type should be a type object')
kwds['indexed'] = False # It never makes sense to index pickled data
super(ObjectProperty, self).__init__(verbose_name, **kwds)
self.to_store = to_store
self.from_store = from_store
self.object_type = object_type
def get_value_for_datastore(self, model_instance):
"""Get value from property to send to datastore.
We retrieve the value of the attribute and return the result of invoking the to_store function on it
See base class method documentation for details.
"""
value = getattr(model_instance, self.name, None)
return self.to_store(value)
def make_value_from_datastore(self, rep):
"""Get value from datastore to assign to the property.
We take the value passed, convert it to str() and return the result of invoking the from_store function
on it. The Property class assigns this returned value to the property.
See base class method documentation for details.
"""
# It passes us a unicode, even though I returned a str, so this is required
rep = str(rep)
return self.from_store(rep)
def validate(self, value):
"""Validate reference.
Returns:
A valid value.
Raises:
BadValueError for the following reasons:
- Object not of correct type.
"""
value = super(ObjectProperty, self).validate(value)
if value is not None and not isinstance(value, self.object_type):
raise db.KindError('Property %s must be of type %s' % (self.name, self.object_type))
return value