I am using the google cloud datastore python client to write an entity into the datastore which contains an embedded entity. An example entity might look like:
data_type: 1
raw_bytes: <unindexed blob>
values: <indexed embedded entity>
I checked the data from the console and the data is getting saved correctly and the values are present.
Next, I need to run a query from a python app engine application. I have represented the above as the following entity in my app engine code:
class DataValues(ndb.Model):
param1 = ndb.BooleanProperty()
param2 = ndb.IntegerProperty()
param3 = ndb.IntegerProperty()
class MyEntity(ndb.Expando):
data_type = ndb.IntegerProperty(required=True)
raw_bytes = ndb.BlobProperty()
values = ndb.StructuredProperty(DataValues)
One of the filters in the query depends on a property in values. Sample query code is as below:
MyEntity.query().filter(MyEntity.data_type == 1).filter(MyEntity.values.param1 == True).get()
I have created the corresponding composite index in my index.yaml
The query runs successfully but the resulting entity contains the embedded entity values as None. All other property values are present.
What can be the issue here ?
Add properties of DataValues entity as properties of the MyEntity.
This is a bit of a guess, but since datastore attributes are kind of keyed by both their name (in this case values) and the name of the "field type/class" (i.e. StructuredProperty), this might fix your problem:
class EmbeddedProperty(ndb.StructuredProperty):
pass
class MyEntity(ndb.Expando):
data_type = ndb.IntegerProperty(required=True)
raw_bytes = ndb.BlobProperty()
values = EmbeddedProperty(DataValues)
Give it a shot and let me know if values starts coming back non-null.
I struggled with the same problem, wanting to convert the embedded entity into a Python dictionary. One possible solution, although not a very elegant one, is to use a GenericProperty:
class MyEntity(ndb.Model):
data_type = ndb.IntegerProperty(required=True)
raw_bytes = ndb.BlobProperty()
values = ndb.GenericProperty()
values will then be read as an "Expando" object: Expando(param1=False,...). You can access the individual values with values.param1, values.param2 etc. I would prefer having a custom model class, but this should do the job.
Related
I am trying to make a python program that uses the google cloud data store in python and i am having some trouble with the DataStore input\output system. this is my first time working with google cloud, and i am also somewhat new to python.
I am trying to build a very simple database, with only 1 type of entity model- 'Command', which has 2 variables- 'name' which i want to be the key, and 'value'. all the entities will have one parent, because the google cloud guide said this will put all the entities in the same entity group, which helps sort them? (i am not sure about this, so explanation will be nice)
class Command(ndb.Model):
value = ndb.IntegerProperty()
# no 'name' variable, since it's the key.
def parent_key():
return ndb.Key(Command, DEFAULT_PARENT_NAME)
when the user uses a 'set' command, the code will either insert a new entity with the given name and value, or if the name exists already, it will change the existing value to the given value.
(assume 'variable_name' is the name and 'variable_value' is the value)
this is the code for the 'set' command:
variable_name = self.request.get('name')
variable_value = self.request.get('value')
newcommand = Command(id=variable_name, value=int(variable_value), parent=parent_key()) # create a new command model
newcommand.put()
this inserts a new command, but doesn't check if it is already in the datastore.
i want the 'get' command to extract the value of an existing name in the database (or return an error, if it doesn't exist), given the name (in a string)
in the online manual i found how to extract stuff from the database given a key, but here i dont have a key, i have a string.
I don't know how to complete the 'set' and 'get' commands and would appreciate some help with this.
Thanks in advance!
I'm trying to use Python and NDB to access the datastore, which contains one entity:
I've defined by NDB model with the following code:
class Test(ndb.Model):
name = ndb.StringProperty()
val = ndb.IntegerProperty()
Then I run a query for the entity:
query = Test.get_by_id("Testing")
This returns a NoneType with no val field. I tried setting the argument to name=Testing instead of Testing, but that doesn't help.
What can I do to access my entity in Python? Do I need to identify the project's ID somewhere?
Also, I've been using Flask to serve as the microframework. But all the NDB example code I've seen uses webapp2. Should I use webapp2 instead?
Capitalization matters in Python. Instead of "Test", you need to query from the model "test".
We use GoogleInfo table, in that table, we store scopes of application. e.g."https://www.googleapis.com/auth/drive.appdata,https://www.googleapis.com/auth/drive.file"
and I am trying to get a result based on some keyword like "drive" from scopes, based on that keyword I am trying to get a result. following is my code. Please suggest.
String keywords[] = {"admin","drive","gmail","userinfo"};
Query query = pm.newQuery("SELECT scopes FROM com.cloudcodes.gcontrol.dataaccesslayer.insights.google.drive.GoogleInfo where :p.contains(scopes)");
result = (List) query.execute(Arrays.asList(keywords));
List tempResult = new ArrayList();
tempResult.addAll(result);
return tempResult;
Seems that you are using JDO [1] and the PersistanceManager [2].
As far as I can see the query you are trying to execute might be wrong. Check how to perform queries with JDO [3].
Your query is:
String keywords[] = {"admin","drive","gmail","userinfo"};
Query query = pm.newQuery("SELECT scopes FROM com.cloudcodes.gcontrol.dataaccesslayer.insights.google.drive.GoogleInfo where :p.contains(scopes)");
result = (List) query.execute(Arrays.asList(keywords));
Looks like you want to do something like:
//Query for all persons with lastName equal to Smith or Jones
Query q = pm.newQuery(Person.class, ":p.contains(lastName)");
q.execute(Arrays.asList("Smith", "Jones"));
Person.class is the kind
Arrays.asList("Smith", "Jones")
This parameter ":p.contains(lastName)" that defines lastName is the property we want to check on.
You are setting the class as com.cloudcodes.gcontrol.dataaccesslayer.insights.google.drive.GoogleInfo, I suppose that this is the full Java name package where the class habits, and the class name is GoogleInfo. So you could try:
Query q = pm.newQuery(GoogleInfo.class, ":p.contains(scopes)");
q.execute(Arrays.asList("admin","drive","gmail","userinfo"));
You want to retrieve scopes. I assume that you want to use the REST API. So inside :p.contains(“scopes”) might go another property related to your keyWords that is in the Entity you want to retrieve, maybe an array property?
Here I share with you some docs that might be useful [4][5].
Hope this helps!
[1] https://cloud.google.com/appengine/docs/standard/java/datastore/jdo/overview-dn2
[2] http://massapi.com/method/javax/jdo/PersistenceManager.newQuery-4.html
[3] https://cloud.google.com/appengine/docs/standard/java/datastore/jdo/queries
[4] https://cloud.google.com/datastore/docs/concepts/overview#comparison_with_traditional_databases
[5] https://cloud.google.com/datastore/docs/reference/gql_reference
I had NDB class before and added new property receive_news just now:
class User(ndb.Model):
'''
Index
Key: user_id
'''
lang = ndb.StringProperty(required=None, indexed=True)
receive_news = ndb.BooleanProperty(required=None, indexed=True)
I would like to get list of users, who would like to receive my news (all users currently). I tried the following options:
keys = User.query(User.receive_news != True).fetch(keys_only=True)
keys = User.query(User.receive_news == None).fetch(keys_only=True)
both returns 0. How should I work properly with this new property?
Datastore indexes are only updated when entities are being written to the datastore, so only entities written after the new index was created will be added to it.
To have the pre-existing entities added to the index (so that they can be found by the query) you'll have to get and then re-write them. Maybe using something along these lines (you'll have to split it in separate requests/tasks if they are too many)
keys = []
for user in ndb.get_multi(User.query().fetch(keys_only=True)):
if user.receive_news is None:
user.receive_news = True
keys.append(user.key)
ndb.put_multi(keys)
Do I pay a penalty on query performance if I choose to query repeated property? For example:
class User(ndb.Model):
user_name = ndb.StringProperty()
login_providers = ndb.KeyProperty(repeated=true)
fbkey = ndb.Key("ProviderId", 1, "ProviderName", "FB")
for entry in User.query(User.login_providers == fbkey):
# Do something with entry.key
vs
class User(ndb.Model)
user_name = ndb.StringProperty()
class UserProvider(ndb.Model):
user_key = ndb.KeyProperty(kind=User)
login_provider = ndb.KeyProperty()
for entry in UserProvider.query(
UserProvider.user_key == auserkey,
UserProvider.login_provider == fbkey
):
# Do something with entry.user_key
Based on the documentation from GAE, it seems that Datastore take care of indexing and the first less verbose option would be using the index. However, I failed to find any documentation to confirm this.
Edit
The sole purpose of UserProvider in the second example is to create a one-to-many relationship between a user and it's login_provider. I wanted to understand if it worth the trouble of creating a second entity instead of querying on repeated property. Also, assume that all I need is the key from the User.
No. But you'll raise your write costs because each entry needs to be indexed, and write costs are based on the number of indexes updated.