I'm running a Kind based Query using Java API on GAE.
Here is the sample code:
DatastoreService dataStore = DatastoreServiceFactory.getDatastoreService();
Filter value1Filter = new FilterPredicate(PROPERTY_1, FilterOperator.EQUAL, value1);
Filter value2Filter = new FilterPredicate(PROPERTY_2, FilterOperator.EQUAL, value2);
Filter myFilter = CompositeFilterOperator.and(value1Filter, value2Filter);
Query findQuery = new Query("MyKIND").setFilter(myFilter);
myEntity = dataStore.prepare(findQuery).asSingleEntity();
PROPERTY_1 and PROPERTY_2 are indexed properties. This query works some times and now fails consistently for certain values. Let's say these values are value1 and value2 shown above.
However, if I run the same query using SELECT in the datastore viewer with same values, value1 and value2, it works and shows the result. I earlier suspected this to be a result of eventual consistency. However, the indexed values have been written long back (months earlier) for them to be replicated across the other instances.
Is there a way to correct this situation? Unfortunately, I don't have the key of the entity to query with.
Related
I am using the google cloud datastore python client to write an entity into the datastore which contains an embedded entity. An example entity might look like:
data_type: 1
raw_bytes: <unindexed blob>
values: <indexed embedded entity>
I checked the data from the console and the data is getting saved correctly and the values are present.
Next, I need to run a query from a python app engine application. I have represented the above as the following entity in my app engine code:
class DataValues(ndb.Model):
param1 = ndb.BooleanProperty()
param2 = ndb.IntegerProperty()
param3 = ndb.IntegerProperty()
class MyEntity(ndb.Expando):
data_type = ndb.IntegerProperty(required=True)
raw_bytes = ndb.BlobProperty()
values = ndb.StructuredProperty(DataValues)
One of the filters in the query depends on a property in values. Sample query code is as below:
MyEntity.query().filter(MyEntity.data_type == 1).filter(MyEntity.values.param1 == True).get()
I have created the corresponding composite index in my index.yaml
The query runs successfully but the resulting entity contains the embedded entity values as None. All other property values are present.
What can be the issue here ?
Add properties of DataValues entity as properties of the MyEntity.
This is a bit of a guess, but since datastore attributes are kind of keyed by both their name (in this case values) and the name of the "field type/class" (i.e. StructuredProperty), this might fix your problem:
class EmbeddedProperty(ndb.StructuredProperty):
pass
class MyEntity(ndb.Expando):
data_type = ndb.IntegerProperty(required=True)
raw_bytes = ndb.BlobProperty()
values = EmbeddedProperty(DataValues)
Give it a shot and let me know if values starts coming back non-null.
I struggled with the same problem, wanting to convert the embedded entity into a Python dictionary. One possible solution, although not a very elegant one, is to use a GenericProperty:
class MyEntity(ndb.Model):
data_type = ndb.IntegerProperty(required=True)
raw_bytes = ndb.BlobProperty()
values = ndb.GenericProperty()
values will then be read as an "Expando" object: Expando(param1=False,...). You can access the individual values with values.param1, values.param2 etc. I would prefer having a custom model class, but this should do the job.
I'm interested in migrating from JDO queries to Datastore queries to make use of the AsyncDatastore API.
However, I'm unable to make the following query work in Datastore queries:
//JDO query (working correctly)
PersistenceManager pm = PMF.get().getPersistenceManager();
Query query = pm.newQuery("SELECT FROM "
+ Tasks.class.getName()
+ " WHERE archivado==false & arrayUsers=="
+ user.getId()
+ " & taskDate != null & taskDate > best_before_limit "
+ "PARAMETERS Date best_before_limit "
+ "import java.util.Date");
List <Tasks> results= (List<Tasks>) pm.newQuery(query).execute(new Date());
//Datastore query (returning zero entities)
AsyncDatastoreService datastore = DatastoreServiceFactory.getAsyncDatastoreService();
com.google.appengine.api.datastore.Query query = new com.google.appengine.api.datastore.Query("Tasks");
Filter userFilter = new FilterPredicate("arrayUsers", FilterOperator.EQUAL,user.getId());
Filter filterPendingTasks = new FilterPredicate("taskDate", FilterOperator.LESS_THAN_OR_EQUAL , new Date());
Filter completeFilter = CompositeFilterOperator.and(filterPendingTasks,userFilter);
query.setFilter(completeFilter);
List<Entity> results = datastore.prepare(query).asList(FetchOptions.Builder.withDefaults());
Apart from the fact that I have to build my Task objects out of the Entities resulting from the query, these should be the same.
The problem is that the query must look up if the passed user id (user.getId()) is present in the array (arrayUsers). JDO does this without any issues, but no joy with Datastore queries so far.
Any ideas about what is wrong with my code?
As was pointed out by the users commenting, you use different properties in your datastore query. If you have such a query and you don't have EXACTLY the index for this, it won't work. Without seeing what indexes you have, I say this query looks good to me, so either you don't have data that returns there (unlikely, since your JDO query does it), or you're missing a filter.
In general, in datastore when querying for one of the values to equal something specific, you indeed would use something like this :
new Query("Widget").setFilter(new FilterPredicate("x", FilterOperator.EQUAL, 1))
Since you're using an equality filter, you won't get funky results (as you can see in the docs (look for "Properties with multiple values can behave in surprising ways")).
Currently, there is no way to do this using Datastore Queries due to the lack of a "CONTAINS" operator or similar.
The alternative to keep using JDO (at least for this kind of queries).
We have started to get a lot of DatastoreTimeoutException lately for this basic query:
select id from Later where at < '2013-07-04' limit 500 (Some Pseudo SQL)
In code it looks like:
DatastoreService datastore = DatastoreServiceFactory.getDatastoreService();
Query query = new Query("Later").setKeysOnly();
Filter f = new FilterPredicate("at",FilterOperator.LESS_THAN, new Date());
query.setFilter(f);
query.addSort("at", SortDirection.DESCENDING);
PreparedQuery pq = datastore.prepare(query);
return pq.asList(FetchOptions.Builder.withLimit(500));
The table have aprox. 1.5 million entities. Is this normal datastore behavior?
UPDATE: If we remove the filter it is working better. Of course we need the filter so thats not an solution in the long run.
Change to using asIterator instead of asList.
From the docs:
When iterating through the results of a query using the
PreparedQuery.asIterable() and PreparedQuery.asIterator() methods, the
Datastore retrieves the results in batches. By default each batch
contains 20 results, but you can change this value using
FetchOptions.chunkSize(). You can continue iterating through query
results until all are returned or the request times out.
class MyEntity(db.Model):
timestamp = db.DateTimeProperty()
title = db.StringProperty()
number = db.FloatProperty()
db.GqlQuery("SELECT * FROM MyEntity WHERE title = 'mystring' AND timestamp >= date('2012-01-01') AND timestamp <= date('2012-12-31') ORDER BY timestamp DESC").fetch(1000)
This should fetch ~600 entities on app engine. On my dev server it behaves as expected, builds the index.yaml, I upload it, test on server but on app engine it does not return anything.
Index:
- kind: MyEntity
properties:
- name: title
- name: timestamp
direction: desc
I try splitting the query down on datastore viewer to see where the issue is and the timestamp constraints work as expected. The query returns nothing on WHERE title = 'mystring' when it should be returning a bunch of entities.
I vaguely remember fussy filtering where you had to call .filter("prop =",propValue) with the space between property and operator, but this is a GqlQuery so it's not that (and I tried that format with the GQL too).
Anyone know what my issue is?
One thing I can think of:
I added the list of MyEntity entities into the app via BulkLoader.py prior to the new index being created on my devserver & uploaded. Would that make a difference?
The last line you wrote is probably the problem.
Your entities in the actual real datastore are missing the index required for the query.
As far as I know, when you add a new index, App Engine is supposed to rebuild your indexes for you. This may take some time. You can check your admin page to check the state of your indexes and see if it's still building.
Turns out there's a slight bug in the bulkloader supplied with App Engine SDK - basically autogenerated config transforms strings as db.Text, which is no good if you want these fields indexed. The correct import_transform directive should be:
transform.none_if_empty(str)
This will instruct App Engine to index the uploaded field as a db.StringProperty().
I have an appengine app which has been running for about a year now, i have mainly been using JDO queries until now, but i am trying to collect stats and the queries are taking too long. I have the following entity (Device)
public class Device implements Serializable{
...
#Persistent
private Set<Key> feeds;// Key for the Feed entity
...
}
So I want to get a count of how many Devices have a certain Feed. I was doing it in JDOQL before as such (uses javax.jdo.Query):
Query query = pm.newQuery("select from Device where feeds.contains(:feedKey)");
Map<String, Object> paramsf = new HashMap<String, Object>();
paramsf.put("feedKey",feed.getId());
List<Device> results = (List<Device>) query.executeWithMap(paramsf);
Though this code times out now. I was trying to use the Datastore API so I could set chunk size,etc to see if i could speed the query up or use a cursor, but I am unsure how to search in a Set field. I was trying this (uses com.google.appengine.api.datastore.Query)
Query query = new Query("Device");
query.addFilter("feeds", FilterOperator.IN, feed.getId());
query.setKeysOnly();
final FetchOptions fetchOptions = FetchOptions.Builder.withPrefetchSize(100).chunkSize(100);
QueryResultList<Entity> results = dss.prepare(query).asQueryResultList(fetchOptions);
Essentially i am unsure how to search in the one-to-many filed (feeds) for a single key. Is it possible to index it somehow?
hope it makes sense....
Lists (and other things that are implemented as lists, like sets) are indexed individually. As a result, you can simply use an equality filter in your query, the same as if you were filtering on a single value rather than a list. A record will be returned if any of the items in the list match.