I have several entities that I am searching across that include dates, and the Search API works great across all of them except for one thing - sorting.
Here's the data model for one of my entities (simplified of course):
class DepositReceipt(ndb.Expando):
#Sets creation date
creation_date = ndb.DateTimeProperty(auto_now_add=True)
And the code to create the search.Document where de is an instance of the entity:
document = search.Document(doc_id=de.key.urlsafe(),
fields=[search.TextField(name='deposit_key', value=de.key.urlsafe()),
search.DateField(name='created', value=de.creation_date),
search.TextField(name='settings', value=de.settings.urlsafe()),
])
This returns a valid document.
And finally the problem line. I took this snippet from the official GAE Search API tutorial and just changed the direction of the sort to DESCENDING and changed the search expression to created (the date property from the Document above).
expr_list = [search.SortExpression(
expression="created", default_value='',
direction=search.SortExpression.DESCENDING)]
I don't think this is important, but the rest of the search code looks like this:
sort_opts = search.SortOptions(expressions=expr_list)
query_options = search.QueryOptions(
cursor=query_cursor,
limit=_NUM_RESULTS,
sort_options=sort_opts)
query_obj = search.Query(query_string=query, options=query_options)
search_results = search.Index(name=index_name).search(query=query_obj)
In production, I get this error message:
InvalidRequest: Failed to parse search request "settings:ag5zfmdoaWRvbmF0aW9uc3IQCxIIU2V0dGluZ3MYmewDDA"; failed to parse date
Changing the expression="created" to anything else works perfectly fine. This also happens across my other entity types that use dates, so I have no idea what's going on. Advice?
I think default_value needs to be a valid date, rather than '' as you have it.
Related
If I have records in mongoDb without "_class" property, query by example does not work. My database is populated by third-party non-java microservice, by the way.
Example:
{
"_id":"5ec3f00d98326d4c0ead815f",
"first_name":"firstName",
"last_name":"lastName"
}
Then MongoRepository.findAll(Example<S> example) is not able to find that record. If I add correct "_class" field manually, all works as expected.
Has someone solved this issue?
Spring Data mongo v.3.0.0.RC1
Ok, there is UntypedExampleMatcher that must be used in this case:
ExampleMatcher matcher = UntypedExampleMatcher.matching()
.withIgnoreNullValues()
.withIgnoreCase();
Entity probe = ...
Example<Entity> entityExample = Example.of(probe, matcher);
entityRepo.findAll(entityExample);
But this approach does not work by some reason. It is very long-running, and ends with exception at the end.
UPDATE:
Because of incorrect "probe", my request tried to fetch from DB thousands of records, therefore it ended up with exception. After fixing it, search with QBE approach works as a charm.
When executing service.calendars().clear(calendarId=calenderID).execute() on a SECONDARY calendar, one gets the following answer:
googleapiclient.errors.HttpError: https://www.googleapis.com/calendar/v3/calendars/XXXXXXXXXXXXXXXXc%40group.calendar.google.com/clear? returned "Invalid Value">
The documentation in https://developers.google.com/calendar/v3/reference/calendars/clear
seems to evolve on 'primary' calendars, though it is not consistent over the programming languages, in ruby the example quotes 'calendarId', while in the other programming languages, 'primary' is used.
How are events on a secondary calendar deleted?
For deleting an event in a secondary calendar, you must use the Events: delete endpoint. A Python example would look like this:
service.events().delete(calendarId='secondary-calendar-id', eventId='secondary-calendar-event-id').execute()
If you want to get a list of all the calendars you have and in that way get their ids, you can use the CalendarList: list endpoint. A Python example would look like this:
page_token = None
while True:
calendar_list = service.calendarList().list(pageToken=page_token).execute()
for calendar_list_entry in calendar_list['items']:
print calendar_list_entry['summary']
page_token = calendar_list.get('nextPageToken')
if not page_token:
break
I'm struggling with a KeyProperty query, and can't see what's wrong.
My model is
class MyList(ndb.Model):
user = ndb.KeyProperty(indexed=True)
status = ndb.BooleanProperty(default=True)
items = ndb.StructuredProperty(MyRef, repeated=True, indexed=False)
I create an instance of MyList with the appropriate data and can run the following properly
cls = MyList
lists = cls.query().fetch()
Returns
[MyList(key=Key('MyList', 12), status=True, items=..., user=Key('User', 11))]
But it fails when I try to filter by user, i.e. finding lists where the user equals a particular entity; even when using the one I've just used for insert, or from the previous query result.
key = lists[0].user
lists = cls.query(cls.user=key).fetch()
Returns
[]
But works fine with status=True as the filter, and I can't see what's missing?
I should add it happens in a unit testing environment with the following v3_stub
self.policy = datastore_stub_util.PseudoRandomHRConsistencyPolicy(probability=0)
self.testbed.init_datastore_v3_stub(
require_indexes=True,
root_path="%s/../"%(os.path.dirname(__file__)),
consistency_policy=self.policy
)
user=Key('User', 11) is a key to a different class: User. Not MyList
Perhaps you meant:
user = ndb.KeyProperty(kind='User', indexed=True)
Your code looks fine, but I have noticed some data integrity issues when developing locally with NDB. I copied your model and code, and I also got the empty list at first, but then after a few more attempts, the data is there.
Try it a few times?
edit: possibly related?
google app engine ndb: put() and then query(), there is always one less item
I have a model that looks like this:
class Report(models.Model):
updater = models.CharField(max_length=15)
pub_date = models.DateTimeField(auto_add_now=True)
identifier = models.CharField(max_length=100)
... and so on...
There are some more fields but they are irrelevant to the question. Now the site has very simple functions - the users can see older reports and their data, and can edit them or add new ones.
However, the identifier field is actually an integer that symbolizes a log file that is being reported. Most of the times, each report has one log. But sometimes it has more than one. I did it as a CharField because I built the site to replace an older sharepoint 2003 website, where that field was treated as simple text. So I want that in my next version, it would be like it should be, i.e. like this:
class Report(models.Model):
updater = models.CharField(max_length=15)
pub_date = models.DateTimeField(auto_add_now=True)
... and so on...
class Log(models.Model):
report = models.ForeignKey(Report)
identifier = models.IntegerField()
The problem is, since in the old site that field was a CharField, people used this as they liked. Meaning, even if they updated various logs in the same report they just did it like this <logid1>, <logid2>. Sometimes they added some text <logid1> which is related to <logid2>.
So I want to change this, but I don't want to lose all the old data, and I can't fix all those edge cases (the DB contains around 22 thousand reports). I thought about adding this to report:
def disp_id(self):
if self.pub_date < ... #the day I'll do the update
return self.identifier
else:
return ', '.join([log.identifier for log in self.log_set.all()])
But then I'm not really getting rid of the old field now am I? I'm just adding a new one and keeping the original null from a certain date.
As far as I know, what I want to do is impossible. I'm only asking because I know that maybe I'm not the first one to deal with this sort of thing and maybe there is a solution that I'm not aware of.
Hope my explanation is clear enough, thanks in advance!
class Report(models.Model):
updater = models.CharField(max_length=15)
pub_date = models.DateTimeField(auto_add_now=True)
identifier = models.CharField(null=True)
... and so on...
logs = models.ManyToManyField(Log,null=True)
class Log(models.Model):
identifier = models.IntegerField()
Make the above model , and then make a script as follow:
ident_list = []
for reports in Report.objects.all():
identifiers = reports.identifiers.split(',')
for idents in identifiers:
if not idents in ident_list:
log = Log.create(**{'identifier' : int(idents)})
ident_list.append(int(idents))
else:
log = Log.objects.get(identifier = int(idents))
report.log.add(log)
Check the data before removing the column identifiers from the table Report.
Does it solves your purpose now ?
On this question I solved the problem of querying Google Datastore to retrieve stuff by user (com.google.appengine.api.users.User) like this:
User user = userService.getCurrentUser();
String select_query = "select from " + Greeting.class.getName();
Query query = pm.newQuery(select_query);
query.setFilter("author == paramAuthor");
query.declareParameters("java.lang.String paramAuthor");
greetings = (List<Greeting>) query.execute(user);
The above works fine - but after a bit of messing around I realized this syntax in not very practical as the need to build more complicated queries arises - so I decided to manually build my filters and now I got for example something like the following (where the filter is usually passed in as a string variable but now is built inline for simplicity):
User user = userService.getCurrentUser();
String select_query = "select from " + Greeting.class.getName();
Query query = pm.newQuery(select_query);
query.setFilter("author == '"+ user.getEmail() +"'");
greetings = (List<Greeting>) query.execute();
Obviously this won't work even if this syntax with field = 'value' is supported by JDOQL and it works fine on other fields (String types and Enums). The other strange thing is that looking at the Data viewer in the app-engine dashboard the 'author' field is stored as type User but the value is 'user#gmail.com', and then again when I set it up as parameter (the case above that works fine) I am declaring the parameter as a String then passing down an instance of User (user) which gets serialized with a simple toString() (I guess).
Anyone any idea?
Using string substitution in query languages is always a bad idea. It's far too easy for a user to break out and mess with your environment, and it introduces a whole collection of encoding issues, etc.
What was wrong with your earlier parameter substitution approach? As far as I'm aware, it supports everything, and it sidesteps any parsing issues. As far as the problem with knowing how many arguments to pass goes, you can use Query.executeWithMap or Query.executeWithArray to execute a query with an unknown number of arguments.