I am trying to lookup users in appengine datastore using their email address. I'm using Go.
This code finds no users.
var users []entity.User
q := datastore.NewQuery("users").Filter("AccountEmail =", "email#address.com")
_, err := q.GetAll(c, &users)
If I change the query to lookup the user with the "Id" property instead it works fine.
var users []entity.User
q := datastore.NewQuery("users").Filter("Id", "185804764220139124118")
_, err := q.GetAll(c, &users)
I've confirmed the property name and value for "AccountEmail" is correct. "AccountEmail" It is indexed too.
Is there some special formatting that needs to be done with an email address to get the query to work?
In order to find the user by email (AccountEmail), all of the following conditions must be true. Please check and ensure each "test" passes:
An entity with property name AccountEmail must exists. Don't forget that the property name is case-sensitive. Note that the datastore name and the struct field name may be different, tags can be used to change it, e.g.
AccountEmail string `datastore:"email"`
The property must be indexed. Note that whether a property is indexed may vary from entity to entity, so you may have an entity where AccountEmail is indexed and another one where AccountEmail is not indexed.
AccountEmail must have a type string. I assume this is trivial and is so. But note that it is possible to save a property with the User type which is different from the string type and when you list entities in the Datastore viewer for example, the email will be displayed just like if it would be an email string, but obviously it is different.
To find the user with AccountEmail="email#address.com", the value saved must be "email#address.com" exactly. Lower and upper case letters are different! Spaces (and all whitespace characters) matter! Please check if the saved value is exactly this as you will not see trailing spaces when printed for example, but they will cause a mismatch! Also some unicode characters have the same visual appearance (they look the same) but their unicode codepoint is not the same and will also cause a mismatch.
Related
I have an Article type structured like this:
type Article struct {
Title string
Content string `datastore:",noindex"`
}
In an administrative portion of my site, I list all of my Articles. The only property I need in order to display this list is Title; grabbing the content of the article seems wasteful. So I use a projection query:
q := datastore.NewQuery("Article").Project("Title")
Everything works as expected so far. Now I decide I'd like to add two fields to Article so that some articles can be unlisted in the public article list and/or unviewable when access is attempted. Understanding the datastore to be schema-less, I think this might be very simple. I add the two new fields to Article:
type Article struct {
Title string
Content string `datastore:",noindex"`
Unlisted bool
Unviewable bool
}
I also add them to the projection query, since I want to indicate in the administrative article list when an article is publicly unlisted and/or unviewable:
q := datastore.NewQuery("Article").Project("Title", "Unlisted", "Unviewable")
Unfortunately, this only returns entries that have explicitly included Unlisted and Unviewable when Put into the datastore.
My workaround for now is to simply stop using a projection query:
q := datastore.NewQuery("Article")
All entries are returned, and the entries that never set Unlisted or Unviewable have them set to their zero value as expected. The downside is that the article content is being passed around needlessly.
In this case, that compromise isn't terrible, but I expect similar situations to arise in the future, and it could be a big deal not being able to use projection queries. Projections queries and adding new properties to datastore entries seem like they're not fitting together well. I want to make sure I'm not misunderstanding something or missing the correct way to do things.
It's not clear to me from the documentation that projection queries should behave this way (ignoring entries that don't have the projected properties rather than including them with zero values). Is this the intended behavior?
Are the only options in scenarios like this (adding new fields to structs / properties to entries) to either forgo projection queries or run some kind of "schema migration", Getting all entries and then Puting them back, so they now have zero-valued properties and can be projected?
Projection queries source the data for fields from the indexes not the entity, when you have added new properties pre-existing records do not appear in those indexes you are performing the project query on. They will need to be re-indexed.
You are asking for those specific properties and they don't exist hence the current behaviour.
You should probably think of a projection query as a request for entities with a value in a requested index in addition to any filter you place on a query.
I have a set of entries in the datastore and I would like to search/retrieve them as user types query. If I have full string it's easy:
q := datastore.NewQuery("Products").Filter("Name =", name).Limit(20)
but I have no idea how to do it with partial string, please help.
q := datastore.NewQuery("Products").Filter("Name >", name).Limit(20)
There is no like operation on app engine but instead you can use '<' and '>'
example:
'moguz' > 'moguzalp'
EDIT: GAH! I just realized that your question is Go-specific. My code below is for Python. Apologies. I'm also familiar with the Go runtime, and I can work on translating to Python to Go later on. However, if the principles described are enough to get you moving in the right direction, let me know and I wont' bother.
Such an operation is not directly supported on the AppEngine datastore, so you'll have to roll your own functionality to meet this need. Here's a quick, off-the-top-of-my-head possible solution:
class StringIndex(db.Model):
matches = db.StringListProperty()
#classmathod
def GetMatchesFor(cls, query):
found_index = cls.get_by_key_name(query[:3])
if found_index is not None:
if query in found_index.matches:
# Since we only query on the first the characters,
# we have to roll through the result set to find all
# of the strings that matach query. We keep the
# list sorted, so this is not hard.
all_matches = []
looking_at = found_index.matches.index(query)
matches_len = len(foundIndex.matches)
while start_at < matches_len and found_index.matches[looking_at].startswith(query):
all_matches.append(found_index.matches[looking_at])
looking_at += 1
return all_matches
return None
#classmethod
def AddMatch(cls, match) {
# We index off of the first 3 characters only
index_key = match[:3]
index = cls.get_or_insert(index_key, list(match))
if match not in index.matches:
# The index entity was not newly created, so
# we will have to add the match and save the entity.
index.matches.append(match).sort()
index.put()
To use this model, you would need to call the AddMatch method every time that you add an entity that would potentially be searched on. In your example, you have a Product model and users will be searching on it's Name. In your Product class, you might have a method AddNewProduct that creates a new entity and puts it into the datastore. You would add to that method StringIndex.AddMatch(new_product_name).
Then, in your request handler that gets called from your AJAXy search box, you would use StringIndex.GetMatchesFor(name) to see all of the stored products that begin with the string in name, and you return those values as JSON or whatever.
What's happening inside the code is that the first three characters of the name are used for the key_name of an entity that contains a list of strings, all of the stored names that begin with those three characters. Using three (as opposed to some other number) is absolutely arbitrary. The correct number for your system is dependent on the amount of data that you are indexing. There is a limit to the number of strings that can be stored in a StringListProperty, but you also want to balance the number of StringIndex entities that are in your datastore. A little bit of math with give you a reasonable number of characters to work with.
If the number of keywords is limited you could consider adding an indexed list property of partial search strings.
Note that you are limited to 5000 indexes per entity, and 1MB for the total entity size.
But you could also wait for Cloud SQL and Full Text Search API to be avaiable for the Go runtime.
I just started developing a GAE app with the Go runtime, so far it's been a pleasure. However, I have encountered the following setback:
I am taking advantage of the flexibility that the datastore provides by having several different structs with different properties being saved with the same entity name ("Item"). The Go language datastore reference states that "the actual types passed do not have to match between Get and Put calls or even across different App Engine requests", since entities are actually just a series of properties, and can therefore be stored in an appropriate container type that can support them.
I need to query all of the entities stored under the entity name "Item" and encode them as JSON all at once. Using that entity property flexibility to my advantage, it is possible to store queried entities into an arbitrary datastore.PropertyList, however, the Get and GetAll functions return ErrFieldMismatch as an error when a property of the queried entities cannot be properly represented (that is to say, incompatible types, or simply a missing value). All of these structs I'm saving are user generated and most values are optional, therefore saving empty values into the datastore. There are no problems while saving these structs with empty values (datastore flexibility again), but there are when retrieving them.
It is also stated in the datastore Go documentation, that it is up to the caller of the Get methods to decide if the errors returned due to empty values are ignorable, recoverable, or fatal. I would like to know how to properly do this, since just ignoring the errors won't suffice, as the destination structs (datastore.PropertyList) of my queries are not filled at all when a query results in this error.
Thank you in advance, and sorry for the lengthy question.
Update: Here is some code
query := datastore.NewQuery("Item") // here I use some Filter calls, as well as a Limit call and an Order call
items := make([]datastore.PropertyList, 0)
_, err := query.GetAll(context, &items) // context has been obviously defined before
if err != nil {
// something to handle the error, which in my case, it's printing it and setting the server status as 500
}
Update 2: Here is some output
If I use make([]datastore.PropertyList, 0), I get this:
datastore: invalid entity type
And if I use make(datastore.PropertyList, 0), I get this:
datastore: cannot load field "Foo" into a "datastore.Property": no such struct field
And in both cases (the first one I assume can be discarded) in items I get this:
[]
According to the following post the go datastore module doesn't support PropertyList yet.
Use a pointer to a slice of datastore.Map instead.
I write these codes
Query q = pm.newQuery(User.class);
q.setFilter("textPosts != null"); // textPosts is a Text
List<User> users = (List<User>) q.execute();
resp.getWriter().println("user num: " + users.size());
And I get
user num: 0
I'm sure that the number should greater than 0.
I probably miss something important.
Thanks in advance!
Querying Unindexed properties:
A query with a filter or sort order on a property will never match an entity with that property unindexed
So your entity is not found because Text properties are never indexed.
This is also relevant (but in your case Text being unindexed prevails) - entities with non-existing properties are not found with queries:
The App Engine datastore distinguishes between an entity without a given property and an entity with a null value for a property. JDO does not support this distinction: every field of an object has a value, possibly null.
So if you have an Entity without a property, then it will not be found with a query, since a query requires that the property exists.
Maybe because Text will not be indexed for query purposes, as it specified in javadoc: http://code.google.com/intl/en/appengine/docs/java/javadoc/com/google/appengine/api/datastore/Text.html
Btw, i'm sure that it must throw exception at this case.
I have a TClientDataSet, which is provided by a TTable’s dataset.
The dataset has two fields: postalcode (string, 5) and street (string, 20)
At runtime I want to display a third field (string, 20). The routine of this field is getting the postalcode as a parameter and gives back the city belongs to this postalcode.
The problem is only about adding a calculated field to the already existing ones. Filling the data itself is not the problem.
I tried:
cds.SetProvider(Table1);
cds.FieldDefs.Add('city', ftString, 20);
cds.Open;
cds.Edit;
cds.FieldByName('city').AsString := 'Test'; // --> errormessage (field not found)
cds.Post;
cds is my clientdataset, Table1 is a paradox Table, but the problem is the same with other databases.
Thanks in advance
If you want to add additional fields other than those exist in the underlying data, you need to also add the existing fields manually as well. The dataset needs to be closed when you're adding fields, but you can have the necessary metadata with FieldDefs.Update if you don't want to track all field details manually. Basically something like this:
var
i: Integer;
Field: TField;
begin
cds.SetProvider(Table1);
// add existing fields
cds.FieldDefs.Update;
for i := 0 to cds.FieldDefs.Count - 1 do
cds.FieldDefs[i].CreateField(cds);
// add calculated field
Field := TStringField.Create(cds);
Field.FieldName := 'city';
Field.Calculated := True;
Field.DataSet := cds;
cds.Open;
end;
Also see this excellent article by Cary Jensen.
Well i found a simpler solution, as i have 24 fields in my sql i didnt wanted to add them all manually so i added a dummy field to the sql statement instead like:
select ' ' as city, the rest of the fields ...
which i can modify in my program OnAfterOpen event.
Well i had to define in the sql how long that field should be by leaving enough empty spaces, for instance 5 empty spaces for 5 characters, so i must know how long the city name could be.
You should use CreateDataset after add field:
cds.SetProvider(Table1);
cds.FieldDefs.Add('city', ftString, 20);
cds.CreateDataset;
cds.Open;
cds.Edit;
cds.FieldByName('city').AsString := 'Test';
cds.Post;
Would like to share more accurate Query for unexisting fields. I bet it's better to use cast, neither spaces!
select E.NAME, E.SURNAME, cast(null as varchar(20)) as CITY
from EMPLOYEE E
e.g. | Marc'O | Polo | <NULL> |
It's more accurate, can definetly see field size, understandable, easy, safe!
if you want to combine already existing "dynamic" data fields (from provider side) with additional client side persistent fields (calculated, lookup, internalcalc, aggregate) you should subclass CDS. just introduce extra boolean property CombineFields and either override BindFields (in newer delphi versions) or the entire InternalOpen (as I did in d2006/2007) with the following line
if DefaultFields or CombineFields then CreateFields; { TODO -ovavan -cSIC : if CombineFields is true then persistent fields will coexist with Default ones }
that will allow you to avoid all that runtime mess with FieldDefs/CreateField