In Google App Engine's NDB, say that I need an entity to:
contain sometimes a string
contain sometimes a list of strings
the field is never required
I never need both at the same time
Given the queries on a repeated property with a single element work the same as with a non-repeated property, will there be a performance difference or any disadvantages in having a single repeated field even when I will sometimes use it as a regular StringProperty? (leaving aside the better readability of having different fields for repeated and non-repeated)
Basically:
class model1(ndb.Model):
str1 = ndb.StringProperty(repeated=True)
vs
class model2(ndb.Model):
str1 = ndb.StringProperty()
str2 = ndb.StringProperty(repeated=True)
Thanks.
Yes of course, but the field will be a list.
If you don't store anything it will be an empty list. If you add a string it will be a list with one value.
Remember that as this field will be a list, then to add values just call "field.append(yourstring)
Related
I would like to avoid costly repeated data base queries in Anylogic. I have seen the following thread in Stack Overflow What is the fastest way to look up continuous data on Anylogic (Java, SQL) where a simple three step answer is provided but I'm not sure what the second point of the three actually means:
Save all rows as instances of that class at model start-up into a map - you can use Origin/Destination as the key (use Anylogic's Pair object) and the class instance as the value
I have created a class that takes as inputs the information from each column of my database. I would now like to save each row as an instance of that class - is there an easy way to do this? I may be missing something simple as I'm new to Anylogic.
I'm also unsure of how to create a mapping, if anyone could add more detail to point 2 above I'd be very grateful!
this is effectively the best advice, you created the class, which is a great step, but now, one element of that class, will be used as the key... for example the name... for instance if your class has firstName as one variable and lastName as another variable, you will use a string that is the concatenation of firstName and lastName as your key. Of course any key is fine, assuming that it is unique for all your table. Also an integer as an id is ok too.
create a collection of type LinkedHashMap
Create a class (you did that)
Your collection will take as the key a String (first + last name) and as the value of the elment the class...
now, when you read your database you will have something like this:
for(Tuple t : yourQueryResults){
YourClass yc=new YourClass(t.get(db.var1),t.get(db.var2));
String totalName=t.get(db.first_name)+"_"+t.get(db.last_name);
yourCollection.put(totalName,yc);
}
Now every time you want to find someone with the a name, for example "John Doe", instead of making a query, you will do
yourCollection.get("John_Doe").theVarYouWant;
if you use an id instead of the name, you can set an int as the key, and then you will just do yourCollection.get(theId).theVarYouWant
I have an Article type structured like this:
type Article struct {
Title string
Content string `datastore:",noindex"`
}
In an administrative portion of my site, I list all of my Articles. The only property I need in order to display this list is Title; grabbing the content of the article seems wasteful. So I use a projection query:
q := datastore.NewQuery("Article").Project("Title")
Everything works as expected so far. Now I decide I'd like to add two fields to Article so that some articles can be unlisted in the public article list and/or unviewable when access is attempted. Understanding the datastore to be schema-less, I think this might be very simple. I add the two new fields to Article:
type Article struct {
Title string
Content string `datastore:",noindex"`
Unlisted bool
Unviewable bool
}
I also add them to the projection query, since I want to indicate in the administrative article list when an article is publicly unlisted and/or unviewable:
q := datastore.NewQuery("Article").Project("Title", "Unlisted", "Unviewable")
Unfortunately, this only returns entries that have explicitly included Unlisted and Unviewable when Put into the datastore.
My workaround for now is to simply stop using a projection query:
q := datastore.NewQuery("Article")
All entries are returned, and the entries that never set Unlisted or Unviewable have them set to their zero value as expected. The downside is that the article content is being passed around needlessly.
In this case, that compromise isn't terrible, but I expect similar situations to arise in the future, and it could be a big deal not being able to use projection queries. Projections queries and adding new properties to datastore entries seem like they're not fitting together well. I want to make sure I'm not misunderstanding something or missing the correct way to do things.
It's not clear to me from the documentation that projection queries should behave this way (ignoring entries that don't have the projected properties rather than including them with zero values). Is this the intended behavior?
Are the only options in scenarios like this (adding new fields to structs / properties to entries) to either forgo projection queries or run some kind of "schema migration", Getting all entries and then Puting them back, so they now have zero-valued properties and can be projected?
Projection queries source the data for fields from the indexes not the entity, when you have added new properties pre-existing records do not appear in those indexes you are performing the project query on. They will need to be re-indexed.
You are asking for those specific properties and they don't exist hence the current behaviour.
You should probably think of a projection query as a request for entities with a value in a requested index in addition to any filter you place on a query.
in an app i have an entity that contains a list of other entities (let's say an event holding a list of assigned employees)
using objectify - i need to find all the events a particular employee is assigned to.
is there a basic way to filter a query if it contains the parameter - kind of the opposite of the query in
... quick pseudocode
findAll(Employee employee) {
...
return ofy.query(Event.class).filter("employees.contains", employee).list();
}
any help would be greatly appreciated
i tried just doing filter("employees", employee) after seeing this http://groups.google.com/group/objectify-appengine/browse_thread/thread/77ba676192c08e20 - but unfortunately this returns me an empty list
currently i'm doing something really inefficient - going through each event, iterating through the employees and adding them to a new list if it contains the given employee just to have something that works - i know this is not right though
let me add one thing,
the above query is not actually what it is, i was just using that because i did not think this would make a difference.
The Employee and Events are in the same entity group with Business as a parent
the actual query i am using is the following
ofy.query(Event.class).ancestor(businessKey).filter("employees", employee).list();
unfortunately this is still returning an empty list - does having the ancestor(key) in there mess up the filter?
solution, the employees field was not indexed correctly.
I added the datastore-indexes file to create a composite index, but was testing originally on a value that I added before the employees field was indexed, this was something stupid i was doing - simply having an index on the "business" field and the "employees" field fixed everything. the datastore-indexes file did not appear to be necessary, after deleting it and trying again everything worked fine.
Generally, you do this one of two ways:
Put a property of Set<Key<Employee>> on the Event
or
Put a property of Set<Key<Event>> on the Employee
You could also create a relationship entity, but if you're just doing filtering on values with relatively low counts, usually it's easier to just put the set property on one entity or the other.
Then filter as you describe:
ofy.query(Event.class).filter("employees", employee).list()
or
ofy.query(Employee.class).filter("events", event).list()
The list property should hold a Keys to the target entity. If you pass in an entity to the filter() method, Objectify will understand that you want to filter by the key instead.
Example :
/***************************************************/
#Entity
#Cache
public class News {
#Id Long id;
String news ;
#Index List<Long> friend_list = new ArrayList<Long>();
// My friends who can see my news , exemele : friend_list.add(id_f1); friend_list.add(id_f2); friend_list.add(id_f3);
//To make an operation on "friend_list", it is obligatory to index it
}
/*************************************************/
public News(Long id_f){
List<Long> friend_id = new ArrayList<Long>();
friend_id.add(id_f);
Query<Nesw> query = ofy().load().type(News.class).filter("friend_list in",friend_id).limit(limit);
//To filter a list, just after the name of the field you want to filter, add "IN".
//here ==> .filter("friend_list in",friend_id);
// if friend_list contains "id_friend" ==> the query return value
.........
}
I have a set of entries in the datastore and I would like to search/retrieve them as user types query. If I have full string it's easy:
q := datastore.NewQuery("Products").Filter("Name =", name).Limit(20)
but I have no idea how to do it with partial string, please help.
q := datastore.NewQuery("Products").Filter("Name >", name).Limit(20)
There is no like operation on app engine but instead you can use '<' and '>'
example:
'moguz' > 'moguzalp'
EDIT: GAH! I just realized that your question is Go-specific. My code below is for Python. Apologies. I'm also familiar with the Go runtime, and I can work on translating to Python to Go later on. However, if the principles described are enough to get you moving in the right direction, let me know and I wont' bother.
Such an operation is not directly supported on the AppEngine datastore, so you'll have to roll your own functionality to meet this need. Here's a quick, off-the-top-of-my-head possible solution:
class StringIndex(db.Model):
matches = db.StringListProperty()
#classmathod
def GetMatchesFor(cls, query):
found_index = cls.get_by_key_name(query[:3])
if found_index is not None:
if query in found_index.matches:
# Since we only query on the first the characters,
# we have to roll through the result set to find all
# of the strings that matach query. We keep the
# list sorted, so this is not hard.
all_matches = []
looking_at = found_index.matches.index(query)
matches_len = len(foundIndex.matches)
while start_at < matches_len and found_index.matches[looking_at].startswith(query):
all_matches.append(found_index.matches[looking_at])
looking_at += 1
return all_matches
return None
#classmethod
def AddMatch(cls, match) {
# We index off of the first 3 characters only
index_key = match[:3]
index = cls.get_or_insert(index_key, list(match))
if match not in index.matches:
# The index entity was not newly created, so
# we will have to add the match and save the entity.
index.matches.append(match).sort()
index.put()
To use this model, you would need to call the AddMatch method every time that you add an entity that would potentially be searched on. In your example, you have a Product model and users will be searching on it's Name. In your Product class, you might have a method AddNewProduct that creates a new entity and puts it into the datastore. You would add to that method StringIndex.AddMatch(new_product_name).
Then, in your request handler that gets called from your AJAXy search box, you would use StringIndex.GetMatchesFor(name) to see all of the stored products that begin with the string in name, and you return those values as JSON or whatever.
What's happening inside the code is that the first three characters of the name are used for the key_name of an entity that contains a list of strings, all of the stored names that begin with those three characters. Using three (as opposed to some other number) is absolutely arbitrary. The correct number for your system is dependent on the amount of data that you are indexing. There is a limit to the number of strings that can be stored in a StringListProperty, but you also want to balance the number of StringIndex entities that are in your datastore. A little bit of math with give you a reasonable number of characters to work with.
If the number of keywords is limited you could consider adding an indexed list property of partial search strings.
Note that you are limited to 5000 indexes per entity, and 1MB for the total entity size.
But you could also wait for Cloud SQL and Full Text Search API to be avaiable for the Go runtime.
I was hoping to implement an easy, but effective text search for App Engine that I could use until official text search capabilities for app engine are released. I see there are libraries out there, but its always a hassle to install something new. I'm wondering if this is a valid strategy:
1) Break each property that needs to be text-searchable into a set(list) of text fragments
2) Save record with these lists added
3) When searching, just use equality filters on the list properties
For example, if I had a record:
{
firstName="Jon";
lastName="Doe";
}
I could save a property like this:
{
firstName="Jon";
lastName="Doe";
// not case sensative:
firstNameSearchable=["j","o", "n","jo","on","jon"];
lastNameSerachable=["D","o","e","do","oe","doe"];
}
Then to search, I could do this and expect it to return the above record:
//pseudo-code:
SELECT person
WHERE firstNameSearchable=="jo" AND
lastNameSearchable=="oe"
Is this how text searches are implemented? How do you keep the index from getting out of control, especially if you have a paragraph or something? Is there some other compression strategy that is usually used? I suppose if I just want something simple, this might work, but its nice to know the problems that I might run into.
Update:::
Ok, so it turns out this concept is probably legitimate. This blog post also refers to it: http://googleappengine.blogspot.com/2010/04/making-your-app-searchable-using-self.html
Note: the source code in the blog post above does not work with the current version of Lucene. I installed the older version (2.9.3) as a quick fix since google is supposed to come out with their own text search for app engine soon enough anyway.
The solution suggested in the response below is a nice quick fix, but due to big table's limitations, only works if you are querying on one field because you can only use non-equality operators on one property in a query:
db.GqlQuery("SELECT * FROM MyModel WHERE prop >= :1 AND prop < :2", "abc", u"abc" + u"\ufffd")
If you want to query on more than one property, you can save indexes for each property. In my case, I'm using this for some auto-suggest functionality on small text fields, not actually searching for word and phrase matches in a document (you can use the blog post's implementation above for this). It turns out this is pretty simple and I don't really need a library for it. Also, I anticipate that if someone is searching for "Larry" they'll start by typing "La..." as opposed to starting in the middle of the word: "arry". So if the property is for a person's name or something similar, the index only has the substrings starting with the first letter, so the index for "Larry" would just be {"l", "la", "lar", "larr", "larry"}
I did something different for data like phone numbers, where you may want to search for one starting from the beginning or middle digits. In this case, I just stored the entire set of substrings starting with strings of length 3, so the phone number "123-456-7890" would be: {"123","234", "345", ..... "123456789", "234567890", "1234567890"}, a total of (10*((10+1)/2))-(10+9) = 41 indexes... actually what I did was a little more complex in order to remove some unlikely to-be-used substrings, but you get the idea.
Then your query would be:
(Pseaudo Code)
SELECT * from Person WHERE
firstNameSearchIndex == "lar"
phonenumberSearchIndex == "1234"
The way that app engine works is that if the query substrings match any of the substrings in the property, then that is counted as a match.
In practice, this won't scale. For a string of n characters, you need n factorial index entries. A 500 character string would need 1.2 * 10^1134 indexes to capture all possible substrings. You will die of old age before your entity finishes writing to the datastore.
Implementations like search.SearchableModel create one index entry per word, which is a bit more realistic. You can't search for arbitrary substrings, but there is a trick that lets you match prefixes:
From the docs:
db.GqlQuery("SELECT * FROM MyModel
WHERE prop >= :1 AND prop < :2",
"abc", u"abc" + u"\ufffd")
This matches every MyModel entity with
a string property prop that begins
with the characters abc. The unicode
string u"\ufffd" represents the
largest possible Unicode character.
When the property values are sorted in
an index, the values that fall in this
range are all of the values that begin
with the given prefix.