What is the correct way to split full name field?
For example I have a full name field with value:
"Dede Dada Gege Jojo"
And I want to do split on the backend.
Which one is correct one and why?
First name: "Dede", Last name: "Dada Gege Jojo"
First name: "Dede Dada Gege", Last name: "Jojo"
Neither is guaranteed to be correct. If you want correct, give the user the opportunity to enter their names (of which there may be more than two, or in some cases there might only be one) in separate fields.
Also see the classic article Falsehoods Programmers Believe About Names for a variety of other things that can go wrong here.
Related
I'm using Entity Framework with a SQL Express database and now I have to make a search function to find users based on a value typed in a textbox, where the end user just can type in everything he wants (like Google)
What is the best way to create a search function for this. The input should search all columns.
So for example, I have 4 columns. firstname,lastname,address,emailaddress.
When someone types in the searchbox foo, all columns need to be searched for everything that contains foo.
So I thought I just could do something like
context.Users.Where(u =>
u.Firstname.Contains('foo') ||
u.Lastname.Contains('foo') ||
u.Address.Contains('foo') ||
u.EmailAddress.Contains('foo')
);
But... The end user may also type in foo bar. And then the space in the search value becomes an and requirement. So all columns should be searched and for example firstname might be foo and lastname can be bar.
I think this is to complex for a Linq query?
Maybe I should create a search index and combine all columns into the search index like:
[userId] [indexedValue] where indexedValue is [firstname + " "+ lastname + " "+ address +" " + emailaddress].
Then first split the search value based on spaces and then search for columns that have all words in the search value. Is that a good approach?
The first step with any project is managing expectation. Find the minimum viable solution for the business' need and develop that. Expand on it as the business value is proven. Providing a really flexible and intelligent-feeling search capability would of course make the business happy, but it can often not do what they expect it to do, or perform to a standard that they need, where a simpler solution would do what they need, be simpler to develop and execute faster.
If this represents the minimum viable solution and you want to "and" conditions based on spaces:
public IQueryable<User> SearchUser(string criteria)
{
if(string.IsNullOrEmpty(criteria))
return new List<User>().AsQueryable();
var criteriaValues = criteria.Split(' ');
var query = context.Users.AsQueryable();
foreach(var value in criteriaValues)
{
query = query.Where(u =>
|| u.Firstname.Contains(value)
|| u.Lastname.Contains(value)
|| u.Address.Contains(value)
|| u.EmailAddress.Contains(value));
}
return query;
}
The trouble with trying to index the combined values is that there is no guarantee that for a value like "foo bar" that "foo" represents a first name and "bar" represents a last name, or that "foo" represents a complete vs. partial value. You'd also want to consider stripping out commas and other punctuation as someone might type "smith, john"
When it comes to searching it might pay to perform a bit more of a pattern match to detect what the user might be searching for. For instance a single word like "smith" might search an exact match for first name or last name and display results. If there were no matches then perform a Contains search. If it contains 2 words then a First & last name match search assuming "first last" vs. "last, first" If the value has an "#" symbol, default to an e-mail address search, if it starts with a number, then an address search. Each detected search option could have a first pass search (expecting more exact values) then a 2nd pass more broad search assumption if it comes back empty. There could be even 3rd and 4th pass searches available with broader checks. When results are presented there could be a "more results..." button provided to trigger a 2nd, 3rd, 4th, etc. pass search if the returned results didn't return what the user was expecting.
The idea being when it comes to searching: Try to perform the most typical, narrow expected search and allow the user to broaden the search if they so desire. The goal would be to try and "hit" the most relevant results early, helping mold how users enter their criteria, and then tuning to better perform based on user feedback rather than try and write queries to return as many possible hits as possible. The goal is to help users find what they are looking for on the first page of results. Either way, building a useful search will add complexity of leverage new 3rd party libraries. First determine if that capability is really required.
If a user asks the following sentence:
For some reason Watson uses the first date for the both $checkin and $checkout variables even though it detects the second date.
You can refer to the "dialog node" screenshot to see how the nodes are setup.
How can I get Watson to recognize the first date is the checkin date and the second one is the checkout date. Is there a way I could tell Watson after the first date is used if a second one is detected use it to fill the next slot?
I've found something about the #sys-date range_link entity. But the documentation is not detailed.
This is easy to do, but comes with issues you need to be aware of.
Slots allows you to define variables as they are read. For example.
Will generate this:
The issue is that you are assuming that people will ask in the same order that you need the information. If this doesn't happen then this will fail.
You can mitigate this by shaping how the user may ask the question. For example:
"Please let me know where you are leaving and going to"
The person is more likely to respond with the exit date first.
BETA
This is likely to change and doesn't fully work as you expect. You can enable the beta #sys-date in the options. So I wouldn't recommend relying on this until it is final.
You first need to check for range_link. This will tell you if it detected that two dates are connected to each other.
Then you can do the following:
From Date: <? entities['sys-date'].filter("d", "d.role.type == 'date_from'")[0]?.value ?>
To Date: <? entities['sys-date'].filter("d", "d.role.type == 'date_to'")[0]?.value ?>
What this does is find the exact record that has the role of date_from and returns the value. Likewise for date_to.
You end up with something like this.
Let's say I have 3 entries in my store
{
category: "Science",
author: "Charles Darwin",
content: "Lorem ipsum dolor..."
}
{
category: "Science",
author: "Albert Einstein",
content: "sit amet..."
}
{
category: "Philosophy",
author: "Albert Einstein",
content: "consectetur adipisicing elit..."
}
Is it possible to get all the entries "where category = 'Science'", without creating an index on category?
Is it possible to get all the entries "where category = 'Science' AND author = 'Albert Einstein'"?
Is it possible to get all the entries that contain the word "Lorem" is the content field?
Or should I use a different database for these kinds of queries
Is it possible to get all the entries "where category = 'Science'", without creating an index on category?
Yes: just iterate through the entire object store and manually look for objects with that category. Even if you write some convenience function to do this easily, it'll still be pretty slow compared to using an index. So in practice, you would want to use an index there.
Is it possible to get all the entries "where category = 'Science' AND author = 'Albert Einstein'"?
Yes, you can use a "compound index" as is described in this question here. However, the big caveat is that it's not supported in IE10. If that is a problem for your application, then you can only index on individual fields. For any other constraints besides one indexed field, you'll have to iterate through all the results and manually compare. There are various libraries built on top of IndexedDB that aim to make this easier, but I haven't used any of them so I can't help you there. Either way, it's going to be pretty slow if you can't use compound indexes.
Is it possible to get all the entries that contain the word "Lorem" is the content field?
You might be noticing a pattern here... Yes: just iterate through the entire object store and manually look for objects with "Lorem" in the content field. There is no special support built in for full text searching. Theoretically one could imagine a full text search API built on top of IndexedDB (just keep track of all the words), but I'm not sure if anyone has built that yet (and if they have built it, I'm not sure how performance would be).
Or should I use a different database for these kinds of queries
If you need to do a lot of queries like this (or, God forbid, even more complex queries) and you have the option of using a different database, then use a different database. But you might not have that option, depending on what you're trying to accomplish.
I was hoping to implement an easy, but effective text search for App Engine that I could use until official text search capabilities for app engine are released. I see there are libraries out there, but its always a hassle to install something new. I'm wondering if this is a valid strategy:
1) Break each property that needs to be text-searchable into a set(list) of text fragments
2) Save record with these lists added
3) When searching, just use equality filters on the list properties
For example, if I had a record:
{
firstName="Jon";
lastName="Doe";
}
I could save a property like this:
{
firstName="Jon";
lastName="Doe";
// not case sensative:
firstNameSearchable=["j","o", "n","jo","on","jon"];
lastNameSerachable=["D","o","e","do","oe","doe"];
}
Then to search, I could do this and expect it to return the above record:
//pseudo-code:
SELECT person
WHERE firstNameSearchable=="jo" AND
lastNameSearchable=="oe"
Is this how text searches are implemented? How do you keep the index from getting out of control, especially if you have a paragraph or something? Is there some other compression strategy that is usually used? I suppose if I just want something simple, this might work, but its nice to know the problems that I might run into.
Update:::
Ok, so it turns out this concept is probably legitimate. This blog post also refers to it: http://googleappengine.blogspot.com/2010/04/making-your-app-searchable-using-self.html
Note: the source code in the blog post above does not work with the current version of Lucene. I installed the older version (2.9.3) as a quick fix since google is supposed to come out with their own text search for app engine soon enough anyway.
The solution suggested in the response below is a nice quick fix, but due to big table's limitations, only works if you are querying on one field because you can only use non-equality operators on one property in a query:
db.GqlQuery("SELECT * FROM MyModel WHERE prop >= :1 AND prop < :2", "abc", u"abc" + u"\ufffd")
If you want to query on more than one property, you can save indexes for each property. In my case, I'm using this for some auto-suggest functionality on small text fields, not actually searching for word and phrase matches in a document (you can use the blog post's implementation above for this). It turns out this is pretty simple and I don't really need a library for it. Also, I anticipate that if someone is searching for "Larry" they'll start by typing "La..." as opposed to starting in the middle of the word: "arry". So if the property is for a person's name or something similar, the index only has the substrings starting with the first letter, so the index for "Larry" would just be {"l", "la", "lar", "larr", "larry"}
I did something different for data like phone numbers, where you may want to search for one starting from the beginning or middle digits. In this case, I just stored the entire set of substrings starting with strings of length 3, so the phone number "123-456-7890" would be: {"123","234", "345", ..... "123456789", "234567890", "1234567890"}, a total of (10*((10+1)/2))-(10+9) = 41 indexes... actually what I did was a little more complex in order to remove some unlikely to-be-used substrings, but you get the idea.
Then your query would be:
(Pseaudo Code)
SELECT * from Person WHERE
firstNameSearchIndex == "lar"
phonenumberSearchIndex == "1234"
The way that app engine works is that if the query substrings match any of the substrings in the property, then that is counted as a match.
In practice, this won't scale. For a string of n characters, you need n factorial index entries. A 500 character string would need 1.2 * 10^1134 indexes to capture all possible substrings. You will die of old age before your entity finishes writing to the datastore.
Implementations like search.SearchableModel create one index entry per word, which is a bit more realistic. You can't search for arbitrary substrings, but there is a trick that lets you match prefixes:
From the docs:
db.GqlQuery("SELECT * FROM MyModel
WHERE prop >= :1 AND prop < :2",
"abc", u"abc" + u"\ufffd")
This matches every MyModel entity with
a string property prop that begins
with the characters abc. The unicode
string u"\ufffd" represents the
largest possible Unicode character.
When the property values are sorted in
an index, the values that fall in this
range are all of the values that begin
with the given prefix.
I have a web app, I'd like the user to supply their real name, for friend searches. I'm not sure whether to store this as two separate fields in my user class, or as a single field:
class User {
#Persistent
private String mFirstName;
#Persistent
private String mLastName;
}
.. or ..
class User {
#Persistent
private String mFullName;
}
I'm only going to use it to let users search for people. For example, they might search for "John", or "John Doe", or "Doe". I'm not sure what the app engine query engine allows us to do here, in terms of partial matches and such - has anyone gone through this and can recommend a good solution? I'm leaning towards just storing the full name to make searches easier,
Thanks
It's not just a question of how you end up storing names, it also matters how you ask for names on your web form.
If you prompt with a first and last field, the vast majority of inputs will probably conform, but you'll still have many exceptions (prefixes, middle names, suffixes, punctuation, etc).
If you clearly prompt with separate prefix, first name, middle name, last name, suffix fields, there'll be even fewer exceptions, but users might get peeved or confused.
You might even offer both: an easy one-field input or preparsed multifield input. Explore what other web sites do and see if you find them appealing/confusing/whatever.
Also keep in mind that if you input separate fields you can always easily join them later, but if you input only a single field you won't have the typist's help if you need to parse it later.
Short answer: store a full name.
Long answer here (Falsehoods programmers believe about names, by Patrick McKenzie).
I’m going to list assumptions your systems probably make about names. All of these assumptions are wrong.
(#1) People have exactly one canonical full name.
(#20) People have last names, family names, or anything else which is shared by folks recognized as their relatives.