Searching through descriptions - theory

There's a movie which name I can't remember. It's about a carnival or amusement park with a horror house and a bunch of teens who are murdered one by one by something with a clowns mask. I've seen this movie about 20 years ago, and it's sequel, but can't remember it exactly. (And also forgot it's title.) As a result, I started wondering about how to solve something technical.
Assume that I have a database with the story plot and other data of each and every movie published. (Something like the IMDb.) And I would have an edit field where a user can just enter a description in plain text. The system would then start analysing this text to find the movie(s) that would qualify to this description.
For example (different movie), I enter this in the edit field: "Some movie about an Egyptian king who attacks a bunch of indians on horseback, but he's badly wounded and his horse dies while he lost this battle."
The system should then report the movie "Alexander" from 2004 as answer, but possibly a few more. (Even allowing a few errors in the description.)
To create such a system where a description gets analysed to find a matching record by searching through descriptions, what techniques should I need for something as complex as that? Not that I want to build something like that right now, but more out of curiosity if I ever want to pick up some interesting new project.
(I wanted to award extra points for those who recognise the movie I've mentioned in the beginning. But one Google-attempt later and I found it myself!)
Btw, it's not the search engine itself that interests me, but analysing the description to get to something a search engine will understand! With the example movie, it's human logic that helped me to find the title. (And it's annoying that this movie isn't for sale in the Netherlands.) Human logic will always be a requirement but it's about analysing the user input, which is in the form of a story or description, with possible errors.

You should check out document classification.
A few document classification techniques
Naive Bayes classifier
tf–idf

For what I can tell by your own comments, Google is the technique to be used. ;-) But, honestly, I think more or less any search engine would do.
Edit: heh, you removed your comment, but I do remember you mentioned Google as the one deserving extra points.
Edit+: well, you mentioned Google again, but I don't want to remove my first edit. ;-)

Pure speculation: Would something trivial such as taking every word of more than 4 letters in the description "Egyptian, Indian, horse battle etc." and fuzzy matching against a database of such summaries work? Perhaps with some normalisation eg. king == leader == emperor?
Hmmm ... Young Man, Girlfriend, swimming pool, mother, wedding does that get us to The Graduate? Well I guess with a small amount of specifics "Robinson" it might.

You can do lots of interesting stuff with the imdb keyword search:
http://akas.imdb.com/keyword/carnival/clown/murder/
You can specify multiple keywords, it suggests movies and more keywords which are in similar context with your given keywords.
The data contained in imdb is publicy available for non-commercial use and can be downloaded as text files. You could build a database from it.

Related

Using Watson as a testing tool

I'm wondering about using Watson assistant as a simple tool for informal testing of medical students. I'm a bit confused as to whether this is an appropriate use. I have played around but am quite stuck.
I have a symptom X in mind, that, if the user asks about, Watson would spit out 3 questions sequentially, and test the users responses against some specific terms.
These questions look like
1. how much water does a 'symptom X' patient drink ?
Watson would take their input and compare it against definition somehow
what are the 3 diseases that can manifest with 'symptom X' ?
Watson would then take their input and compare it against the known list
what tests should be run immediately on a patient presenting with 'symptom X' ?
Watson would then compare their input to known list
Am I way off base with how I am using trying to use it?
-So far I have set up
intent = test_me (eg Can you test me)
#entity = symptom X
My first dialog node is if #test_me and #symptom X ->
'Sure, I can test you on symptom X'. I'm going to ask you 3 questions on this.
Pause.
Response -> how much water does a 'symptom X' patient drink ?
Their response would be along the lines of 'more than 100ml/kg/day'
How can I evaluate this response?
Is what I'm trying to do beyond the scope of a chatbot / WA?
The simple way would be by adding NLU (Natural Language Understanding) to the solution. If the language is English, NLU by default would get the 100ml as a Quantity and you can also use the syntax enchantment, if you need to apply a different rule when the user writes things like "more".
If there are more complexity to the sentences and NLU by default is not enough, you can train a custom model using WKS (Watson Knowledge Studio) and use it with NLU. The same applies for languages where the default model doesn't give you enough info.
NLU also have some understand of a good number of medical terms, that seems to be of use for your solution.
If you need to do it using only Watson Assistant, the only solution I can imagine is to use regex to get the number and the type (ml/day/km/etc). Something like "(\d+)(\w{2})"

how can I create .csv file for watson Natural Language Classifier

My .csv format is like this
Good evening,Greeting
good morning,Greeting
good afternoon,Greeting
hi everybody,Greeting
1,quantity
2,quantity
3,quantity
4,quantity
5,quantity
I would like you to give me *,OrderPlacement
I need *,OrderPlacement
I needed *,OrderPlacement
Please send *,OrderPlacement
Now input text is I need 3 pens then NLC set higher confidence to quantity. So how can I resolve this error?
There are a number of issues.
The purpose of NLC is to train off questions in a natural language format. Using those questions, it is able to determine the intent of a question it is never seen before.
Because of this, your training data has to be proper English. These lines will not work:
1,quantity
2,quantity
3,quantity
4,quantity
5,quantity
Instead it should be in a way that someone may ask, and not all the same pattern. For example:
I would like to buy 1 item,quantity
Can I get 2 items?, quantity
Please add three items to my basket, quantity
I want to purchase four boxes of your product, quantity
Please send me five boxes, quantity
Even then I would recommend not to manufacture questions. As you are training it on how you speak, and not your end user.
Also as #Leo mentioned, NLC is not a parser. So these lines will not work:
I would like you to give me *,OrderPlacement
I need *,OrderPlacement
I needed *,OrderPlacement
Please send *,OrderPlacement
Because they are incomplete sentences. You have lost the intent of the original question.
Lastly NLC requires a minimum of 5 questions per intent to correctly train.
You can read more details on how it works in the service API document.

Extract Array of Values for Watson Dialog Variables

In DevPost Watson Developer Challenge for Conversational Applications post, I saw Watson (maybe) able to analyze following phrase "I want to visit Tokyo, Sydney, Manchester, and Reykjavik during a trip that takes 30 days".
Is there a better way to extract those array of locations without having to predefine max no of location variables (i.e. set location1 - 5) and manually specify various grammar items like $ (Locations)={location1} * (Locations)={location2} * (Locations)={location3} * (Locations)={location4} as per Pizza example dialog? I would like to follow up with comment such as "That's a lot" if location > 4, or "Sure" if less.
You could try something like alchemy or relationship extraction to identify all of the languages, and then simply add them to the user profile in Dialog. But today, the best way to do this within a broader conversation will be to do it the same way the pizza sample does as you outlined above.

How to store keywords for an article

In SOLR, I have a document that has id, words (indexed), raw_text fields. I want to search just words field this way:
Words are infinitives of an article (or say keywords). For parsing and lemmatization(stemming) I use another tool, so that's not the point of the question.
E.g.: for these two articles(texts) words would be:
1 Yesterday I didn't go to work, because it was holiday.
words: yesterday go work because holiday
2 Tommorrow I am going to work in the morning and in the evening I am going shopping.
words: tommorrow go work morning evening go shop
3 words: go tommorrow work
In a search for "go " I want to have 2 retreived first (be more relevant) because of having more "go"-s than 1. Also I want to use longer queries with a bunch of words and have retrieved articles containig most of them most times.
E.g: search for: "go tommorrow work" would return 2 more relevant than 3 because there are two "go"-s contrary to only one in 3
So the question: how should I store words? multiValued or just single ? What field type should be used?
Thank you!
(Single-valued) text would suit you.
The text comes with tokenization, stemming and stop word analyzers.
Stemming uses heuristics to derive the root of a word. Among other things, it would find the root of your articles even in infinitive form :-)
Trying it out for your samples (with a few additions):
Original: Yesterday [yesterday's] I didn't go to work [working, workable], because it was holiday [holidays].
Stemmed: Yesterdai yesterdai s I didn t go to work work workabl becaus it wa holidai holidai
Original: Tommorrow I am going [go,going,gone] to work in the morning [mornings] and in the evening I am going shopping [shoppers, shops].
Stemmed: Tommorrow I am go go go gone to work in the morn morn and in the even I am go shop shopper shop
Because it uses heuristics, "workable" does not share roots with "work" and "gone" doesnt share roots with "go". But its a tradeoff that works much simpler and faster while not diminishing the result quality.
And "didnt" and "I" are stop words according to this list, so they are automatically eliminated.
If you ever observe unacceptable results too often, take the trouble to implement Wordnet. They have lemma, part of speech and other natural language goodies.

Does anyone know of a good library for mapping a person's name to his or her gender? [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
I am looking for a library or database that can provide guesses about whether a person is male or female based on his or her name or nickname. Something like
john => "M",
mary => "F",
alex => "A", #ambiguous
I am looking for something that supports names other than English names (such as Japanese, Indian, etc.).
Before I get another answer along the lines of "you are going to offend people by assuming their sex/gender" let me be clear, my application does not interact with anyone. It does not send emails or contact anyone in anyway. There are no users to ask. In many cases, the person in question is dead, and the only information I have is name, birth date, and date of death. The reason I want to know the sex of the individual is to make the grammar of the output nicer and to aid in possible searches that may come latter.
gender.c is an open source C program that does a good job.
It comes with data for 44568 first names from all around the world.
There is good documentation and a description of the file format (basically plain text)
so it should not be to difficult to read it from your own application.
Here is what the author says:
A few words on quality of data
The dictionary of first names has been prepared with utmost care.
For example, the Turkish, Indian and Korean names in this dictionary
have all been independently classified by several native speakers.
I also took special care to list only those names which can currently
be found.
The lesson from this?
Any modifications should be done very cautiously (and they must also
adhere to the sorting required by the search algorithm).
For example, knowing that "Sascha" is a boy's name in Germany,
the author never assumed the English "Sasha" to be a girl's name.
Knowing that "Jan" is a boy's name in Germany, I never assumed it to be
also a English short form of "Janet".
Another case in point is the name "Esra". This is a boy's name in
Germany, but a girl's name in Turkey.
The program calculates a probability for the name being male of female.
It can do so with the name as input alone or with the name and country of origin,
which gives significantly better results.
You can download it from the website of the German computer magazine c't
40 000 Namen.
The article is in German but don't worry, all documentation is English.
Here is the direct ftp link 0717-182.zip if you are not interested in the article.
The zip-File contains the source code, an windows executable, the database
and the documentation.
The gender of a name is something that cannot be inferred programmatically in the general case. You need a name database. Here is a free name database from the US Census Bureau.
EDIT: The link for the 2010 name is dead but there are working links and a libraries in the comments.
"I tell ya, life ain't easy for a boy named 'Sue.'"
...So, why make it any harder? If you need to know the sex, just ask... Otherwise, don't worry about it.
I've builded a free API that gives a probabilistic guess on the gender based on a first name. Instead of using any of the above mentioned approaches, i instead use a huge dataset of profiles from social networks to provide a probabilistic guess along with a certainty factor. It also supports optional filtering through country or language id's. It's getting better by the day as more profiles are added to the dataset.
It's free to use at http://genderize.io
ONE thing you should consider is using a tool that takes demographics into account, as naming conventions will rely heavily on this.
Example
http://api.genderize.io?name=kim
{"name":"kim","gender":"female","probability":"0.89","count":1440}
http://api.genderize.io?name=kim&country_id=dk
{"name":"kim","gender":"male","probability":"0.95","count":44,"country_id":"dk"}
Here are two oddball approaches that may not even work, and likely wouldn't work en masse without violating the terms of a license:
Use the Facebook API (which I know virtually nothing about, it may not even be possible) to perform two searches: one for FB male users with that first name, and one for female. Use the two numbers to decide the probability of gender.
Much looser but more scalable, use the Google API and search for the name plus the gender-specific pronouns, and compare the numbers. For instance, there are 592,000,000 results for searching for "Richard his" (not as a phrase), but only 179,000,000 for "Richard her".
Given your stated constraints, your best option is to re-phrase whatever it is you're writing to be gender-neutral unless you know what gender they want to be called in each instance.
If writing in English, remember that singular “they” is grammatically fine as a gender-neutral third-person singular pronoun.
A good example is the title of this question. As is currently:
… mapping a person's name to his or her sex?
That would be less awkward if written:
… mapping a person's name to their sex?
It's also poor practice to assume that users must be male or female. There are a small but significant number of "intersex" people, most of whom are heartily sick of not having a box to tick..
bignose: interesting on the "singular they". I didn't realize it had such a long history.
It's not a service, but a little app with a database:
http://www.codeproject.com/KB/cpp/genderizer.aspx
And this tool is in german:
http://www.faq-o-matic.net/2011/06/01/zu-einem-vornamen-das-geschlecht-finden/
And another one in VB:
http://www.vbarchiv.net/tipps/tipp_1925-geschlecht-anhand-des-vornamens-ermitteln.html
I think in combination with some "Most used firstname in 2011" lists you should be able to build something decent.
The python package SexMachine will do that for you. Given any first name it returns if it's male, female or unisex. It relies on the data from the gender.c program by Jorg Michael.
The only thing you'll get from trying to automate it is a bunch of unhappy users. From that census data:
JAMES, JOHN, ROBERT, MICHAEL, WILLIAM, DAVID, RICHARD, CHARLES, JOSEPH, THOMAS, CHRISTOPHER, DANIEL, PAUL, MARK, DONALD, GEORGE, KENNETH, STEVEN, EDWARD, BRIAN, RONALD, ANTHONY, KEVIN, JASON, MATTHEW, GARY, TIMOTHY, JOSE, LARRY, JEFFREY, FRANK, SCOTT, ERIC, STEPHEN, ANDREW, RAYMOND, GREGORY, JOSHUA, JERRY, DENNIS, WALTER, PATRICK, PETER, HAROLD, HENRY, CARL, ARTHUR, RYAN, JOE, JUAN, JACK, ALBERT, JUSTIN, TERRY, GERALD, KEITH, SAMUEL, WILLIE, LAWRENCE, ROY, BRANDON, ADAM, FRED, BILLY, LOUIS, JEREMY, AARON, RANDY, EUGENE, CARLOS, RUSSELL, BOBBY, VICTOR, MARTIN, JESSE, SHAWN, CLARENCE, SEAN, CHRIS, JOHNNY, JIMMY, ANTONIO, TONY, LUIS, MIKE, DALE, CURTIS, NORMAN, ALLEN, GLENN, TRAVIS, LEE, MELVIN, KYLE, FRANCIS, JESUS, RAY, JOEL, EDDIE, TROY, ALEXANDER, MARIO, FRANCISCO, MICHEAL, OSCAR, JAY, ALEX, JON, RONNIE, TOMMY, LEON, LEO, WESLEY, DEAN, DAN, LEWIS, COREY, MAURICE, VERNON, ROBERTO, CLYDE, SHANE, SAM, LESTER, CHARLIE, TYLER, GENE, BRETT, ANGEL, LESLIE, CECIL, ANDRE, ELMER, GABRIEL, MITCHELL, ADRIAN, KARL, CORY, CLAUDE, JAMIE, JESSIE, CHRISTIAN, LONNIE, CODY, JULIO, KELLY, JIMMIE, JORDAN, JAIME, CASEY, JOHNNIE, SIDNEY, JULIAN, DARYL, VIRGIL, MARSHALL, PERRY, MARION, TRACY, RENE, FREDDIE, AUSTIN, JACKIE, JOEY, EVAN, DANA, DONNIE, SHANNON, ANGELO, SHAUN, LYNN, CAMERON, BLAKE, KERRY, JEAN, IRA, RUDY, BENNIE, ROBIN, LOREN, NOEL, DEVIN, KIM, GUADALUPE, CARROLL, SAMMY, MARTY, TAYLOR, ELLIS, DALLAS, LAURENCE, DREW, JODY, FRANKIE, PAT, MERLE, TERRELL, DARNELL, TOMMIE, TOBY, VAN, COURTNEY, JAN, CARY, SANTOS, AUBREY, MORGAN, LOUIE, STACY, MICAH, BILLIE, LOGAN, DEMETRIUS, ROBBIE, KENDALL, ROYCE, MICKEY, DEVON, ASHLEY, CAREY, SON, MARLIN, ALI, SAMMIE, MICHEL, RORY, KRIS, AVERY, ALEXIS, GERRY, STACEY, CARMEN, SHELBY, RICKIE, BOBBIE, OLLIE, DENNY, DION, ODELL, MARY, COLBY, HOLLIS, KIRBY, CRUZ, MERRILL, LANE, CLEO, BLAIR, NUMBERS, CLAIR, BERNIE, JOAN, DOMINIQUE, TRISTAN, JAME, GALE, LAVERNE, ALVA, STEVIE, ERIN, AUGUSTINE, YOUNG, JOHNIE, ARIEL, DUSTY, LINDSEY, TRACEY, SCOTTIE, SANDY, SYDNEY, GAIL, DORIAN, LAVERN, REFUGIO, IVORY, ANDREA, SANG, DEON, CAROL, YONG, BERRY, TRINIDAD, SHIRLEY, MARIA, CHANG, ROSARIO, DANNIE, FRANCES, THANH, CONNIE, TORY, LUPE, DEE, SUNG, CHI, QUINN, MINH, THEO, LOU, CHUNG, VALENTINE, JAMEY, WHITNEY, SOL, CHONG, PARIS, OTHA, LACY, DONG, ANTONIA, KELLEY, CARROL, SHAYNE, VAL, JUDE, BRITT, HONG, LEIGH, GAYLE, JAE, NICKY, LESLEY, MAN, KASEY, JEWELL, PATRICIA, LAUREN, ELISHA, MICHAL, LINDSAY, and JEWEL
are all names that work for both males and females. If a girl's name is Robert and everyone, including your software, keeps on calling her a man, she'd be rather pissed.
Although databases are probably the most practical solution, if you want to have some fun maybe you could try writing a neural net (or using a neural net library) that takes in the name and outputs one of those 3 options (F,M,A).
You could train it using the datasets that exist in the databases suggested by other answers, as well as with any other data you have.
This solution would allow you to handle names not specifically categorised previously, and also handle different languages. You might want to pass the language (if you know it) as an input to the neural net as well.
I don't know that I can say neural nets (or any other machine learning) would do a good job of categorising though.
It's culture/region dependent: take Andrea, for Italians is only masculine, for Sweden is a female name while Andreas is for men; Shawn is ambiguous in English.
If a language has declination, like Latin or Russian, the final letters will change according to grammatical rules,
Another source of ambiguities is Family names identical to Personal names.
In my opinion it's impossibile to solve in general.
The idea will clearly not work in most languages.
However if you could tell the nationality beforehand you could have more luck.
In most Slav languages (e.g. russian, polish, bulgarian) you could safely assume that all surnames ending with -va -cha -ska (-a in general are feminine) while -v -ch -shi are masculine.
In fact any surname has feminine and masculine form depending on the ending.
The same names used in other countries (e.g. US) might use only the masculine form though.
The same could be said for first names (-a -ya are feminine) but it is not 100% accurate.
But in general you would hardly get a library that is sufficiently accurate.
I haven't used it, but IBM has a Global Name Analytics library (for a price!) that seems pretty comprehensive.
The Z Directory (at vettrasoft.com) has a C-language function, works something like so:
void func()
{
char c = z_guess_sex_byfirstname ("Lon");
switch(c)
{
case 'M': std::cout << "It's a boy!\n"; break;
case 'F': std::cout << "It's a girl!\n"; break;
case 'B': std::cout << "this name is for both sexes\n"; break;
case '?': std::cout << "sex unknown sorry\n"; break;
}
}
it's database driven, the table has something like 10,000+ names I think, but you need to
download and install the z directory (includes many other topo items like countries, geographical landmarks, airports, states, area codes, postal-zip codes, etc along with
c++ functions and objects to access the data). However the names are very English-language
oriented. The table is a work in progress and gradually updated.
Name-gender maps can work but in multicultural countries it's more like guessing. I can give you one example: Marian in Polish is a typical masculine name, whereas the same name in Great Britain is a female name. In the era of people immigrating all over the world, I'm not sure such database would be very accurate. Good luck!
Some cultures have unisex names - like mine. What do you do then? I think the answer is plain and simple - don't assume - you could cause offence. Just ask if its needed, otherwise gender neutrality.
Well, not anymore. IBM patented that idea a while ago.
So if you're looking for any level of flexability (something other than a list of names), you'll either have to (gasp!) ask the user, or simply pay IBM for the rights :)
In any case, such autodetection is annoying for many people who have gender-ambiguous names, or even just mean parents. Let's not make this any harder for them.
It's not free, but this is a nice library that I have used before:
NetGender for .NET allows you to
quickly and easily build Name
Verification, Parsing and Gender
Determination into your custom
applications. Accurately verify
whether a particular field contains a
valid individual or company. NetGender
uses a 100,000+, ethnically diverse,
Name Dictionary in combination with an
8,000+ Company Name Dictionary to
ensure precise gender determination.
http://www.softwarecompany.com/dotnet/netgender.htm
It's interesting that you say you have birth date. That could help. I've seen databases of histories of name popularity.
In the film Splash (1984), it was funny that Darryl Hannah's character chooses the name "Madison" from a Madison Avenue street sign, because obviously "Madison" is not a girl's name.
24 years later, Madison is the 4th most popular name for girl babies!
Name history from the gov't. (Check out Mary's sad decline through the last 100 years.)
When I wrote to the White House as a child, Richard Nixon (or, perhaps a secretary) responded to me with some photos of the historic place, addressed to "Miss Rhett Anderson." "Miss Rhett?" It doesn't even make sense! Can we REALLY not tell the difference between Clark Gable's Rhett (with a mustache, in Gone With The Wind!) and Vivian Lee's Scarlett? I shall never forgive him, despite Neil Young's assurance that "even Richard Nixon has got soul."
I'm pretty sure no such service could exist with an acceptable level of accuracy. Here are the problems which I think are insurmountable:
There are plenty of names which are for both men and women.
There's a lot of different names in this world, even if you only consider one country.
There is the "A Boy Named Sue" issue, raised so eloquently by Johnny Cash :-)
Check out http://genderchecker.com/
You can have a look at my python gender detection project https://github.com/muatik/genderizer
It tries to detect authors' genders looking their names and/or sample text(for example tweets) of them.
And it also supports mongodb, memcached for performance.
This is not really a programming problem - it comes down to getting a probability table.
AFAIK there are no public databases in distilled forms. You could either build this from census data, or buy the data from someone.
For example, this is someone who sells the probability table for Canada.
IMHO, it is a generally bad idea to determine sex from an individuals name. A lot of names are intersexual (good grief, is this even a word ?? :-), and also they may be one sex in one culture and another in another.
A few stupid examples, just a few that came to mind (from my part of the world, CE)
Vanja - female, in eastern countries from here, mostly male
Alex - intersex (short for Sandra, female, and Sandro, male)
Robin - in western cultures, can be both
In some parts of the world, a persons sex can be determined by looking at how the name ends. For example, Marija, Sandra, Ivana, Petra, Sara, Lucija, Ana - you can see that most of these female names end in "ja" or "ra". There are other examples as well.
Still, I think it's better just to ask the user for sex.
Got this from hacker news discussion about this
I know of no such service. You can perhaps find the data you are looking for, however. The US government publishes data about the prevalence of names and the gender of the person they're attached to. The Social Security Administration has such a page, and the census may as well, but I haven't taken the time to look. Perhaps other world governments do similar things.
I know of no such service, however ..
you could start with a raw list of person names or
guess gender according to some rules (e.g. -o => male, -ela, -a => female)
In some countries (e.g. germany) the name a person can be given is limited by law - maybe there are some publications concerning that matter, which could be harvested (but I don't know of any in the moment).
What I would do is make a hack which takes the name and searches it against the facebook api. Then looks at the resulting users and count how many of them are female or male. You then can return a percentage. Not so insurmountable anymore. :)
Just ask people, and if they are nice they will give you their 'M's or 'F's , and if they are not then give'em an 'A' .

Resources