Getting Alexa CategoryListings by Country - alexa

From what I've understood about Alexa's CategoryListings action ( see this for more info ( check API Reference/Actions/CategoryListings )), Alexa basically scrapes DMOZ
for websites in specific categories and then checks the ranking in it's database ( if you choose to sorty by 'Popularity' ).
This makes sense to me and works fine, but two questions arise:
Does Alexa support ordering by Popularity but Country-specific ( e.g. stackoverflow.com ranks 56 in the world, but 13 in India )
If it doesn't, why? I mean, if my assumptions that they have to scrape DMOZ is correct, it doesn't seem like too much to add country ranking as well. I could, of course, get the global rank of some category sites and then query each site for specific country but I'd have to pay additionally for that. Is the reason behind this financial or technical?
I currently need the answer to only the first question, since it's essential to the thing I'm doing. The second question is out of sheer curiosity.

To answer my question:
No, this is not possible (at least to this date) to do this. What I did was get the list of global sites by popularity and then filtered out top 1000 by querying each of them and getting it's rank in a specific country.

Related

Extract Array of Values for Watson Dialog Variables

In DevPost Watson Developer Challenge for Conversational Applications post, I saw Watson (maybe) able to analyze following phrase "I want to visit Tokyo, Sydney, Manchester, and Reykjavik during a trip that takes 30 days".
Is there a better way to extract those array of locations without having to predefine max no of location variables (i.e. set location1 - 5) and manually specify various grammar items like $ (Locations)={location1} * (Locations)={location2} * (Locations)={location3} * (Locations)={location4} as per Pizza example dialog? I would like to follow up with comment such as "That's a lot" if location > 4, or "Sure" if less.
You could try something like alchemy or relationship extraction to identify all of the languages, and then simply add them to the user profile in Dialog. But today, the best way to do this within a broader conversation will be to do it the same way the pizza sample does as you outlined above.

Getting all members of a group and its subgroups

I have groups as such:
GroupA
GroupB
Users
GroupG
Users
So the goal is to get all users that are members of parent group GroupA.
I have the following filter:
(&(objectCategory=Person)(objectClass=User)(mail=*MyEmailDomain.com)(memberOf=CN=GroupB,OU=MyOU3,OU=MyOU2,OU=MyOU1,DC=MyDomain,DC=LOCAL))
Which works for the lowest level groups.
From research, it seems that this should work, but doesn't:
(&(objectCategory=Person)(objectClass=User)(mail=*MyEmailDomain.com)(memberof:1.2.840.113556.1.4.1941:=(CN=GroupA,OU=MyOU3,OU=MyOU2,OU=MyOU1,DC=MyDomain,DC=LOCAL)))
If it matters, I'm using Active Directory Explorer to get the Distinguished Names, and the LDAP Input step in Pentaho's Data Integration tool (Kettle/PDI) to retrieve the data.
I love the fact that I always find the answer to my questions as soon as I post them somewhere. I need to learn to post much earlier and maybe I will spend less time searching :)
Found a random stackoverflow post that indicated there's an error in the msdn article for this and it has too many parenthesis.
This won't work:
(&(objectCategory=Person)(objectClass=User)(mail=*MyEmailDomain.com)(memberof:1.2.840.113556.1.4.1941:=(CN=GroupA,OU=MyOU3,OU=MyOU2,OU=MyOU1,DC=MyDomain,DC=LOCAL)))
But this DOES work:
(&(objectCategory=Person)(objectClass=User)(mail=*MyEmailDomain.com)(memberof:1.2.840.113556.1.4.1941:=CN=GroupA,OU=MyOU3,OU=MyOU2,OU=MyOU1,DC=MyDomain,DC=LOCAL))
(no parenthesis around the Distinguished Name)
Hi This does not fetch the users recursively. This is just giving the list of users of parent group only.
(&(objectCategory=Person)(objectClass=User)(mail=*MyEmailDomain.com)(memberof:1.2.840.113556.1.4.1941:=CN=GroupA,OU=MyOU3,OU=MyOU2,OU=MyOU1,DC=MyDomain,DC=LOCAL))

Searching through descriptions

There's a movie which name I can't remember. It's about a carnival or amusement park with a horror house and a bunch of teens who are murdered one by one by something with a clowns mask. I've seen this movie about 20 years ago, and it's sequel, but can't remember it exactly. (And also forgot it's title.) As a result, I started wondering about how to solve something technical.
Assume that I have a database with the story plot and other data of each and every movie published. (Something like the IMDb.) And I would have an edit field where a user can just enter a description in plain text. The system would then start analysing this text to find the movie(s) that would qualify to this description.
For example (different movie), I enter this in the edit field: "Some movie about an Egyptian king who attacks a bunch of indians on horseback, but he's badly wounded and his horse dies while he lost this battle."
The system should then report the movie "Alexander" from 2004 as answer, but possibly a few more. (Even allowing a few errors in the description.)
To create such a system where a description gets analysed to find a matching record by searching through descriptions, what techniques should I need for something as complex as that? Not that I want to build something like that right now, but more out of curiosity if I ever want to pick up some interesting new project.
(I wanted to award extra points for those who recognise the movie I've mentioned in the beginning. But one Google-attempt later and I found it myself!)
Btw, it's not the search engine itself that interests me, but analysing the description to get to something a search engine will understand! With the example movie, it's human logic that helped me to find the title. (And it's annoying that this movie isn't for sale in the Netherlands.) Human logic will always be a requirement but it's about analysing the user input, which is in the form of a story or description, with possible errors.
You should check out document classification.
A few document classification techniques
Naive Bayes classifier
tf–idf
For what I can tell by your own comments, Google is the technique to be used. ;-) But, honestly, I think more or less any search engine would do.
Edit: heh, you removed your comment, but I do remember you mentioned Google as the one deserving extra points.
Edit+: well, you mentioned Google again, but I don't want to remove my first edit. ;-)
Pure speculation: Would something trivial such as taking every word of more than 4 letters in the description "Egyptian, Indian, horse battle etc." and fuzzy matching against a database of such summaries work? Perhaps with some normalisation eg. king == leader == emperor?
Hmmm ... Young Man, Girlfriend, swimming pool, mother, wedding does that get us to The Graduate? Well I guess with a small amount of specifics "Robinson" it might.
You can do lots of interesting stuff with the imdb keyword search:
http://akas.imdb.com/keyword/carnival/clown/murder/
You can specify multiple keywords, it suggests movies and more keywords which are in similar context with your given keywords.
The data contained in imdb is publicy available for non-commercial use and can be downloaded as text files. You could build a database from it.

Does anyone know of a good library for mapping a person's name to his or her gender? [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
I am looking for a library or database that can provide guesses about whether a person is male or female based on his or her name or nickname. Something like
john => "M",
mary => "F",
alex => "A", #ambiguous
I am looking for something that supports names other than English names (such as Japanese, Indian, etc.).
Before I get another answer along the lines of "you are going to offend people by assuming their sex/gender" let me be clear, my application does not interact with anyone. It does not send emails or contact anyone in anyway. There are no users to ask. In many cases, the person in question is dead, and the only information I have is name, birth date, and date of death. The reason I want to know the sex of the individual is to make the grammar of the output nicer and to aid in possible searches that may come latter.
gender.c is an open source C program that does a good job.
It comes with data for 44568 first names from all around the world.
There is good documentation and a description of the file format (basically plain text)
so it should not be to difficult to read it from your own application.
Here is what the author says:
A few words on quality of data
The dictionary of first names has been prepared with utmost care.
For example, the Turkish, Indian and Korean names in this dictionary
have all been independently classified by several native speakers.
I also took special care to list only those names which can currently
be found.
The lesson from this?
Any modifications should be done very cautiously (and they must also
adhere to the sorting required by the search algorithm).
For example, knowing that "Sascha" is a boy's name in Germany,
the author never assumed the English "Sasha" to be a girl's name.
Knowing that "Jan" is a boy's name in Germany, I never assumed it to be
also a English short form of "Janet".
Another case in point is the name "Esra". This is a boy's name in
Germany, but a girl's name in Turkey.
The program calculates a probability for the name being male of female.
It can do so with the name as input alone or with the name and country of origin,
which gives significantly better results.
You can download it from the website of the German computer magazine c't
40 000 Namen.
The article is in German but don't worry, all documentation is English.
Here is the direct ftp link 0717-182.zip if you are not interested in the article.
The zip-File contains the source code, an windows executable, the database
and the documentation.
The gender of a name is something that cannot be inferred programmatically in the general case. You need a name database. Here is a free name database from the US Census Bureau.
EDIT: The link for the 2010 name is dead but there are working links and a libraries in the comments.
"I tell ya, life ain't easy for a boy named 'Sue.'"
...So, why make it any harder? If you need to know the sex, just ask... Otherwise, don't worry about it.
I've builded a free API that gives a probabilistic guess on the gender based on a first name. Instead of using any of the above mentioned approaches, i instead use a huge dataset of profiles from social networks to provide a probabilistic guess along with a certainty factor. It also supports optional filtering through country or language id's. It's getting better by the day as more profiles are added to the dataset.
It's free to use at http://genderize.io
ONE thing you should consider is using a tool that takes demographics into account, as naming conventions will rely heavily on this.
Example
http://api.genderize.io?name=kim
{"name":"kim","gender":"female","probability":"0.89","count":1440}
http://api.genderize.io?name=kim&country_id=dk
{"name":"kim","gender":"male","probability":"0.95","count":44,"country_id":"dk"}
Here are two oddball approaches that may not even work, and likely wouldn't work en masse without violating the terms of a license:
Use the Facebook API (which I know virtually nothing about, it may not even be possible) to perform two searches: one for FB male users with that first name, and one for female. Use the two numbers to decide the probability of gender.
Much looser but more scalable, use the Google API and search for the name plus the gender-specific pronouns, and compare the numbers. For instance, there are 592,000,000 results for searching for "Richard his" (not as a phrase), but only 179,000,000 for "Richard her".
Given your stated constraints, your best option is to re-phrase whatever it is you're writing to be gender-neutral unless you know what gender they want to be called in each instance.
If writing in English, remember that singular “they” is grammatically fine as a gender-neutral third-person singular pronoun.
A good example is the title of this question. As is currently:
… mapping a person's name to his or her sex?
That would be less awkward if written:
… mapping a person's name to their sex?
It's also poor practice to assume that users must be male or female. There are a small but significant number of "intersex" people, most of whom are heartily sick of not having a box to tick..
bignose: interesting on the "singular they". I didn't realize it had such a long history.
It's not a service, but a little app with a database:
http://www.codeproject.com/KB/cpp/genderizer.aspx
And this tool is in german:
http://www.faq-o-matic.net/2011/06/01/zu-einem-vornamen-das-geschlecht-finden/
And another one in VB:
http://www.vbarchiv.net/tipps/tipp_1925-geschlecht-anhand-des-vornamens-ermitteln.html
I think in combination with some "Most used firstname in 2011" lists you should be able to build something decent.
The python package SexMachine will do that for you. Given any first name it returns if it's male, female or unisex. It relies on the data from the gender.c program by Jorg Michael.
The only thing you'll get from trying to automate it is a bunch of unhappy users. From that census data:
JAMES, JOHN, ROBERT, MICHAEL, WILLIAM, DAVID, RICHARD, CHARLES, JOSEPH, THOMAS, CHRISTOPHER, DANIEL, PAUL, MARK, DONALD, GEORGE, KENNETH, STEVEN, EDWARD, BRIAN, RONALD, ANTHONY, KEVIN, JASON, MATTHEW, GARY, TIMOTHY, JOSE, LARRY, JEFFREY, FRANK, SCOTT, ERIC, STEPHEN, ANDREW, RAYMOND, GREGORY, JOSHUA, JERRY, DENNIS, WALTER, PATRICK, PETER, HAROLD, HENRY, CARL, ARTHUR, RYAN, JOE, JUAN, JACK, ALBERT, JUSTIN, TERRY, GERALD, KEITH, SAMUEL, WILLIE, LAWRENCE, ROY, BRANDON, ADAM, FRED, BILLY, LOUIS, JEREMY, AARON, RANDY, EUGENE, CARLOS, RUSSELL, BOBBY, VICTOR, MARTIN, JESSE, SHAWN, CLARENCE, SEAN, CHRIS, JOHNNY, JIMMY, ANTONIO, TONY, LUIS, MIKE, DALE, CURTIS, NORMAN, ALLEN, GLENN, TRAVIS, LEE, MELVIN, KYLE, FRANCIS, JESUS, RAY, JOEL, EDDIE, TROY, ALEXANDER, MARIO, FRANCISCO, MICHEAL, OSCAR, JAY, ALEX, JON, RONNIE, TOMMY, LEON, LEO, WESLEY, DEAN, DAN, LEWIS, COREY, MAURICE, VERNON, ROBERTO, CLYDE, SHANE, SAM, LESTER, CHARLIE, TYLER, GENE, BRETT, ANGEL, LESLIE, CECIL, ANDRE, ELMER, GABRIEL, MITCHELL, ADRIAN, KARL, CORY, CLAUDE, JAMIE, JESSIE, CHRISTIAN, LONNIE, CODY, JULIO, KELLY, JIMMIE, JORDAN, JAIME, CASEY, JOHNNIE, SIDNEY, JULIAN, DARYL, VIRGIL, MARSHALL, PERRY, MARION, TRACY, RENE, FREDDIE, AUSTIN, JACKIE, JOEY, EVAN, DANA, DONNIE, SHANNON, ANGELO, SHAUN, LYNN, CAMERON, BLAKE, KERRY, JEAN, IRA, RUDY, BENNIE, ROBIN, LOREN, NOEL, DEVIN, KIM, GUADALUPE, CARROLL, SAMMY, MARTY, TAYLOR, ELLIS, DALLAS, LAURENCE, DREW, JODY, FRANKIE, PAT, MERLE, TERRELL, DARNELL, TOMMIE, TOBY, VAN, COURTNEY, JAN, CARY, SANTOS, AUBREY, MORGAN, LOUIE, STACY, MICAH, BILLIE, LOGAN, DEMETRIUS, ROBBIE, KENDALL, ROYCE, MICKEY, DEVON, ASHLEY, CAREY, SON, MARLIN, ALI, SAMMIE, MICHEL, RORY, KRIS, AVERY, ALEXIS, GERRY, STACEY, CARMEN, SHELBY, RICKIE, BOBBIE, OLLIE, DENNY, DION, ODELL, MARY, COLBY, HOLLIS, KIRBY, CRUZ, MERRILL, LANE, CLEO, BLAIR, NUMBERS, CLAIR, BERNIE, JOAN, DOMINIQUE, TRISTAN, JAME, GALE, LAVERNE, ALVA, STEVIE, ERIN, AUGUSTINE, YOUNG, JOHNIE, ARIEL, DUSTY, LINDSEY, TRACEY, SCOTTIE, SANDY, SYDNEY, GAIL, DORIAN, LAVERN, REFUGIO, IVORY, ANDREA, SANG, DEON, CAROL, YONG, BERRY, TRINIDAD, SHIRLEY, MARIA, CHANG, ROSARIO, DANNIE, FRANCES, THANH, CONNIE, TORY, LUPE, DEE, SUNG, CHI, QUINN, MINH, THEO, LOU, CHUNG, VALENTINE, JAMEY, WHITNEY, SOL, CHONG, PARIS, OTHA, LACY, DONG, ANTONIA, KELLEY, CARROL, SHAYNE, VAL, JUDE, BRITT, HONG, LEIGH, GAYLE, JAE, NICKY, LESLEY, MAN, KASEY, JEWELL, PATRICIA, LAUREN, ELISHA, MICHAL, LINDSAY, and JEWEL
are all names that work for both males and females. If a girl's name is Robert and everyone, including your software, keeps on calling her a man, she'd be rather pissed.
Although databases are probably the most practical solution, if you want to have some fun maybe you could try writing a neural net (or using a neural net library) that takes in the name and outputs one of those 3 options (F,M,A).
You could train it using the datasets that exist in the databases suggested by other answers, as well as with any other data you have.
This solution would allow you to handle names not specifically categorised previously, and also handle different languages. You might want to pass the language (if you know it) as an input to the neural net as well.
I don't know that I can say neural nets (or any other machine learning) would do a good job of categorising though.
It's culture/region dependent: take Andrea, for Italians is only masculine, for Sweden is a female name while Andreas is for men; Shawn is ambiguous in English.
If a language has declination, like Latin or Russian, the final letters will change according to grammatical rules,
Another source of ambiguities is Family names identical to Personal names.
In my opinion it's impossibile to solve in general.
The idea will clearly not work in most languages.
However if you could tell the nationality beforehand you could have more luck.
In most Slav languages (e.g. russian, polish, bulgarian) you could safely assume that all surnames ending with -va -cha -ska (-a in general are feminine) while -v -ch -shi are masculine.
In fact any surname has feminine and masculine form depending on the ending.
The same names used in other countries (e.g. US) might use only the masculine form though.
The same could be said for first names (-a -ya are feminine) but it is not 100% accurate.
But in general you would hardly get a library that is sufficiently accurate.
I haven't used it, but IBM has a Global Name Analytics library (for a price!) that seems pretty comprehensive.
The Z Directory (at vettrasoft.com) has a C-language function, works something like so:
void func()
{
char c = z_guess_sex_byfirstname ("Lon");
switch(c)
{
case 'M': std::cout << "It's a boy!\n"; break;
case 'F': std::cout << "It's a girl!\n"; break;
case 'B': std::cout << "this name is for both sexes\n"; break;
case '?': std::cout << "sex unknown sorry\n"; break;
}
}
it's database driven, the table has something like 10,000+ names I think, but you need to
download and install the z directory (includes many other topo items like countries, geographical landmarks, airports, states, area codes, postal-zip codes, etc along with
c++ functions and objects to access the data). However the names are very English-language
oriented. The table is a work in progress and gradually updated.
Name-gender maps can work but in multicultural countries it's more like guessing. I can give you one example: Marian in Polish is a typical masculine name, whereas the same name in Great Britain is a female name. In the era of people immigrating all over the world, I'm not sure such database would be very accurate. Good luck!
Some cultures have unisex names - like mine. What do you do then? I think the answer is plain and simple - don't assume - you could cause offence. Just ask if its needed, otherwise gender neutrality.
Well, not anymore. IBM patented that idea a while ago.
So if you're looking for any level of flexability (something other than a list of names), you'll either have to (gasp!) ask the user, or simply pay IBM for the rights :)
In any case, such autodetection is annoying for many people who have gender-ambiguous names, or even just mean parents. Let's not make this any harder for them.
It's not free, but this is a nice library that I have used before:
NetGender for .NET allows you to
quickly and easily build Name
Verification, Parsing and Gender
Determination into your custom
applications. Accurately verify
whether a particular field contains a
valid individual or company. NetGender
uses a 100,000+, ethnically diverse,
Name Dictionary in combination with an
8,000+ Company Name Dictionary to
ensure precise gender determination.
http://www.softwarecompany.com/dotnet/netgender.htm
It's interesting that you say you have birth date. That could help. I've seen databases of histories of name popularity.
In the film Splash (1984), it was funny that Darryl Hannah's character chooses the name "Madison" from a Madison Avenue street sign, because obviously "Madison" is not a girl's name.
24 years later, Madison is the 4th most popular name for girl babies!
Name history from the gov't. (Check out Mary's sad decline through the last 100 years.)
When I wrote to the White House as a child, Richard Nixon (or, perhaps a secretary) responded to me with some photos of the historic place, addressed to "Miss Rhett Anderson." "Miss Rhett?" It doesn't even make sense! Can we REALLY not tell the difference between Clark Gable's Rhett (with a mustache, in Gone With The Wind!) and Vivian Lee's Scarlett? I shall never forgive him, despite Neil Young's assurance that "even Richard Nixon has got soul."
I'm pretty sure no such service could exist with an acceptable level of accuracy. Here are the problems which I think are insurmountable:
There are plenty of names which are for both men and women.
There's a lot of different names in this world, even if you only consider one country.
There is the "A Boy Named Sue" issue, raised so eloquently by Johnny Cash :-)
Check out http://genderchecker.com/
You can have a look at my python gender detection project https://github.com/muatik/genderizer
It tries to detect authors' genders looking their names and/or sample text(for example tweets) of them.
And it also supports mongodb, memcached for performance.
This is not really a programming problem - it comes down to getting a probability table.
AFAIK there are no public databases in distilled forms. You could either build this from census data, or buy the data from someone.
For example, this is someone who sells the probability table for Canada.
IMHO, it is a generally bad idea to determine sex from an individuals name. A lot of names are intersexual (good grief, is this even a word ?? :-), and also they may be one sex in one culture and another in another.
A few stupid examples, just a few that came to mind (from my part of the world, CE)
Vanja - female, in eastern countries from here, mostly male
Alex - intersex (short for Sandra, female, and Sandro, male)
Robin - in western cultures, can be both
In some parts of the world, a persons sex can be determined by looking at how the name ends. For example, Marija, Sandra, Ivana, Petra, Sara, Lucija, Ana - you can see that most of these female names end in "ja" or "ra". There are other examples as well.
Still, I think it's better just to ask the user for sex.
Got this from hacker news discussion about this
I know of no such service. You can perhaps find the data you are looking for, however. The US government publishes data about the prevalence of names and the gender of the person they're attached to. The Social Security Administration has such a page, and the census may as well, but I haven't taken the time to look. Perhaps other world governments do similar things.
I know of no such service, however ..
you could start with a raw list of person names or
guess gender according to some rules (e.g. -o => male, -ela, -a => female)
In some countries (e.g. germany) the name a person can be given is limited by law - maybe there are some publications concerning that matter, which could be harvested (but I don't know of any in the moment).
What I would do is make a hack which takes the name and searches it against the facebook api. Then looks at the resulting users and count how many of them are female or male. You then can return a percentage. Not so insurmountable anymore. :)
Just ask people, and if they are nice they will give you their 'M's or 'F's , and if they are not then give'em an 'A' .

Autocomplete Dropdown - too much data, timing out

So, I have an autocomplete dropdown with a list of townships. Initially I just had the 20 or so that we had in the database... but recently, we have noticed that some of our data lies in other counties... even other states. So, the answer to that was buy one of those databases with all towns in the US (yes, I know, geocoding is the answer but due to time constraints we are doing this until we have time for that feature).
So, when we had 20-25 towns the autocomplete worked stellarly... now that there are 80,000 it's not as easy.
As I type I am thinking that the best way to do this is default to this state, then there will be much less. I will add a state selector to the page that defaults to NJ then you can pick another state if need be, this will narrow down the list to < 1000. Though, I may have the same issue? Does anyone know of a work around for an autocomplete with a lot of data?
should I post teh codez of my webservice?
Are you trying to autocomplete after only 1 character is typed? Maybe wait until 2 or more...?
Also, can you just return the top 10 rows, or something?
Sounds like your application is suffocating on the amount of data being returned, and then attempted to be rendered by the browser.
I assume that your database has the proper indexes, and you don't have a performance problem there.
I would limit the results of your service to no more than say 100 results. Users will not look at any more than that any how.
I would also only being retrieving the data from the service once 2 or 3 characters are entered which will further reduce the scope of the query.
Good Luck!
Stupid question maybe, but... have you checked to make sure you have an index on the town name column? I wouldn't think 80K names should be stressing your database...
I think you're on the right track. Use a series of cascading inputs, State -> County -> Township where each succeeding one grabs the potential population based on the value of the preceding one. Each input would validate against its potential population to avoid spurious inputs. I would suggest caching the intermediate results and querying against them for the autocomplete instead of going all the way back to the database each time.
If you have control of the underlying SQL, you may want to try several "UNION" queries instead of one query with several "OR like" lines in its where clause.
Check out this article on optimizing SQL.
I'd just limit the SQL query with a TOP clause. I also like using a "less than" instead of a like:
select top 10 name from cities where #partialname < name order by name;
that "Ce" will give you "Cedar Grove" and "Cedar Knolls" but also "Chatham" & "Cherry Hill" so you always get ten.
In LINQ:
var q = (from c in db.Cities
where partialname < c.Name
orderby c.Name
select c.Name).Take(10);

Resources