I'm building an application where users are able to create profiles for themselves by answering a bunch of multiple-choice questions. Users are also able to search for other users by specifying criteria for answers to these questions.
Let's say we have 9 questions q1 .. q9, each with 6 possible answers (0 through 5). This could be represented in a user profile as something like:
class UserProfile(db.Model):
user = db.StringProperty(required=True)
q1 = db.IntegerProperty()
...
q9 = db.IntegerProperty()
Now, consider that a user wants to run a query for users that answered:
0, 1 or 2 for q1
1, 2 or 5 for q2
...
3, 4, or 5 for q9
We could write a query such as:
q = UserProfile.all()
q.filter("q1 IN", [0, 1, 2])
q.filter("q2 IN", [1, 2, 5])
...
q.filter("q9 IN", [3, 4, 5])
Unfortunately, this would generate close to 20,000 sub-queries (assuming that the user specified 3 possible answers for each filter), which is significantly greater than the 30 allowed, not to mention its horrible inefficiency.
Is there a design pattern to do this efficiently?
I can envision a way to turn each of these filters into single equality filters by representing each filter as an integer using binary encoding (e.g., [1, 2, 5] -> b100110 = 38) and storing each user answer in the datastore as a list of queries it would match (e.g., 1 -> bxxxx1x -> [2, 3, 6, 7, 10, 11, .. , 62, 63]). However, this seems a bit kludgy.
I would appreciate if anyone has a more efficient suggestion for an implementation.
UPDATE (on proposed binary encoding):
Nick Johnson raised some interesting concerns about the binary encoding proposed above, so I would like to clarify the proposed encoding in more detail to allow him and others to provide a clear evaluation of its merits and challenges.
I think a concrete example will work best. Also, I think that starting with the query mechanism is also more intuitive.
Continuing with the example from above, let's assume that there are 9 questions with 6 possible answers each (0 through 5). Let's also define that each query will be in the form of a filter on a number of these questions for matching against multiple possible answers (as described above). I propose to convert each query of the form "q2 IN [1, 2, 5]" to an equality filter using binary encoding, where each bit position is 1 if it's one of the queried responses and 0 otherwise. For example, "q2 IN [1, 2, 5]" would translate to "q2 == b100110" or "q2 == 38". Applying this further, the composite query described above would be translated into the following multiple equality filters:
0, 1 or 2 for q1 -> q1 == b000111 -> q1 == 7
1, 2 or 5 for q2 -> q2 == b100110 -> q2 == 38
...
3, 4, or 5 for q9 -> q9 == b111000 -> q9 == 56
To enable turning the "IN" filters into "==" filters, we need to determine in advance which queries (in their binary-encoded form) a profile response will match. For example, if a user selects 2 (among 0 through 5) as the answer, then that response will match any query whose binary encoding has a 1 in the 2-position, i.e. of the form bxxx1xx, where x could be 0 or 1. The set of integers defined by bxxx1xx are [b000100, b000101, b000110, b000111, b001100, b001101, ... , b111100, b111101, b111110, b111111] or in decimal form: [4, 5, 6, 7, 12, 13, ..., 60, 61, 62, 63], which is a list of 32 integers. In general, this "query match set" will have 2^(n-1) elements for a response to a question with n possible answers, because 1 of the n bits in the binary encoding will be fixed to 1, while the others could be 0 or 1.
Therefore, if we had m questions with n possible answers each, then the number of index entries for each entity storing these "query match sets" for each question would be m x (2 ^ (n-1)). If I have:
9 questions with 6 possible answers each, this would require 9 x 2^5 = 288 index entries
10 questions with 8 possible answers each, this would require 10 x 2^7 = 1280 index entries
15 questions with 9 possible answers each, this would require 15 x 2^8 = 3840 index entries
20 questions with 10 possible answers each, this would require 20 x 2^9 = 10240 index entries (which is above the 5000/entity limit imposed by App Engine)
Therefore, I agree that this is not a suitable approach for an arbitrarily large number of questions, especially if the possible number of answers to questions is large also. However, it appears feasible if the number of questions to be indexed is 10-15 and the possible answers don't number more than 6, at least for a majority of the questions.
I will have no more than 10 questions that need to be indexed. Most of them have 3-5 possible answers. Some have 6-7 possible answers, so I'm expecting less than 300 index entries per entity (unless I'm wrong about how I'm calculating the index requirements above).
I don't really view this as a very elegant solution, but:
It appears that indexing overhead could be manageable (i.e. well below the 5000 index rows limit)
It will return exactly what I'm filtering for (rather than getting a partially filtered list of entities, which all need to be transported over the network, only to be filtered further by the application)
I had gathered that the built-in merge-join would be fast enough for this to be effective.
I would still appreciate perspectives on the following questions:
Based on this more detailed explanation, do you think that the indexing requirements could be reasonable? If you think that this still bumps up against limitations, I really would appreciate your insights on this.
Even if the indexing requirements could be reasonable, do you think that writing a query planner would yield a more efficient solution? If so, I would be grateful for (a) a brief explanation of why this would be more efficient and (b) a pointer to how to go about doing this. I'm not sure about how to even get started with a query planner.
There's simply no efficient way to structure the data for queries as you describe them. The only way to do this is to query on the criteria you think will be most restrictive, then filter manually in memory for the remaining criteria.
If you tell us more about the specific sorts of queries people might execute and why, we may be able to provide concrete suggestions for something more efficient.
Related
I am trying to find a way to represent a 36*36 matrix as 4 different numbers in a certain range (-2.0 to 2.0), but I'm struggling to find the best way to achieve this.
The goal is to "generate" 4 unique floating-point "coordinates" based on one input matrix. In other words, some sort of a hashing algorithm.
The numbers in the matrix won't necessarily be unique numbers and the order (position) of a number matters, i.e.
| 1, 2, 3 | | 1, 3, 2 | | 5, 2, 3 |
| 5, 5, 6 | | 5, 5, 6 | | 5, 1, 6 |
should yield distinctly different results.
The matrix itself is stored as an array of unsigned integers (no more than 2 digits, 0 included). It would be preferable if the method will be able to keep the outcomes from the range limits, and the outcomes from different matrices would yield a noticeable amount of variation (ie, if one matrix yields 1.0000001 and another 1.0000003 — not great).
Lastly, the method doesn't have to be scientifically/mathematically valid, but it HAS to be consistent and repeatable, meaning, one particular matrix would always yield the same result.
The language I'm working with is C and I would appreciate immensely any help and advice you guys could offer.
Thanks!
Edit:
This has nothing to do with security/cryptography/rocket science, I don't expect to have every possible permutation to be mapped to a unique outcome, don't require 0 collisions, exceptional speed, or anything of that sort, nor do I worry if the method is somewhat magic. Simply a way to boil a matrix down to 4 numbers, scattered around enough, so that if I'll do the same for 10 matrices, the results would be somewhat different
For example, I can represent a matrix as a scalar using a Frobenius norm, Adler-32, or any other similar method. What might be a decent approach to generate 4 numbers in the desired range, based on that norm\check\hash?
I want to determine whether or not an input array of integers "matches" a set of rules.
The Matcher
The Matcher is built from a set of helper methods to describe rules for input data. These rules are essentially logic gates on arrays of integers:
AND(1, 2) // Requires both 1 AND 2 be present in the input array.
OR(3, 4, 5) // Requires that 3 OR 4 OR 5 be present in the input array.
NOR(6, 7) // Requires that neither 6 NOR 7 be present in the input array.
XOR(8, 9) // Requires that either 8 (X)OR 9 be present in the input array, but not both.
Thus, I could say that, given the input array:
[0, 1, 2, 3]
I could build a Matcher like:
AND(OR(0, 1), AND(1, 2) NOR(4))
Which would match the input, because the input satisfies:
OR(0, 1) // 0 or 1 is present
AND(1, 2) // Both 1 and 2 are present
NOR(4) // 4 is not present
And each of those cumulatively satisfies the overarching AND rule.
The Problem
I need to reduce matchers to the simplest and most basic form that still describes the rules. For example, given the above matcher, a sample reduction could be:
rules = {
or: [1, 2],
xor: [], // No XORs
nor: [4]
}
Each rule has three arrays of sub-rules, comprised of integers or rules.
Notice that the ORs are empty, because 1 is required anyways, which means OR(0, 1) => [0, 1] is redundant because it must be satisfied.
Since Matchers need to be comparable (I need to be able to determine an equivalence between the underlying rules), it becomes a bit more complicated when I get to:
input = [1, 2, 5, 9, 11, 12, 13, 14, 17]
XOR(OR(AND(1, 2), NOR(3, 4), XOR(3 11), AND(11, 14)), AND(1, 5, 17))
Now, a large amount of that is redundant and/or contradictory, so what I was thinking I could do was first place it into a tree-like structure, and then recurse it and reduce unnecessary entries. Any ideas for a better way to do this?
I'm specifically looking for something deterministic (any set of input rules that mean the same thing yield the same final reduced form). If there is a better way to express this problem, I'm interested, and if rules are contradictory it's fine for the reducer to break and throw an exception. This is intended for occasional use in the program, so performance is not much of an issue.
What you are actually dealing with here is propositional logic. Consider the integer your propositions being either false or true depending on whether they are present in the input array.
Your constraints (XOR, AND, etc.) then form a logical formula that is either satisfiable or not. It is in fact hard to figure out for any given formula whether it is satisfiable. However, at first sight this shouldn't concern you because you only have to check whether a given input satisfies the formula.
Unfortunately what you are actually asking is how you can determine whether two propositional formulas are equivalent. It turns out this is equally hard: https://math.stackexchange.com/questions/1050127/how-to-efficiently-determine-if-any-two-propositional-formulas-are-equivalent
I'm working on a problem that on the surface seems easy but I've been unable to find the key to solving it. I think harnessing the power of R it's probably relatively trivial.
The basic premise is this. You have 32 people at a networking event. There are 4 tables that seat 8 people at a time. At the event there are 4 rounds where people meet each other. The idea is to come up with a seating arrangement across the rounds that makes it so every person meets the max number of unique people across the rounds as possible.
So basically you have a 16 (table*round: t1r1, t1r2,...etc.)) column X 32 (number of people) row array of 0/1 (0 = not seated at the table/round, 1 = seated at the table/round). And the array has to meet the following conditions:
People can only have four rounds so all rowsums should be 4.
Each table/round combo can only fit 8 people hence all colsums should be 8
Each person can only be at one table once so rowsum of table columns has to be 1 (t1r1 + t1r2 +t1r3 + t1r4 = 1)
Each person can only be at one per round so rowsum of round columns has to be 1 (t1r1 + t2r1 + t3r1 +t4r1 = 1)
I'm sure there's mathematical solution, but I was thinking that it might be quicker to brute force it in r by generating matrices and saving those that meet these conditions. Is that the best way? I'm open to other suggestions.
I'm trying to compare teams' compositions to known configurations in order to see where I might have a problem :
The trials columns are to be compare against the differents scenarios to see if a column is a superset of a particular scenario (error being default).
Can it be done using index+match/lookup, or do I have to write some VB macro ?
EDIT : I've updated the question with a worksheet with input data.
Worksheet : https://drive.google.com/file/d/0BxwDbXStIEAsUmpONHp1RVRzR2s/edit?usp=sharing
Github Gist : https://gist.github.com/lucasg/11177852 (python script for data gen)
(xlwt module needed to create excel workbooks).
I've simplified the problem using soccer teams : given 7 positions ( 1 goalie, 2 defenders, 2 midfield and 2 forward) and list of presence to certains week-end, I would like to know whether I'm gonna be able to provide a full team or am I to forfeit the match due to lack of key-players.
The positions :
styles = {
"Goalkeeper" : ["Goalkeeper"],
"Defender" : ["Centre back", "Wing"],
"Midfielder" : ["Centre midfield", "Wide"],
"Forward" : ["Centre forward","Winger"]
}
Most football players can play only one position, but some are more versatile and can play any positions in their own field (defense-midfield-attack).
Example of a team (18 pers.):
example_players = {
"Forward": [
[1, "Winger"],
[2, "Winger"],
[3, "Centre forward"],
[4, "Centre forward"]
],
"Defender": [
[5, "Centre back"],
[6, "Centre back", "Wing"],
[7, "Centre back", "Wing"],
[8, "Wing"],
[9, "Centre back"]
],
"Goalkeeper": [
[10, "Goalkeeper"],
[11, "Goalkeeper"]
],
"Midfielder": [
[12, "Centre midfield"],
[13, "Centre midfield"],
[14, "Wide", "Centre midfield"],
[15, "Centre midfield"],
[16, "Centre midfield"],
[17, "Wide", "Centre midfield"],
[18, "Wide", "Centre midfield"]
]
}
To make it more simple, I need at least one person in each zone (goal-def-mid-attack) to be able to play, the most comfortable situation being one person in each of the 7 positions.
ex scenario :
"no_defense_4" : ["Goalkeeper", "Wide", "Winger" ] ,
"no_attack_1" : ["Goalkeeper", "Centre midfield", "Centre back", ] ,
Now, given a list of a hundred weekends, and the list of the presence/abscence of players, I want to know the resulting situation.
I'm looking preferentially for a formula-based solution, since the worksheet will be uploaded and used in google drive
You can represent sets as bit vectors and then use bit operators "equal" or "AND" to test which sets get matched. Using bit vectors as set representation will solve problem of ordering and duplicate values automatically as position of each value in the bit vector is fixed and each bit will be "set" only once, regardless of how many times the value appears in the column that defines the set.
Simple to use bit vector representation in Excel including operators OR, AND, NOT is listed here: http://chandoo.org/wp/2011/07/29/bitwise-operations-in-excel/#comment-207723
For example following function
=POWER(10;0)*MIN(COUNTIF($B$3:$B$12;"T1");1)+POWER(10;1)*MIN(COUNTIF($B$3:$B$12;"T2");1)+POWER(10;2)*MIN(COUNTIF($B$3:$B$12;"S");1)+POWER(10;3)*MIN(COUNTIF($B$3:$B$12;"PL");1)+POWER(10;4)*MIN(COUNTIF($B$3:$B$12;"CC");1)+POWER(10;5)*MIN(COUNTIF($B$3:$B$12;"GC");1)
Converts values in the range $B$3->$B$12 into a bit vector representation having bits 0..5 defined so that the bit is set if the value in any column in the range is equal to:
bit 0 = T1
bit 1 = T2
bit 2 = S
bit 3 = PL
bit 4 = CC
bit 5 = GC
You can add more bits with other values easily by following the same copy/paste pattern.
So to check if certain column matches certain scenario, just compare the bit vectors. Use expression like IF(x=y;"warn2";IF(..)) and substitute bit vector of the column for x and bit vector of the warn2 scenario for y.
If partial matching is needed, you can use the bitwise AND operator as defined in the above article.
This solution as opposed to a VBA-based solution will require some copy/pasting discipline, e.g. when new trial column or new scenario will be added few expressions will have to be copy/pasted and few will have to be updated.
VBA-based solution might solve this maintenance problem automatically for you by using auto-detected CurrentRegions, all necessary logic hidden behind one macro-click.
EDIT: The bit vectors concept applied to the new soccer teams dataset
Worksheet: https://docs.google.com/spreadsheet/ccc?key=0AtZPnBk7a3pvdHcyWDV6ZFFoUTNyWWF0bjl3VFpaRkE&usp=drive_web#gid=0
As it is ambiguous what will be the exact team setup on a given day as one player may be assigned different positions, I have simplified the problem in such a way that instead of "present" or "absent" I expect the table to contain player's position. It should not be a problem to achieve as if you know what positions the player can play then instead of absent,present you can define the set of valid values to be (empty or anything else),Midfielder,Centre midfield,Wide for players 14,17,18. List of valid available values can be configured for each cell using the "Data validation" rules. The abstract role Midfielder stands for "this player can play a midfielder, exact position is not known yet".
To represent positions I use bit vector calculated with this formula
=POWER(10;6)*MIN(COUNTIF(D2:ZZ2;"Goalkeeper");1)+POWER(10;5)*MIN(COUNTIF(D2:ZZ2;"Centre back");1)+POWER(10;4)*MIN(COUNTIF(D2:ZZ2;"Wing");1)+POWER(10;3)*MIN(COUNTIF(D2:ZZ2;"Centre midfield");1)+POWER(10;2)*MIN(COUNTIF(D2:ZZ2;"Wide");1)+POWER(10;1)*MIN(COUNTIF(D2:ZZ2;"Centre forward");1)+POWER(10;0)*MIN(COUNTIF(D2:ZZ2;"Winger");1)
the formula calculates bit vector from a range D2:ZZ2 in such a way so that each position in the range is counted only once and in final vector each position has a fixed place. It is useful to set number format of the vector to custom numeric format 0000000. For example a row containing Wide,Winger,Goalkeeper in any order with any number of repeats will evaluate to vector 1000101 where the left-most bit 6 stands for Goalkeeper and 2nd from the right goes bit 2 standing for Wide. The most comfortable situation is the one with bit vector evaluating to 1111111. The only purpose of this bit vector is to detect the comfortable situation
For matching scenarios to team setups I use another vector composed of 4 digits with this meaning:
leftmost digit 3 - number of goalies (at most 1 counts)
digit 2 - number of defenders (at most 2 counts)
digit 1 - number of midfielders (at most 2 counts)
rightmost digit 0 - number of forwards (at most 2 counts)
The formula to calculate this vector for range D2:ZZ2 looks like this
=POWER(10;3)*MIN(COUNTIF(D2:ZZ2;"Goalkeeper");1)+POWER(10;2)*MIN(COUNTIF(D2:ZZ2;"Defender")+COUNTIF(D2:ZZ2;"Centre back")+COUNTIF(D2:ZZ2;"Wing");2)+POWER(10;1)*MIN(COUNTIF(D2:ZZ2;"Midfielder")+COUNTIF(D2:ZZ2;"Centre midfield")+COUNTIF(D2:ZZ2;"Wide");2)+POWER(10;0)*MIN(COUNTIF(D2:ZZ2;"Forward")+COUNTIF(D2:ZZ2;"Centre forward")+COUNTIF(D2:ZZ2;"Winger");2)
It is useful to set number format of the vector to custom numeric format 0000. This same formula can calculate the 4-digit vector for team setup and for scenario.
Besides position names it can count also abstract position names like Defender.
For example in a row containing Centre back,Centre back,Goalkeeper,Goalkeeper,Goalkeeper,Defender,Defender,Midfielder,Midfielder,Winger the vector looks like 1221.
There are (1+1)*(2+1)*(2+1)*(2+1) = 54 different possible scenarios. I assume each of them is listed in the constraints sheet. You should be able to generate them all in python quite easily.
There are 2 sheets constraints with scenarios and events with days and team setups. The lookup formula that takes the vector calculated for a team setup in row #2 and searches the constraints sheet for a row with exactly the same vector and returns the value from the value column looks like this
=IFERROR(VLOOKUP($A2;constraints!$A:$B;2;FALSE);"?")
$A2 - contains the 4-digit vector formula for the team setup
constraints!$A - column in the sheet with scenarios containing the 4-digit vector formula for the scenario
constraints!$B - column in the sheet with scenarios containing the scenario name - the thing you are looking for
2 - index of column constraints!$B
FALSE - means the lookup column does not have to be sorted
? - fallback value if no matching scenario was found (should not occur)
The Google docs link above contains the formulas, example 3 days and example 11 scenarios.
If there's something unclear let me know and I'll improve the answer as the Google docs link will vanish some day
Let's say I want to determine the probability that I will upvote a question on SO, based only on which tags are present or absent.
Let's also imagine that I have plenty of data about past questions that I did or did not upvote.
Is there a machine learning algorithm that could take this historical data, train on it, and then be able to predict my upvote probability for future questions? Note that it must be the probability, not just some arbitrary score.
Let's assume that there will be up-to 7 tags associated with any given question, these being drawn from a superset of tens of thousands.
My hope is that it is able to make quite sophisticated connections between tags, rather than each tag simply contributing to the end result in a "linear" way (much as words do in a Bayesian spam filter).
So for example, it might be that the word "java" increases my upvote probability, except when it is present with "database", however "database" might increase my upvote probability when present with "ruby".
Oh, and it should be computationally reasonable (training within an hour or two on millions of questions).
What approaches should I research here?
Given that there probably aren't many tags per message, you could just create "n-gram" tags and apply naive Bayes. Regression trees would also produce an empirical probability at the leaf nodes, using +1 for upvote and 0 for no upvote. See http://www.stat.cmu.edu/~cshalizi/350-2006/lecture-10.pdf for some readable lecture notes and http://sites.google.com/site/rtranking/ for an open source implementation.
You can try several methods (linear regression, SMV, neural networks). The input vector should consist of all possible tags, where each tag represents one dimension.
Then each record in a training set has to be transformed to the input vector according to the tags. For example let's say you have different combinations of 4 tags in your training set (php, ruby, ms, sql) and you define an unweighted input vector [php, ruby, ms, sql]. Let's say you have the following 3 records whic are transformed to weighted input vectors:
php, sql -> [1, 0, 0, 1]
ruby -> [0, 1, 0, 0]
ms, sql -> [0, 0, 1, 1]
In case you use linear regression you use the following formula
y = k * X
where y represents an answer (upvote/downvote) in your case and by inserting known values (X - weighted input vectors).
How ta calculate weights in case you use linear regression you can read here but the point is to create binary input vectors which size is equal (or larger in case you take into account some other variables) to the number of all tags and then for each record you set weights for each tag (0 if it is not included or 1 otherwise).