How do I speed up mongo db queries having many boolean fields? - database

Question

Let's assume my collection user is like this:
[
{
"name":"user1",
"u_id":"1",
"is_happy":true,
"like_baseball":false,
"is_strong":true,
},
{
"name":"user2",
"u_id":"2",
"is_happy":false,
"like_baseball":false,
"is_strong":true,
},
{
"name":"user3",
"u_id":"3",
"is_happy":true,
"like_baseball":false,
"is_strong":false,
},
...
]
There are 1m documents in this collection.
I create two indexes:
1.
{
"is_happy": 1,
"like_baseball": 1,
"is_strong": 1,
}
2.
{
"u_id": 1,
}
We all know the first index cannot help speed up the query below since the selectivity of it is bad:
db.user.find({
is_happy: true,
like_baseball: true,
is_strong: false,
})
MongoDB document provides two ways to deal with bad selectivity
Separate one collection into two collections. (In my case, separate happy and unhappy humans into two collections.) However, I have three boolean fields, which make the separating task hard.
Create a compound index of the field and other fields with a lot of value. (In my case, I can create a compound index of three boolean fields and u_id.) However, this means I have to include the u_id in all queries, which I cannot guarantee.
Since both ways are not suitable for me, I am wondering if there is another way to speed up the query. Thank you all! :)

Answer 1

Sounds like this might be a good use for the attribute pattern. See https://www.mongodb.com/blog/post/building-with-patterns-the-attribute-pattern for details...
By the way, separating into two collections will likely not provide the performance improvements you seek.
if you have the following index:
{
"is_happy": 1,
"like_baseball": 1,
"is_strong": 1,
}
and issue the following query...
db.baseball.find({ is_happy: true, like_baseball: true, is_strong: false })
running an explain plan shows a good ratio (1:1) between Keys Examined and nReturned.
db.baseball.find({ is_happy: true, like_baseball: true, is_strong: false }).explain("allPlansExecution")
All Plans Execution Results:
{
"queryPlanner" : {
"plannerVersion" : 1,
"namespace" : "barrystuff.baseball",
"indexFilterSet" : false,
"parsedQuery" : {
"$and" : [
{
"is_happy" : {
"$eq" : true
}
},
{
"is_strong" : {
"$eq" : false
}
},
{
"like_baseball" : {
"$eq" : true
}
}
]
},
"winningPlan" : {
"stage" : "FETCH",
"inputStage" : {
"stage" : "IXSCAN",
"keyPattern" : {
"is_happy" : 1,
"like_baseball" : 1,
"is_strong" : 1
},
"indexName" : "is_happy_1_like_baseball_1_is_strong_1",
"isMultiKey" : false,
"multiKeyPaths" : {
"is_happy" : [ ],
"like_baseball" : [ ],
"is_strong" : [ ]
},
"isUnique" : false,
"isSparse" : false,
"isPartial" : false,
"indexVersion" : 2,
"direction" : "forward",
"indexBounds" : {
"is_happy" : [
"[true, true]"
],
"like_baseball" : [
"[true, true]"
],
"is_strong" : [
"[false, false]"
]
}
}
},
"rejectedPlans" : [ ]
},
"executionStats" : {
"executionSuccess" : true,
"nReturned" : 3,
"executionTimeMillis" : 0,
"totalKeysExamined" : 3,
"totalDocsExamined" : 3,
"executionStages" : {
"stage" : "FETCH",
"nReturned" : 3,
"executionTimeMillisEstimate" : 0,
"works" : 4,
"advanced" : 3,
"needTime" : 0,
"needYield" : 0,
"saveState" : 0,
"restoreState" : 0,
"isEOF" : 1,
"invalidates" : 0,
"docsExamined" : 3,
"alreadyHasObj" : 0,
"inputStage" : {
"stage" : "IXSCAN",
"nReturned" : 3,
"executionTimeMillisEstimate" : 0,
"works" : 4,
"advanced" : 3,
"needTime" : 0,
"needYield" : 0,
"saveState" : 0,
"restoreState" : 0,
"isEOF" : 1,
"invalidates" : 0,
"keyPattern" : {
"is_happy" : 1,
"like_baseball" : 1,
"is_strong" : 1
},
"indexName" : "is_happy_1_like_baseball_1_is_strong_1",
"isMultiKey" : false,
"multiKeyPaths" : {
"is_happy" : [ ],
"like_baseball" : [ ],
"is_strong" : [ ]
},
"isUnique" : false,
"isSparse" : false,
"isPartial" : false,
"indexVersion" : 2,
"direction" : "forward",
"indexBounds" : {
"is_happy" : [
"[true, true]"
],
"like_baseball" : [
"[true, true]"
],
"is_strong" : [
"[false, false]"
]
},
"keysExamined" : 3,
"seeks" : 1,
"dupsTested" : 0,
"dupsDropped" : 0,
"seenInvalidated" : 0
}
},
"allPlansExecution" : [ ]
},
"serverInfo" : {
"host" : "Barry-MacBook-Pro.local",
"port" : 27017,
"version" : "4.0.6",
"gitVersion" : "caa42a1f75a56c7643d0b68d3880444375ec42e3"
},
"ok" : 1
}
While selectivity might be bad it is the nature of the data, and the query still requires the results. If you need this query and you want better performance you may want to consider vertical scaling first, then if that still does not meet your needs consider horizontal scaling.
If the data model is stable and the field names used are consistent you might be able to use a covered query for your needs. I suspect your real-world need is not as trivial as the example provided.