MongoDB Aggregation strange behaviour with $or and $text - database

Question

I'm currently facing quite a strange issue, i'm trying to pull from my database some data, based on a $text search and taking into account whatever permissions my user has: my data look like the following:
{
"_id" : ObjectId("5fd0e0c3233c72895e6655c9"),
"Entity" :
{
"Groups" : null,
"Name" : "Terasse"
}
}
I'm doing an aggregation query to both input the search my user queries and it's permissions values, fully formatted, the final query look like this:
db.collection.aggregate([
{
$match: {
$text: {
$search: "Terasse",
$caseSensitive: false,
$diacriticSensitive: false
}
}
},
{
$match: {
$or: [
{
"Entity.Groups": {
"$exists": false
}
},
{
"Entity.Groups": {
"$eq": null
}
},
{
"Entity.Groups": {
"$eq": []
}
},
{
$expr: {
$anyElementTrue: {
$map: {
input: "$Entity.Groups",
as: "group",
in: {
$anyElementTrue: {
$map: {
input: [
"/"
],
as: "userGroup",
in: {
$eq: [
0,
{
$indexOfBytes: [
"$$group",
"$$userGroup"
]
}
]
}
}
}
}
}
}
}
}
]
}
}
])
For a quick explanation, it first does the $text match to find the "Terasse" word in my database,
then run a second match stage to verify that my user can access this data.
My second match stage has an $or, which will first check if the data is correctly formatted before doing a special check to see if my user can access this data.
As you can see, this $or statement is checking that the Groups field of my data is: non-existing, null, or empty.
In this latter case, I would like to return this data no matter what authorization my user have and thus, not executing the very last $expr part at all
This aggregation will work perfectly fine if my Data has "Groups": [ "/" ] for example, but will fail with this error otherwise:
uncaught exception: Error: command failed: {
"ok" : 0,
"errmsg" : "$anyElementTrue's argument must be an array, but is null",
"code" : 17041,
"codeName" : "Location17041"
} : aggregate failed :
From my understanding, this error will happen IF the query will execute till the $expr part AND my Groups field is equal to non existing OR null OR empty, while it should be impossible because the $or statement should return the data as soon as it detects one of the mentionned case.
Finally, the most troubling part is that this second match stage will work perfectly with no errors at all if the first stage IS NOT a $match stage with a $text search
I am completely clueless now, is there an mongo expert that could give me a hand understanding what's happening ?
Thank you.
EDIT : as requested in comments:
this document will not work with the mentioned query
{
"_id": {
"$oid": "5fd0e0c3233c72895e6655c9"
},
"Entity": {
"Groups": null,
"Name": "Terasse"
}
}
this document will work with the mentioned query
{
"_id": {
"$oid": "5fd0e0c3233c72895e6655c9"
},
"Entity": {
"Groups": [ "/" ],
"Name": "Terasse"
}
}
also note that you cannot use mongoplayground to test this, as it requires to create a $text index before-hand (afaik, there is no way to do this in mongoplayground)
EDIT 2:
I am starting to believe that the mongo query system is quite broken when including $text stage, i've reworked the query like this to make sure that it was not due to the $or somewhat not working, and yet, it is still having the same error:
db.collection.aggregate([
{
$match: {
$text: {
$search: "Terasse",
$caseSensitive: false,
$diacriticSensitive: false
}
}
},
{
$match: {
$or: [
{
"Entity.Groups": {
"$exists": false
}
},
{
"Entity.Groups": {
"$eq": null
}
},
{
"Entity.Groups": {
"$eq": []
}
},
{
$and: [
{
"Entity.Groups": {
"$type": "array"
}
},
{
$expr: {
$anyElementTrue: {
$map: {
input: "$Entity.Groups",
as: "group",
in: {
$anyElementTrue: {
$map: {
input: [
"/test"
],
as: "userGroup",
in: {
$eq: [
0,
{
$indexOfBytes: [
"$$group",
"$$userGroup"
]
}
]
}
}
}
}
}
}
}
}
]
}
]
}
}
])
As you can see in this new query, i'm adding an $and check TO MAKE SURE THAT "Entity.Groups" is indeed an array before moving to the $anyElementTrue section and yet, the same error applies.
FINAL EDIT
Thanks to Ray's answer: I've changed my query to the following:
db.collection.aggregate([
{
$match: {
$text: {
$search: "Terasse",
$caseSensitive: false,
$diacriticSensitive: false
}
}
},
{
$addFields: {
"groupsMissing": {
$eq: [
[],
{
$ifNull: [
"$Entity.Groups",
[]
]
}
]
}
}
},
{
$match: {
$or: [
{
"groupsMissing": true
},
{
$expr: {
$anyElementTrue: {
$map: {
input: "$Entity.Groups",
as: "group",
in: {
$anyElementTrue: {
$map: {
input: [
"/test"
],
as: "userGroup",
in: {
$eq: [
0,
{
$indexOfBytes: [
"$$group",
"$$userGroup"
]
}
]
}
}
}
}
}
}
}
}
]
}
}
])
I've used another stage with the $addField as Ray mentionned but remove some of the previous/obsolete stuff, it is now working smoothly, will report if any side-effects re-occurs.
As a final note, i'm still unsure why the previous query that I've did didn't work, and why that solution does, but it seems like adding another stage to the query doing the sanitize checks and then having the second stage only checking the sanitized bool IS WORKING !
Probably it is related to the way mongo is executing the query.
I believe different stages HAS to be ran in a sequential way by mongo which is what I initially expected (though $and should also do that, by the documentation)
Having everything on a single stage is probably making mongo run the query quite differently than written in an effort to optimize it ?
That's all I can guess.

Answer 1

You may want to use $addFields to project some helper fields to make your life easier.
Here is the code I try to modify your version as least as possible.
db.collection.aggregate([
{
$match: {
$text: {
$search: "Terasse",
$caseSensitive: false,
$diacriticSensitive: false
}
}
},
{
$addFields: {
// flag to indicate Entity.Groups is null/missing/empty array
"groupsMissing": {
$eq: [
[],
{
$ifNull: [
"$Entity.Groups",
[]
]
}
]
},
// make Entity.Groups an empty array to avoid $anyElementTrue error
"Entity.Groups": {
$ifNull: [
"$Entity.Groups",
[]
]
}
}
},
{
$match: {
$or: [
// part of the code can be shorten
{
"groupsMissing": true
},
// the code should be the same as your version for the rest
{
$and: [
{
"Entity.Groups": {
"$type": "array"
}
},
{
$expr: {
$anyElementTrue: {
$map: {
input: "$Entity.Groups",
as: "group",
in: {
$anyElementTrue: {
$map: {
input: [
"/test"
],
as: "userGroup",
in: {
$eq: [
0,
{
$indexOfBytes: [
"$$group",
"$$userGroup"
]
}
]
}
}
}
}
}
}
}
}
]
}
]
}
}
])

Answer 2

First, MongoDB provides arrays for storing lists of things. Splitting strings on separators in queries is 1) less performant and 2) more difficult than it needs to be.
With that said, I do not see anything in https://docs.mongodb.com/manual/reference/operator/query/or/ saying the clauses will be evaluated in the order given. Therefore,
while it should be impossible because the $or statement should return the data as soon as it detects one of the mentionned case.
... appears to be an incorrect assumption as to how MongoDB works.
Note that https://docs.mongodb.com/manual/reference/operator/query/and/ does reference short-circuit evaluation.