MongoDB sorting data fails - arrays

Question

im trying to sort around 40k objects in mongo, what i have is two collections, one of comics and other of characters, characters have a field inside with an array of comic ids where they appear. What i want is a pipeline for the aggregation framework that retrieves the comic with the strongest characters (sum of the strength of each character). I am capable of getting the list of comics with the sum of the strength of each character, however when i try to sort it, the database keeps waiting and everything ends up in a timeout. What am i doing wrong?
Characters model:
{
_id: number,
name: string,
info: {
alignment: string // can be "good" or "bad"
}
stats: {
strength: number
},
comics: [] //array of numbers referencing the id of the comic
}
Comics model:
{
_id: number,
name: string
}
And here my query:
db.comics.aggregation(
{
$lookup: {
from: 'characters',
let: {
comic_id: '$_id',
},
as: 'total_comic_str',
pipeline: [
{
$match: {
$expr: {
$and: [
{$in: ['$$comic_id', '$comics']}, // the character is from this comic
{$eq: ['$info.alignment', 'good']} // the character is a hero
]
}
}
},
{
$group: { // group by comic id and accumulate strength of each hero
_id: '$$comic_id',
str: {
$sum: '$stats.strength'
}
}
}
]
}
},
{
$unwind: {
path: '$total_comic_str',
preserveNullAndEmptyArrays: false
}
},
{
$sort: {
'total_comic_str.str': -1
}
},
{
$limit: 1
}
)

Answer 1

You are facing a cursor timeout.
When you have a query cursor (like what returns by find()) you can set noCursorTimeout() (which is generally not a good practice) to prevent the issue.
But when using an aggregation, the Cursor type is different so there is no noCursorTimeout.
As a solution, you can use the $out pipeline to store aggregation result into a temporary collection, then working with the generated collection as you wish.

Answer 2

$lookup with pipeline has shown to have performance issues for large collections
So I would suggest using just the $lookup without pipeline. This will work for your particular dataset that have relatively large characters collection and presumably smaller comics arrays
First, it's better to index what you are going to use in $lookup, so you should add an index for the field comics for this to have a meaningful improvement.
Since the characters will a subdocument array, We are going to use $reduce instead of $group to calculate total strength
Your aggregation pipeline should look like this
[
{
$lookup: {
from: "characters",
localField: "_id", // lookup with _id only we will filter out alignment later
foreignField: "comics",
as: "characters"
}
},
{
$project: {
name: true,
total_strength: {
$reduce: {
input: "$characters",
initialValue: 0,
in: {
$add: [
"$$value",
{
$cond: [
{ $eq: [ "$$this.info.alignment", "good"] }, // calculating only "good" character here
"$$this.stats.strength",
0
]
}
]
}
}
}
}
},
{
$sort: { total_strength: -1 }
},
{
$limit: 1
}
]