How to use one collection to map another collection in MongoDB - database

Question

I have a dataset like the following
[{id:1,month:1,count:1},{id:1,month:2,count:2},{id:1,month:3,count:3}......,
{id:2,month:1,count:1},{id:2,month:2,count:2},{id:2,month:3,count:3}.......,
........
........
{id:19,month:1,count:1},{id:19,month:2,count:2},{id:19,month:3,count:3}.......,]
and the table looks like this.
|id|month|count|
|1 | 1 | 1 |
.............
.........
|19| 12 | 4 |
there is another id as divisonId and it maps to the ids above as the following
{1:[1,2,4,5],2:[3,6,8,9],3:[7,10,....19]}
and the mapping table looks like this.
|divisionId| id|
| 1 | 1 |
| 1 | 2 |
| 1 | 4 |
| 1 | 5 |
| 2 | 3 |
| 2 | 6 |
......
......
so now I need to aggreate the data and sum and regroup them according to the divisonIds.
so eventually the return data should look like the following
[{divsionId:1,month:1,count:19},{divisionId:1,month:2,count:53},{divisionId:1,month:3,count:66}......,
{divisionId:2,month:1,count:21},{divisionId:2,month:2,count:82},{divisionId:2,month:3,count:63}.......,
{divisionId:3,month:1,count:1},{divisionId:3,month:2,count:2},{divisionId:3,month:3,count:3}.......,]
and the table should looks like
| divisionId| month | count |
| 1 | 1 | 200 |
| 1 | 2 | 400 |
| 1 | 3 | 300 |
.....
.....
| 3 | 11 | 500 |
| 3 | 12 | 600 |
so basically, it just map the ids to divisionId, and sum up the individually months across those ids and aggregate a new collection to return data.
I am not allowed to put divisionId to the original table, due to the fact that ids maybe assigned to different divisionIds in the future, or it could have been much easier to just use the aggregate methods.
currently, one way I can do so is to use Javascript to get datas for ids separately according to the mapping, then do the calculation and push it up to mongos to store it as a new collection, so when UI query the data in the future, it will just read the query, saving the expensive calculation. But it would be awesome if I can solve this problem just by using some advanced mongodb syntax. Please let me know if you have some tricks I could use. thanks.

Answer 1

Please try this :
db.divisionIdCollName.aggregate([{
$lookup:
{
from: "idCollectionName",
let: { ids: "$id" },
pipeline: [
{
$match:
{
$expr:
{ $in: ["$id", "$$ids"] }
}
}
],
as: "data"
}
}, { $unwind: { path: "$data", preserveNullAndEmptyArrays: true } },
{ $group: { _id: { divisionId: '$divisionId', month: '$data.month' }, month: { $first: '$data.month' }, count: { $sum: '$data.count' } } },
{$addFields : {divisionId : '$_id.divisionId'}}, {$project : {_id:0}}
])
Result : Mongo playground
You can test results over there - Once you feel aggregation is returning correct results try to add $merge stage to write result to another collection, you could use $out instead of $merge but we're using $merge is because if given name matches with any collection name in database $out will replace entire collection with aggregation result each time query runs which is destructive & should not be used if this query has to update existing records in a collection, Which is why we're using $merge, Please read about those two before you use, So add below stage as last stage after $project.
Note : $merge is new in v 4.2, $out >= v 2.6. If you're doing $merge since you're specifying two fields on: [ "divisionId", "month" ] - So there should be an unique compound index created on collection divisionIdCollNameNew - So yes we need to manually create collection & create unique index as well & then execute query.
Create collection & index :
db.createCollection("divisionIdCollNameNew")
db.divisionIdCollNameNew.createIndex ( { divisionId: 1, month: 1 }, { unique: true } )
Final Stage :
{ $merge : { into: { coll: "divisionIdCollNameNew" }, on: [ "divisionId", "month" ], whenNotMatched: "insert" } }

Answer 2

ASSUMPTION
Collection months has this structure: {id:1,month:1,count:1}
Collection divisions has this structure: {1:[1,2,4,5],2:[3,6,8,9],3:[7,10,19]}
You may perform such query:
db.divisons.aggregate([
{
$addFields: {
data: {
$filter: {
input: {
$objectToArray: "$$ROOT"
},
cond: {
$isArray: "$$this.v"
}
}
}
}
},
{
$unwind: "$data"
},
{
$lookup: {
from: "months",
let: {
ids: "$data.v"
},
pipeline: [
{
$match: {
$expr: {
$in: [
"$id",
"$$ids"
]
}
}
}
],
as: "months"
}
},
{
$unwind: "$months"
},
{
$group: {
_id: {
divisionId: "$data.k",
month: "$months.month"
},
count: {
$sum: "$months.count"
}
}
},
{
$project: {
_id: 0,
divisionId: "$_id.divisionId",
month: "$_id.month",
count: "$count"
}
},
{
$sort: {
divisionId: 1,
month: 1
}
}
])
MongoPlayground
EXPLANATION
Your divisions collection has not normalized key:value pairs, so our first step is to convert 1:[...], 2:[...] pair into [{k:"1", v:[...]}, {k:2, v:[...]}] pairs with $objectToArray operator.
Then, we flatten array from previous step with $unwind and apply $lookup with uncorrelated sub-queries to cross with months collection.
The last steps, we $group by divisionId + month and sum count value.
In order to store the result inside another collection, you need to use $out or $merge operator.