I'm working with this query:
customers.aggregate: [
{$lookup: {
from: "users",
localField: "_id",
foreignField: "customerId",
as: "users"
}},
{$lookup: {
from: "templates",
let: {localField: "$_id"},
pipeline: [{
$match: { $and: [{
$expr: { $eq: ["$customerId", "$$localField"]}},
{module: false}]
}}],
as: "templates"
}},
{$lookup: {
from: "publications",
localField: "_id",
foreignField: "customerId",
as: "publications"
}},
{$lookup: {
from: "documents",
let: {localField: "$_id"},
pipeline: [{
$match: { $and: [{
$expr: { $eq: ["$customerId", "$$localField"]}},
{createdAt: {$gte: {$date: "<someDate>"}}}]
}}],
as: "recentDocuments"
}}
]
In the last lookup stage I'm filtering documents with the customerId field according to the _id field and newer than <someDate> and then joining those documents to respective "customer" object.
And after this step or if possible even in this same step I would also like to add a new field to each resulting "customer" document with the counted number of all the documents (not only those that pass the time filter) from the "documents" collection with the customerId field value corresponding to the customer document's _id. And I also wish not to join those documents to the customer object as I only need a total number of documents with respective customerId. I can only use extended JSON v1 strict mode syntax.
The result would look like:
customers: [
0: {
users: [...],
templates: [...],
publications: [...],
recentDocuments: [...],
totalDocuments: <theCountedNumber>
},
1: {...},
2: {...},
...
]
Use $set and $size
db.customers.aggregate([
{
$lookup: {
from: "documents",
let: { localField: "$_id" },
pipeline: [
{
$match: {
$and: [
{ $expr: { $eq: [ "$customerId", "$$localField" ] } }
]
}
}
],
as: "recentDocuments"
}
},
{
$set: {
totalDocuments: { $size: "$recentDocuments" }
}
}
])
mongoplayground
So on Thursday I've found a proper syntax to solve my problem. It goes like:
db.customers.aggregate([
{
$lookup: {
from: "users",
localField: "_id",
foreignField: "customerId",
as: "users"
}},
{$lookup: {
from: "templates",
let: {localField: "$_id"},
pipeline: [{
$match: { $and: [{
$expr: { $eq: ["$customerId", "$$localField"]}},
{module: false}]
}}],
as: "templates"
}},
{$lookup: {
from: "publications",
localField: "_id",
foreignField: "customerId",
as: "publications"
}},
{$lookup: {
from: "documents",
let: {localField: "$_id"},
pipeline: [{
$match: { $and: [{
$expr: { $eq: ["$customerId", "$$localField"]}},
{createdAt: {$gte: {$date: "<someDate>"}}}]
}}],
as: "recentDocuments"
},
{$lookup: {
from: "documents",
let: {localField: "$_id"},
pipeline: [{
$match: {$and: [{
$expr: {$eq: ["$customerId", "$$localField"]}},
{ $count: "count" }],
as: "documentsNumber"
}}
])
This command would, in the last stage of the aggregate pipeline, go over the documents collection again, but this time would return all the documents instead of filtering by the time period, and then would swap the resulting object for every "customer" object with the array with one item being the number of all the documents. The array could be later "unwinded" with the $unwind action, but it proved to decrease the performance drastically, thus - omitted. I really hope this will help someone to solve a similar problem.
Related
I'm building a view that compiles data from a few different collections so that I don't have to make multiple queries during an API call. I'd like to do some filtering on top of the basic "localField" "foreignField" filtering but can't quite seem to get the right result no matter what I do. The problem seems to be that I need to check which objects of the array contain an array that has an ObjectId included in another array.
I'm using aggregate function while trying to tweak my results. It looks like this:
db.users.aggregate([{ $lookup: { from: "organizations", localField: "context", foreignField: "_id", as: "context" } }, { $lookup: { from: "roles", localField: "roles", foreignField: "_id", as: "roles_with_details" } }, { $lookup: { from: "service_modules", localField: "context.enabled_service_modules", foreignField: "_id", as: "service_modules" } }, { $project: { "context.enabled_service_modules": 0, "password": 0 } }])
This returns data in the following format:
[
{
_id: ObjectId("63907d27ba21557a3455a24b"),
first_name: 'Tep',
last_name: 'Tes',
display_name: 'Tep Tes',
created: ISODate("2022-12-07T11:46:47.230Z"),
last_seen: 1670618355349,
username: 'example#example.com',
email: 'example#example.com',
enable_local_login: 'true',
connected_logins: [],
roles: [
ObjectId("63907c7fba21557a3455a247"),
ObjectId("63907c7fba21557a3455a248")
],
favourite_sm: [ ObjectId("6390832cba21557a3455a250") ],
context: [
{
_id: ObjectId("639074e7ba21557a3455a23f"),
organization_name: 'vip',
display_name: 'Vip',
enabled_login_methods: [ 'ldap' ],
login_method_configurations: [],
created: ISODate("2022-12-07T11:11:35.568Z")
}
],
roles_with_details: [
{
_id: ObjectId("63907c7fba21557a3455a247"),
role: 'jt',
description: 'Has access to membership data'
},
{
_id: ObjectId("63907c7fba21557a3455a248"),
role: 'at',
description: 'Has access to tenant information'
}
],
service_modules: [
{
_id: ObjectId("6390832cba21557a3455a250"),
service_name: 'Jt',
permissions: [
ObjectId("63907c7fba21557a3455a245"),
ObjectId("63907c7fba21557a3455a246"),
ObjectId("63907c7fba21557a3455a247")
],
route: 'jt'
},
{
_id: ObjectId("6390832cba21557a3455a24c"),
service_name: 'Mc',
permissions: [
ObjectId("63907c7fba21557a3455a245"),
ObjectId("63907c7fba21557a3455a246")
],
route: 'mc'
},
{
_id: ObjectId("6390832cba21557a3455a24e"),
service_name: 'Ms',
permissions: [
ObjectId("63907c7fba21557a3455a245"),
ObjectId("63907c7fba21557a3455a246")
],
route: 'ms'
},
{
_id: ObjectId("6390832cba21557a3455a251"),
service_name: 'At',
permissions: [
ObjectId("63907c7fba21557a3455a245"),
ObjectId("63907c7fba21557a3455a246"),
ObjectId("63907c7fba21557a3455a248")
],
route: 'at'
}
]
}
]
I would like to filter this result so that "service_modules" array would only include objects which contain at least one ObjectId in their "service_modules.permissions" array, that is also found in the "roles" array. This would mean that the "service_modules" should look like this:
service_modules: [
{
_id: ObjectId("6390832cba21557a3455a250"),
service_name: 'Jt',
permissions: [
ObjectId("63907c7fba21557a3455a245"),
ObjectId("63907c7fba21557a3455a246"),
ObjectId("63907c7fba21557a3455a247")
],
route: 'jt'
},
{
_id: ObjectId("6390832cba21557a3455a251"),
service_name: 'At',
permissions: [
ObjectId("63907c7fba21557a3455a245"),
ObjectId("63907c7fba21557a3455a246"),
ObjectId("63907c7fba21557a3455a248")
],
route: 'at'
}
]
I have tried modifying the $lookup, which joins the service_modules collection, to following trying to use pipeline to match the arrays:
{ $lookup: { from: "service_modules", localField: "context.enabled_service_modules", foreignField: "_id","let":{"sm":"$service_modules","roles":"$roles"},"pipeline":[{$match:{$expr:{$in:["$$sm.permissions","$$roles"]}}}], as: "service_modules" } }
But it gives a following error:
MongoServerError: PlanExecutor error during aggregation :: caused by :: $in requires an array as a second argument, found: missing
I also tried using unwind and group, but trying to unwind with:
{$unwind:"$service_modules.permissions"}
results in empty set, so I wasn't able to proceed to the grouping stage.
How should I filter the $lookup in order to get the result I'd want?
One way you could do it is by using a "$match" in a "pipeline" of the service_modules "$lookup".
db.users.aggregate([
{
$lookup: {
from: "organizations",
localField: "context",
foreignField: "_id",
as: "context"
}
},
{
$lookup: {
from: "roles",
localField: "roles",
foreignField: "_id",
as: "roles_with_details"
}
},
{
$lookup: {
from: "service_modules",
localField: "context.enabled_service_modules",
foreignField: "_id",
as: "service_modules",
"let": {
"roles": "$roles"
},
"pipeline": [
{
"$match": {
"$expr": {
"$gt": [
{"$size": {"$setIntersection": ["$$roles", "$permissions"]}},
0
]
}
}
}
]
}
},
{
$project: {
"context.enabled_service_modules": 0,
"password": 0
}
}
])
Try it on mongoplayground.net.
When i run this query:
db.friendRequests.aggregate([
$lookup: {
from: "users",
localField: "author",
foreignField: "_id",
pipeline: [
{
$match: {
$expr: {
friend_id: new mongoose.Types.ObjectId(userid),
},
},
},
],
as: "userdata",
}
])
It returns every entry in the collection, but theres a pipeline in it. Then why is it not working?
Can you help me? Thanks!
Playground:
https://mongoplayground.net/p/Eh2j8lU4IQl
The friend_id field is present in the friendRequests collection (source for the aggregation) not the users collection which is the target for the $lookup. Therefore that predicate should come in a $match stage that precedes the $lookup:
db.friendRequests.aggregate([
{
$match: {
"friend_id": ObjectId("636a88de3e45346191cf4257")
}
},
{
$lookup: {
from: "users",
localField: "author",
foreignField: "_id",
as: "userdata"
}
}
])
See how it works in this playground example. Note that I changed inventory to users assuming that was just a typo in the collection name in the provided playground link.
Original answer
This syntax is incorrect:
$match: {
$expr: {
friend_id: new mongoose.Types.ObjectId(userid),
},
}
You should change it to either
$match: {
friend_id: new mongoose.Types.ObjectId(userid),
}
Or
$match: {
$expr: {
$eq: [
"$friend_id", new mongoose.Types.ObjectId(userid)
]
},
}
For mongodb version under 5.0 (Thanks for the remark #user20042973):
$lookup with localField and foreignField will ignore a pipeline. Remove them and add a let key in order to enable the pipeline.
i have 3 tables
1) Actor: actor_id, first_name, last_name
2) Film: film_id, title
3) Film_Actor: film_id, actor_id
Example document:
_id
:
60aedac769985522a024daca
actor_id
:
"1"
first_name
:
"Penelope"
last_name
:
"Guiness"
I wanto to result not only first_name, but last_name too. I'm facing problem with using concat in $group function.
my FULLY codes:
db.film.aggregate([
{
$lookup: {
from: "film_actor",
localField: "film_id",
foreignField: "film_id",
as: "film_actor"
}
},
{
$lookup: {
from: "actor",
localField: "film_actor.actor_id",
foreignField: "actor_id",
as: "actor"
}
},
{
$group: {
_id: "$film_id",
title: {"$first":"$title"},
name: {$push: "$actor.first_name"}
}
}
]);
Error report:
$concat only supports strings, not array
desired output:
id:"207"
title:"Dangerous Uptown"
name:Array
0:"Penelope Guiness"
1:"Mary Watson"
2:"Ralph Holts"
3:"Spencer Dani"
What you have done is appreciated, I've done some changes in your code
$lookup to join collections. I have started form Flim collection
$unwind to deconstruct the array
$group to reconstruct the array that we already deconstructed, this will
Since we have nested array we need to use $map to loop over them to collect the first name and lastname
The above stage will end up with again nested array, so we use $reduce to loop again and remove inner arrays using $setUnion
remove some duplicate entries, depends on your requirements
Here is the code
db.Film.aggregate([
{
$lookup: {
from: "Film_Actor",
localField: "film_id",
foreignField: "film_id",
as: "join_flim"
}
},
{ "$unwind": "$join_flim" },
{
$lookup: {
from: "Actor",
localField: "join_flim.actor_id",
foreignField: "actor_id",
as: "join_flim.join_actor"
}
},
{
$group: {
_id: "$_id",
title: { $first: "$title" },
join_flim: { $push: "$join_flim" }
}
},
{
"$project": {
title: 1,
actornames: {
$map: {
input: "$join_flim",
as: "f",
in: {
$map: {
input: "$$f.join_actor",
as: "a",
in: {
$concat: [ "$$a.first_name", " ", "$$a.last_name" ]
}
}
}
}
}
}
},
{
"$project": {
title: 1,
actornames: {
"$reduce": {
"input": "$actornames",
"initialValue": [],
"in": {
"$setUnion": [ "$$this", "$$value" ]
}
}
}
}
}
])
Working Mongo playground
I need to get the sells sum, just with the idCategory, any idea?
I have this 3 schemas in mongondb
category = [{"id":1,"name":"cat1"}, {"id":2,"name":"cat2"}]
product = [{"id":1,"name":"product1", "catId":1}, {"id":2,"name":"product2", "catId":2}]
sells = [{"id":1,"value":80, "productId":1, status:'active'}, {"id":2,"value":90, "productId":2, status:'Inactive'}]
MongoDB collections are not actual schemas, also, I'm assuming:
You're interpreting them as arrays (although they aren't), so you mean each of your Collections have 2 documents inside of them.
You want the sum of sales grouped by product category.
If those are the cases, what you want is a MongoDB Aggregate which you would run on your "sells" collection, "join" with the "product" collection and group by category id.
The base aggregate to do so would be along the following lines:
sells.aggregate([
{
$lookup: {
from: "product",
localField: "productId",
foreignField: "id",
as: "ProductData"
}
},
{
$unwind: "$ProductData"
},
{
$group: {
_id: "$ProductData.catId",
total: { $sum: "$value" }
}
}
]);
If you also want to fetch the Category name after aggregating, all you need to do is insert another $lookup at the end of the pipeline joining with category collection:
sells.aggregate([
{
$lookup: {
from: "product",
localField: "productId",
foreignField: "id",
as: "ProductData"
}
},
{
$unwind: "$ProductData"
},
{
$group: {
_id: "$ProductData.catId",
total: { $sum: "$value" }
}
},
{
$lookup: {
from: "category",
localField: "_id",
foreignField: "id",
as: "CategoryData"
}
},
{
$unwind: "$CategoryData"
},
{
$project: {
name: "$CategoryData.name",
total: 1
}
}
]);
EDIT (adding new case request on comment):
db.getCollection('product').aggregate([
{
$lookup: {
from: "sells",
localField: "id",
foreignField: "productId",
as: "SalesData"
}
},
{
$unwind:
{
path: "$SalesData",
preserveNullAndEmptyArrays: true
}
},
{
$project: {
catId: 1,
value: "$SalesData.value"
}
},
{
$group: {
_id: "$catId",
total: { $sum: "$value" }
}
},
{
$lookup: {
from: "category",
localField: "_id",
foreignField: "id",
as: "CategoryData"
}
},
{
$unwind: "$CategoryData"
},
{
$project: {
name: "$CategoryData.name",
total: 1
}
}
]);
The idea here is to return an array of documents of the users' followers with the information if this user is a friend of that follower or not.
So far I have:
db.getCollection('users').aggregate([
{ $match: { _id: ObjectId("588877d82523b4395039910a") } },
{ $lookup: {
from: 'users',
localField: 'followers',
foreignField: '_id',
as: 's_followers'
}
},
{
$project: {
"s_followers._id": 1,
"s_followers.isFriend": {
$in: ["s_followers.id",
{ $setIntersection: ["$friends", "$followers"] }
]}
}
}
])
But the "s_followers.id" used in the $in operator doesn't seem to retrieve the _id information from the follower, so it always returns false.
When I use a ObjectId directly, I got the result I want:
"s_followers.isFriend": {
$in: [ObjectId("588877d82523b4395039910a"),
{ $setIntersection: ["$friends", "$followers"] }
]}
But I really need this ID to be a reference to the follower _id.
Expected result would be something like:
{
"_id" : ObjectId("588877d82523b4395039910a"),
"s_followers" : [
{
"_id" : ObjectId("5888687e56be8f172844d96f"),
"isFriend" : true
},
{
"_id" : ObjectId("5888ca27d79b8b03949a6e8c"),
"isFriend" : false
}
]
}
Thanks for your help!
UPD: A different approach (maybe easier), would be to use the ID of the user that I have (the one used on $match), but I would still need to get the reference for the follower's follower array
db.getCollection('users').aggregate([
{ $match: { _id: ObjectId("588877d82523b4395039910a") } },
{ $lookup: {
from: 'users',
localField: 'followers',
foreignField: '_id',
as: 's_followers'
}
}, {
$project: {
"firstName": 1,
"s_followers._id": 1,
"s_followers.firstName": 1,
"s_followers.followers": 1,
"s_followers.isFriend": { $in: [ObjectId("588877d82523b4395039910a"), "$s_followers.followers"] }
}
}
])
UPD2: The user data structure (the part that matters)
{
followers: [{ type: mongoose.Schema.Types.ObjectId, ref: "User" }],
friends: [{ type: mongoose.Schema.Types.ObjectId, ref: "User" }],
}
FOR VERSION 3.4.0+
Ok, just got it, I'll post here the code and my understanding of it:
db.getCollection('users').aggregate([
{ $match: { _id: ObjectId("588877d82523b4395039910a") } },
{ $lookup: {
from: 'users',
localField: 'followers',
foreignField: '_id',
as: 's_followers'
}
}, {
$project: {
"firstName": 1,
"s_followers._id": 1,
"s_followers.firstName": 1,
"s_followers.followers": 1,
}
}, {
$unwind: "$s_followers"
}, {
$project: {
"firstName": "$s_followers.firstName",
"isFriend": { $in: [ObjectId("588877d82523b4395039910a"), "$s_followers.followers"] }
}
}
])
My understanding of it:
$match: match the user I'm intended to get the followers of.
$lookup: found each follower detail
$project: select the information I want to return, also get the follower list of each follower
$unwind: create a different document for each follower
With the array unwinded, I can refer to the follower's followers array, and find my object id in it :)
In my example use followers friends list to check current user is friend or not. as {$arrayElemAt:["$s_followers.friends",0]} if want to find in followers then can use "$s_followers.followers"
You can try it.
db.getCollection('user').aggregate([
{ $match: { _id: ObjectId("5714d190e6128b7e7f8d9008") } },
{$unwind:"$followers"},
{ $lookup: {
from: 'user',
localField: 'followers',
foreignField: '_id',
as: 's_followers'
}
},
{$project:{
firstName:1,
s_followers:{$arrayElemAt:["$s_followers",0]},
isFriend:{$cond:[{
$anyElementTrue:{
$map: {"input": {$arrayElemAt:["$s_followers.friends",0]},
"as": "el",
"in": { "$eq": [ "$$el", "$_id" ] }
}
}
},true,false]}
}
},
{$group:{
_id:"$_id",
s_followers:{$push:{_id:"$s_followers._id",isFriend:"$isFriend"}}
}
}
])