Mongodb: Query the size of nested arrays - arrays

I have the following Schema:
Schema({
caller_address: {
type: String,
required: true,
},
traces: [[{
type: mongoose.Schema.Types.ObjectId,
ref: 'Call',
}]]
});
And I would like to retrieve only the objects that have traces with the Calls amount bigger than a specified number. In other words, the size of at least one nested array of traces should be bigger than a specified number.
I'm trying to use $elemMatch and $size, but no success. For now, I have this code:
CallerTraces.find({ 'traces' : { $elemMatch: { $size : { $gt: minTraceSize } }}})
Where minTraceSize is an int.
Could you guys help me?
I would really appreciate it!

Thanks for the sample data. My answer will be a raw MQL solution, not a mongoose solution, so some translation will be required.
I was able to insert two documents based on your comments in your post. I had to change the ObjectId of one of the two sample documents because your samples had the same primary key value and was generating a duplicate key exception.
Insert Sample Data
db.CallerTraces.insert(
{
"_id": ObjectId("6175e7ecc62cff004462d4a6"),
"traces": [
[
ObjectId("6175e7ecc62cff004462d4a4")
]
],
"caller_address": "0x4e204793bc4b8acee32edaf1fbba1f3ea45f7990"
})
db.CallerTraces.insert(
{
"_id": ObjectId("6175e7ecc62cff004462d4a7"),
"traces": [
[
ObjectId("6175e7ecc62cff004462d4a4"),
ObjectId("6175e7ecc62cff004462d4a4")
],
[
ObjectId("6175e7ecc62cff004462d4a4")
]
],
"caller_address": "0x4e204793bc4b8acee32edaf1fbba1f3ea45f7990"
})
If I want to find records having more than 0 items in the array traces I can issue the following:
Find more than zero traces
db.CallerTraces.find({ $expr: { $gt: [ { $size: "$traces" }, 0 ] } })
This returns the following:
Enterprise replSet [primary] barrydb> db.CallerTraces.find({ $expr: { $gt: [ { $size: "$traces" }, 0 ] } })
[
{
_id: ObjectId("6175e7ecc62cff004462d4a6"),
traces: [ [ ObjectId("6175e7ecc62cff004462d4a4") ] ],
caller_address: '0x4e204793bc4b8acee32edaf1fbba1f3ea45f7990'
},
{
_id: ObjectId("6175e7ecc62cff004462d4a7"),
traces: [
[
ObjectId("6175e7ecc62cff004462d4a4"),
ObjectId("6175e7ecc62cff004462d4a4")
],
[ ObjectId("6175e7ecc62cff004462d4a4") ]
],
caller_address: '0x4e204793bc4b8acee32edaf1fbba1f3ea45f7990'
}
]
Find more than 1 trace
If instead I want to find more than one trace I simply alter the query slightly:
db.CallerTraces.find({ $expr: { $gt: [ { $size: "$traces" }, 1 ] } })
... and this returns with the following results:
Enterprise replSet [primary] barrydb> db.CallerTraces.find({ $expr: { $gt: [ { $size: "$traces" }, 1 ] } })
[
{
_id: ObjectId("6175e7ecc62cff004462d4a7"),
traces: [
[
ObjectId("6175e7ecc62cff004462d4a4"),
ObjectId("6175e7ecc62cff004462d4a4")
],
[ ObjectId("6175e7ecc62cff004462d4a4") ]
],
caller_address: '0x4e204793bc4b8acee32edaf1fbba1f3ea45f7990'
}
]
Conclusion
When attempting to evaluate the length of the array within the query processor we must elect to use the $eval option as the syntax for MQL does not consider your use case. The $eval is somewhat of a catch-all option for things that do not fit nicely in the MQL framework.
UPDATE #1
OP introduced additional requirements. Rather than look at the count of the array, we must consider the count of the array within the array (nested inner array). Since the find() method with the $expr cannot evaluate nested arrays we must instead use the aggregation framework and unwind the outer array. This example stores the original form in a new field called original then replaces root after all the evaluation is complete. Since unwinding can result in duplicates in the pipeline we finalize with a $group to suppress duplicates.
Solution
db.CallerTraces.aggregate([
{
$addFields: {
"original._id": "$_id",
"original.traces": "$traces",
"original.caller_address": "$caller_address"
}
},
{
$unwind: "$traces"
},
{
$match: { $expr: { $gt: [ { $size: "$traces" }, 1 ] } }
},
{
$replaceRoot: { newRoot: "$original" }
},
{
$group:
{
_id: "$_id",
traces: { "$first": "$traces" },
caller_address: { "$first": "$caller_address" }
}
}
])

Related

MongoDB Track data changes

I want to track changes on MongoDB Documents. The big Challenge is that MongoDB has nested Documents.
Example
[
{
"_id": "60f7a86c0e979362a25245eb",
"email": "walltownsend#delphide.com",
"friends": [
{
"name": "Hancock Nelson"
},
{
"name": "Owen Dotson"
},
{
"name": "Cathy Jarvis"
}
]
}
]
after the update/change
[
{
"_id": "60f7a86c0e979362a25245eb",
"email": "walltownsend#delphide.com",
"friends": [
{
"name": "Daphne Kline" //<------
},
{
"name": "Owen Dotson"
},
{
"name": "Cathy Jarvis"
}
]
}
]
This is a very basic example of a highly expandable real world use chase.
On a SQL Based Database, I would suggest some sort of this solution.
The SQL way
users
_id
email
60f7a8b28db7c78b57bbc217
cathyjarvis#delphide.com
friends
_id
user_id
name
0
60f7a8b28db7c78b57bbc217
Hancock Nelson
1
60f7a8b28db7c78b57bbc217
Suarez Burt
2
60f7a8b28db7c78b57bbc217
Mejia Elliott
after the update/change
users
_id
email
60f7a8b28db7c78b57bbc217
cathyjarvis#delphide.com
friends
_id
user_id
name
0
60f7a8b28db7c78b57bbc217
Daphne Kline
1
60f7a8b28db7c78b57bbc217
Suarez Burt
2
60f7a8b28db7c78b57bbc217
Mejia Elliott
history
_id
friends_id
field
preUpdate
postUpdate
0
0
name
Hancock Nelson
Daphne Kline
If there is an update and the change has to be tracked before the next update, this would work for NoSQL as well. If there is a second Update, we have a second line in the SQL database and it't very clear. On NoSQL, you can make a list/array of the full document and compare changes during the indexes, but there is very much redundant information which hasn't changed.
Have a look at Set Expression Operators
$setDifference
$setEquals
$setIntersection
Be ware, these operators perform set operation on arrays, treating arrays as sets. If an array contains duplicate entries, they ignore the duplicate entries. They ignore the order of the elements.
In your example the update would result in
removed: [ {name: "Hancock Nelson" } ],
added: [ {name: "Daphne Kline" } ]
If the number of elements is always the same before and after the update, then you could use this one:
db.collection.insertOne({
friends: [
{ "name": "Hancock Nelson" },
{ "name": "Owen Dotson" },
{ "name": "Cathy Jarvis" }
],
updated_friends: [
{ "name": "Daphne Kline" },
{ "name": "Owen Dotson" },
{ "name": "Cathy Jarvis" }
]
})
db.collection.aggregate([
{
$set: {
difference: {
$map: {
input: { $range: [0, { $size: "$friends" }] },
as: "i",
in: {
$cond: {
if: {
$eq: [
{ $arrayElemAt: ["$friends", "$$i"] },
{ $arrayElemAt: ["$updated_friends", "$$i"] }
]
},
then: null,
else: {
old: { $arrayElemAt: ["$friends", "$$i"] },
new: { $arrayElemAt: ["$updated_friends", "$$i"] }
}
}
}
}
}
}
},
{
$set: {
difference: {
$filter: {
input: "$difference",
cond: { $ne: ["$$this", null] }
}
}
}
}
])

How to push a new element into existing array or create one if it doesn't exist yet in MongoDb?

I have a script creating a document, updating it and cleaning up.
db.getCollection('things').insert( { _id: 1001,
elemo: { a: "A", b: "B" },
histo: [ ] } } )
db.getCollection('things').update( { _id: 1001 },
[ { $set: {
histo: { $concatArrays: [ "$histo", ["$elemo"] ] } } } ] )
db.getCollection("things").find({ _id: 1001})
db.getCollection('things').remove({ _id: 1001 })
For certain reasons, I'd like to retain the functionality but can't guarantee that the originally empty array actually exists. I need to perform my update in such a way so that an existing array will get an additional element, while a non-existing (yet) one will get created (including said element).
db.getCollection('things').insert( { _id: 1001,
elemo: { a: "A", b: "B" } } )
db.getCollection('things').update( { _id: 1001 },
[ { $set: {
histo: { $concatArrays: [ "$histo", ["$elemo"] ] } } } ] )
db.getCollection("things").find({ _id: 1001})
db.getCollection('things').remove({ _id: 1001 })
The above only creates the field but its value is null, and so additional amendments to it result in null. I'm rather certain that it needs something more around $concatArrays but I can't figure out what. First, I thought I could go $ifnull but it didn't recognize that command (no error, no insertion, no coalescing, nothing).
You can make use of $cond or $ifNull (as you guessed) to check if the key exists or not inside the $concatArrays operator.
Using $cond Method
db.collection.update({
_id: 1001
},
[
{
$set: {
histo: {
"$concatArrays": [
{
"$cond": {
"if": {
"$not": [
"$histo"
]
},
"then": [],
"else": "$histo",
}
},
[
"$elemo"
],
],
}
}
}
])
Mongo Playground Sample Execution
Using $ifNull Method
db.collection.update({
_id: 1001
},
[
{
$set: {
histo: {
"$concatArrays": [
{
"$ifNull": [
"$histo",
[]
],
},
[
"$elemo"
],
],
}
}
}
])
Mongo Playground Sample Execution

Mongo DB find value in array of multiple nested arrays

I need to check if an ObjectId exists in a non nested array and in multiple nested arrays, I've managed to get very close using the aggregation framework, but got stuck in the very last step.
My documents have this structure:
{
"_id" : ObjectId("605ce5f063b1c2eb384c2b7f"),
"name" : "Test",
"attrs" : [
ObjectId("6058e94c3994d04d28639616"),
ObjectId("6058e94c3994d04d28639627"),
ObjectId("6058e94c3994d04d28639622"),
ObjectId("6058e94c3994d04d2863962e")
],
"variations" : [
{
"varName" : "Var1",
"attrs" : [
ObjectId("6058e94c3994d04d28639616"),
ObjectId("6058e94c3994d04d28639627"),
ObjectId("6058e94c3994d04d28639622"),
ObjectId("60591791d4d41d0a6817d23f")
],
},
{
"varName" : "Var2",
"attrs" : [
ObjectId("60591791d4d41d0a6817d22a"),
ObjectId("60591791d4d41d0a6817d255"),
ObjectId("6058e94c3994d04d28639622"),
ObjectId("60591791d4d41d0a6817d23f")
],
},
],
"storeId" : "9acdq9zgke49pw85"
}
Let´s say I need to check if this if this _id exists "6058e94c3994d04d28639616" in all arrays named attrs.
My aggregation query goes like this:
db.product.aggregate([
{
$match: {
storeId,
},
},
{
$project: {
_id: 0,
attrs: 1,
'variations.attrs': 1,
},
},
{
$project: {
attrs: 1,
vars: '$variations.attrs',
},
},
{
$unwind: '$vars',
},
{
$project: {
attr: {
$concatArrays: ['$vars', '$attrs'],
},
},
},
]);
which results in this:
[
{
attr: [
6058e94c3994d04d28639616,
6058e94c3994d04d28639627,
6058e94c3994d04d28639622,
6058e94c3994d04d2863962e,
6058e94c3994d04d28639616,
6058e94c3994d04d28639627,
6058e94c3994d04d28639622,
60591791d4d41d0a6817d23f,
60591791d4d41d0a6817d22a,
60591791d4d41d0a6817d255,
6058e94c3994d04d28639622,
60591791d4d41d0a6817d23f
]
},
{
attr: [
60591791d4d41d0a6817d22a,
60591791d4d41d0a6817d255,
6058e94c3994d04d28639622,
60591791d4d41d0a6817d23f,
6058e94c3994d04d28639624,
6058e94c3994d04d28639627,
6058e94c3994d04d28639628,
6058e94c3994d04d2863963e
]
}
]
Assuming I have two products in my DB, I get this result. Each element in the outermost array is a different product.
The last bit, which is checking for this key "6058e94c3994d04d28639616", I could not find a way to do it with $group, since I dont have keys to group on.
Or with $match, adding this to the end of the aggregation:
{
$match: {
attr: "6058e94c3994d04d28639616",
},
},
But that results in an empty array. I know that $match does not query arrays like this, but could not find a way to do it with $in as well.
Is this too complicated of a Schema? I cannot have the original data embedded, since it is mutable and I would not be happy to change all products if something changed.
Will this be very expensive if I had like 10000 products?
Thanks in advance
You are trying to compare string 6058e94c3994d04d28639616 with ObjectId. Convert the string to ObjectId using $toObjectId operator when perform $match operation like this:
{
$match: {
$expr: {
$in: [{ $toObjectId: "6058e94c3994d04d28639616" }, "$attr"]
}
}
}

MongoDB - How to get all documents not being referenced by any document in a different collection

We have two collections, Teams and Matches. Every time a Match is reported, a new document is saved in that collection and its added to an array in the Team documents (teams[i].matches).
A now solved bug in our system has caused that the new Matches document were not referenced in their respectives Teams documents.
Is there a query for Mongo DB 3.6.9 that can help us find the Matches not referenced in Teams?
An aggregation pipeline may help you, using $lookup.
$lookup fetches documents from "Teams" that match the pipeline's $match.
let: { match_id: "$_id" } create a variable match_id corresponding to Match's _id.
$match expression keeps only Teams with match_id into Team's matches array.
as: "matches" stores Team that validate previous $match.
Last $match after $lookup step keeps matches array that are empty (Matches with no Teams)
db.Matches.aggregate([
{
$lookup: {
from: "Teams",
let: { match_id: "$_id" },
pipeline: [{
$match: {
$expr: {
$in: [ "$$match_id", "$matches" ]
}
}
}],
as: "matches"
},
},
{
$match: {
$expr: { $eq: [{ $size: "$matches" }, 0] }
}
}
]);
This has been tested with the following collection template and Mongo playground online editor :
db={
"Matches": [
{ "_id": 0 },
{ "_id": 1 },
{ "_id": 2 },
{ "_id": 3 },
{ "_id": 4 },
],
"Teams": [
{
"_id": 0,
matches: [ 0, 3 ],
},
{
"_id": 1,
matches: [],
},
{
"_id": 2,
matches: [ 0 ],
},
{
"_id": 3,
matches: [ 2 ],
}
]
}
The resulting output is :
[
{
"_id": 1,
"matches": []
},
{
"_id": 4,
"matches": []
}
]

Update array content within another array that don't have key

I have mongoDB content as below:
[
{
"_id":{
"$oid":"57c6699711bd6a0976cabe8a"
},
"ID":"1111",
"FullName":"AAA",
"Category":[
{
"CategoryId":{
"$oid":"57c66ebedcba0f63c1ceea51"
},
"_id":{
"$oid":"57e38a8ad190ea1100649798"
},
"Value":[
{
"Name":""
}
]
},
{
"CategoryId":{
"$oid":"57c3df061eb1e59d3959cc40"
},
"_id":{
"$oid":"57e38a8ad190ea1100649797"
},
"Value":[
[
"111",
"XXXX",
"2005"
],
[
"1212",
"YYYY",
"2000"
],
[
"232323",
"ZZZZZ",
"1999"
]
]
}
]
},
{
"_id":{
"$oid":"57c6699711bd6a0976cabe8a"
},
"ID":"1111",
"FullName":"BBB",
"Category":[
{
"CategoryId":{
"$oid":"57c66ebedcba0f63c1ceea51"
},
"_id":{
"$oid":"57e38a8ad190ea1100649798"
},
"Value":[
{
"Name":""
}
]
},
{
"CategoryId":{
"$oid":"57c3df061eb1e59d3959cc40"
},
"_id":{
"$oid":"57e38a8ad190ea1100649797"
},
"Value":[
[
"4444",
"XXXX",
"2005"
],
[
"7777",
"GGGG",
"2000"
],
[
"8888",
"ZZZZZ",
"1999"
]
]
}
]
}
]
Here I have an array named 'Category' where it contains objects with different category id.
I need to
select a particular category id - '57c3df061eb1e59d3959cc40'
From the above selected Category, we get 'Value' array
From Value array need to find if the second value is equal to 'ZZZZZ' ie. value[1] == 'ZZZZZ'
And now, update the matched value arrays with a new value at the end
Eg:
[
"232323",
"ZZZZZ",
"1999"
]
should be updated to
[
"232323",
"ZZZZZ",
"1999",
"update1"
]
and
[
"8888",
"ZZZZZ",
"1999"
]
should be updated to
[
"8888",
"ZZZZZ",
"1999",
"update1"
]
I have tried as below:
resume.update({
"Category.CategoryId": new ObjectId('57c3df191eb1e59d3959cc43'),
"Category.Value.$.1": 'ZZZZZ'
},
{"$set": {"Category.Value.$.3": "update1"}
}, function(err, resData){
res.send(resData);
});
But, nothing gets updated. Its there any way to get this work. Please help to update the inner array.
Thanks in advance.
Your goal is not possible at the moment since you need to update two positional elements.
There is a JIRA trackable for the sort of behaviour you want here: https://jira.mongodb.org/browse/SERVER-831
It's a problem since you need to match two elements positions:
the Category element with the matched CategoryId
the Value element in the Value array of arrays
If one of these wouldn't be an array it would have been possible.
Anyway, Your update try above was wrong. IF this feature was possible (and it is not!!!) it would have been something like this:
db.resume.update(
{
Category: {
$elemMatch: {
CategoryId: ObjectId('57c3df061eb1e59d3959cc40'),
Value: {
$elemMatch: {
'1': 'ZZZZZ'
}
}
}
}
},
{
$push: {
'Category.$.Value.$': 'update1'
}
}
)
The positional $ operator should be used during the update and not the find like you did, and it will update the first element that matched the query.
Doing the above will return the error:
Too many positional (i.e. '$') elements found in path 'Category.$.Value.$'
Because of the missing feature I explained at the top.
So, currently (version 3.2) you will not be able to do this unless you change your schema.

Resources