I am new to the mongoDB aggregation pipeline and have a really basic question but could not find the answer anywhere. I would like to simply convert the following block:
"exclude" : [
{
"name" : "Accenture"
},
{
"name" : "Aon Consulting"
}
]
to:
"exclude" : [
"Accenture",
"Aon Consulting"
]
using the aggregation pipeline but I cannot seem to find how to do it even after going through the documentation on https://docs.mongodb.com/manual/reference/operator/aggregation/. Thanks for your help.
While #chridam's answer is correct, there is no need to use $map.
Simple $addFields/$project would be sufficient:
db.collection.aggregate([
{
$addFields: {
exclude : '$exclude.name'
}
}
])
You certainly were in the right direction in using the aggregation framework to handle the transformation. The main operator that maps the object keys in the array to an array of the key values only would be $map in this case.
Use it together within a $addFields pipeline to project the transformed field as follows:
db.collection.aggregate([
{
"$addFields": {
"exclude": {
"$map": {
"input": "$exclude",
"as": "el",
"in": "$$el.name"
}
}
}
}
])
In the above, the $addFields pipeline stage adds new fields to documents and if the name of the new field is the same as an existing field name (including _id), $addFields overwrites the existing value of that field with the value of the specified expression.
So essentially the above replaces the excludes array with the transformed array using $map. $map works by applying an expression to each element of the input array. The expression references each element individually with the variable name ($$el) specified in the as field and returns an array with the applied results.
Related
Passing an object to an existing field in $set or $addFields merges objects rather than replaces them, e.g.
https://mongoplayground.net/p/aXe-rExjCXr
// Collection
[
{
"_id": "123",
"options": {
"size": "Large",
"color": "Red"
}
}
]
// Aggregate
db.collection.aggregate([
{
$set: {
options: {
size: "Small"
}
}
}
]);
// Expect
[
{
"_id": "123",
"options": {
"size": "Small"
}
}
]
// Actual
[
{
"_id": "123",
"options": {
"size": "Small",
"color": "Red" // <-- Not expected?
}
}
]
(It get's even weirder with arrays)
Is it possible to have it behave like non-object values and simply replace the field?
For context, I want to use this in an aggregate update pipeline.
This is the expected behaviour, and as far as i know there is not plan to change, as far as i remembered there was a jira with this, but they closed it, meaning that it will not change i think.
$set/$addFields replace always except
array field and i add document => array with all members that document
document field and i add document => merge documents (this is your case here)
$project replace always except
array field and i add document => array with all members that document
Solutions
You can override this "weird" behaviour especially in case of
arrays, by $unset the old field first for example, and then $set
Based on the jira in the comment bellow, we can also use $literal to avoid this, but when we use $literal we have to be sure that we dont use expressions because they will not be evaluated.
(expressions like path references, variables, operators etc)
I am trying to update a value in the nested array but can't get it to work.
My object is like this
{
"_id": {
"$oid": "1"
},
"array1": [
{
"_id": "12",
"array2": [
{
"_id": "123",
"answeredBy": [], // need to push "success"
},
{
"_id": "124",
"answeredBy": [],
}
],
}
]
}
I need to push a value to "answeredBy" array.
In the below example, I tried pushing "success" string to the "answeredBy" array of the "123 _id" object but it does not work.
callback = function(err,value){
if(err){
res.send(err);
}else{
res.send(value);
}
};
conditions = {
"_id": 1,
"array1._id": 12,
"array2._id": 123
};
updates = {
$push: {
"array2.$.answeredBy": "success"
}
};
options = {
upsert: true
};
Model.update(conditions, updates, options, callback);
I found this link, but its answer only says I should use object like structure instead of array's. This cannot be applied in my situation. I really need my object to be nested in arrays
It would be great if you can help me out here. I've been spending hours to figure this out.
Thank you in advance!
General Scope and Explanation
There are a few things wrong with what you are doing here. Firstly your query conditions. You are referring to several _id values where you should not need to, and at least one of which is not on the top level.
In order to get into a "nested" value and also presuming that _id value is unique and would not appear in any other document, you query form should be like this:
Model.update(
{ "array1.array2._id": "123" },
{ "$push": { "array1.0.array2.$.answeredBy": "success" } },
function(err,numAffected) {
// something with the result in here
}
);
Now that would actually work, but really it is only a fluke that it does as there are very good reasons why it should not work for you.
The important reading is in the official documentation for the positional $ operator under the subject of "Nested Arrays". What this says is:
The positional $ operator cannot be used for queries which traverse more than one array, such as queries that traverse arrays nested within other arrays, because the replacement for the $ placeholder is a single value
Specifically what that means is the element that will be matched and returned in the positional placeholder is the value of the index from the first matching array. This means in your case the matching index on the "top" level array.
So if you look at the query notation as shown, we have "hardcoded" the first ( or 0 index ) position in the top level array, and it just so happens that the matching element within "array2" is also the zero index entry.
To demonstrate this you can change the matching _id value to "124" and the result will $push an new entry onto the element with _id "123" as they are both in the zero index entry of "array1" and that is the value returned to the placeholder.
So that is the general problem with nesting arrays. You could remove one of the levels and you would still be able to $push to the correct element in your "top" array, but there would still be multiple levels.
Try to avoid nesting arrays as you will run into update problems as is shown.
The general case is to "flatten" the things you "think" are "levels" and actually make theses "attributes" on the final detail items. For example, the "flattened" form of the structure in the question should be something like:
{
"answers": [
{ "by": "success", "type2": "123", "type1": "12" }
]
}
Or even when accepting the inner array is $push only, and never updated:
{
"array": [
{ "type1": "12", "type2": "123", "answeredBy": ["success"] },
{ "type1": "12", "type2": "124", "answeredBy": [] }
]
}
Which both lend themselves to atomic updates within the scope of the positional $ operator
MongoDB 3.6 and Above
From MongoDB 3.6 there are new features available to work with nested arrays. This uses the positional filtered $[<identifier>] syntax in order to match the specific elements and apply different conditions through arrayFilters in the update statement:
Model.update(
{
"_id": 1,
"array1": {
"$elemMatch": {
"_id": "12","array2._id": "123"
}
}
},
{
"$push": { "array1.$[outer].array2.$[inner].answeredBy": "success" }
},
{
"arrayFilters": [{ "outer._id": "12" },{ "inner._id": "123" }]
}
)
The "arrayFilters" as passed to the options for .update() or even
.updateOne(), .updateMany(), .findOneAndUpdate() or .bulkWrite() method specifies the conditions to match on the identifier given in the update statement. Any elements that match the condition given will be updated.
Because the structure is "nested", we actually use "multiple filters" as is specified with an "array" of filter definitions as shown. The marked "identifier" is used in matching against the positional filtered $[<identifier>] syntax actually used in the update block of the statement. In this case inner and outer are the identifiers used for each condition as specified with the nested chain.
This new expansion makes the update of nested array content possible, but it does not really help with the practicality of "querying" such data, so the same caveats apply as explained earlier.
You typically really "mean" to express as "attributes", even if your brain initially thinks "nesting", it's just usually a reaction to how you believe the "previous relational parts" come together. In reality you really need more denormalization.
Also see How to Update Multiple Array Elements in mongodb, since these new update operators actually match and update "multiple array elements" rather than just the first, which has been the previous action of positional updates.
NOTE Somewhat ironically, since this is specified in the "options" argument for .update() and like methods, the syntax is generally compatible with all recent release driver versions.
However this is not true of the mongo shell, since the way the method is implemented there ( "ironically for backward compatibility" ) the arrayFilters argument is not recognized and removed by an internal method that parses the options in order to deliver "backward compatibility" with prior MongoDB server versions and a "legacy" .update() API call syntax.
So if you want to use the command in the mongo shell or other "shell based" products ( notably Robo 3T ) you need a latest version from either the development branch or production release as of 3.6 or greater.
See also positional all $[] which also updates "multiple array elements" but without applying to specified conditions and applies to all elements in the array where that is the desired action.
I know this is a very old question, but I just struggled with this problem myself, and found, what I believe to be, a better answer.
A way to solve this problem is to use Sub-Documents. This is done by nesting schemas within your schemas
MainSchema = new mongoose.Schema({
array1: [Array1Schema]
})
Array1Schema = new mongoose.Schema({
array2: [Array2Schema]
})
Array2Schema = new mongoose.Schema({
answeredBy": [...]
})
This way the object will look like the one you show, but now each array are filled with sub-documents. This makes it possible to dot your way into the sub-document you want. Instead of using a .update you then use a .find or .findOne to get the document you want to update.
Main.findOne((
{
_id: 1
}
)
.exec(
function(err, result){
result.array1.id(12).array2.id(123).answeredBy.push('success')
result.save(function(err){
console.log(result)
});
}
)
Haven't used the .push() function this way myself, so the syntax might not be right, but I have used both .set() and .remove(), and both works perfectly fine.
I am trying to update a value in the nested array but can't get it to work.
My object is like this
{
"_id": {
"$oid": "1"
},
"array1": [
{
"_id": "12",
"array2": [
{
"_id": "123",
"answeredBy": [], // need to push "success"
},
{
"_id": "124",
"answeredBy": [],
}
],
}
]
}
I need to push a value to "answeredBy" array.
In the below example, I tried pushing "success" string to the "answeredBy" array of the "123 _id" object but it does not work.
callback = function(err,value){
if(err){
res.send(err);
}else{
res.send(value);
}
};
conditions = {
"_id": 1,
"array1._id": 12,
"array2._id": 123
};
updates = {
$push: {
"array2.$.answeredBy": "success"
}
};
options = {
upsert: true
};
Model.update(conditions, updates, options, callback);
I found this link, but its answer only says I should use object like structure instead of array's. This cannot be applied in my situation. I really need my object to be nested in arrays
It would be great if you can help me out here. I've been spending hours to figure this out.
Thank you in advance!
General Scope and Explanation
There are a few things wrong with what you are doing here. Firstly your query conditions. You are referring to several _id values where you should not need to, and at least one of which is not on the top level.
In order to get into a "nested" value and also presuming that _id value is unique and would not appear in any other document, you query form should be like this:
Model.update(
{ "array1.array2._id": "123" },
{ "$push": { "array1.0.array2.$.answeredBy": "success" } },
function(err,numAffected) {
// something with the result in here
}
);
Now that would actually work, but really it is only a fluke that it does as there are very good reasons why it should not work for you.
The important reading is in the official documentation for the positional $ operator under the subject of "Nested Arrays". What this says is:
The positional $ operator cannot be used for queries which traverse more than one array, such as queries that traverse arrays nested within other arrays, because the replacement for the $ placeholder is a single value
Specifically what that means is the element that will be matched and returned in the positional placeholder is the value of the index from the first matching array. This means in your case the matching index on the "top" level array.
So if you look at the query notation as shown, we have "hardcoded" the first ( or 0 index ) position in the top level array, and it just so happens that the matching element within "array2" is also the zero index entry.
To demonstrate this you can change the matching _id value to "124" and the result will $push an new entry onto the element with _id "123" as they are both in the zero index entry of "array1" and that is the value returned to the placeholder.
So that is the general problem with nesting arrays. You could remove one of the levels and you would still be able to $push to the correct element in your "top" array, but there would still be multiple levels.
Try to avoid nesting arrays as you will run into update problems as is shown.
The general case is to "flatten" the things you "think" are "levels" and actually make theses "attributes" on the final detail items. For example, the "flattened" form of the structure in the question should be something like:
{
"answers": [
{ "by": "success", "type2": "123", "type1": "12" }
]
}
Or even when accepting the inner array is $push only, and never updated:
{
"array": [
{ "type1": "12", "type2": "123", "answeredBy": ["success"] },
{ "type1": "12", "type2": "124", "answeredBy": [] }
]
}
Which both lend themselves to atomic updates within the scope of the positional $ operator
MongoDB 3.6 and Above
From MongoDB 3.6 there are new features available to work with nested arrays. This uses the positional filtered $[<identifier>] syntax in order to match the specific elements and apply different conditions through arrayFilters in the update statement:
Model.update(
{
"_id": 1,
"array1": {
"$elemMatch": {
"_id": "12","array2._id": "123"
}
}
},
{
"$push": { "array1.$[outer].array2.$[inner].answeredBy": "success" }
},
{
"arrayFilters": [{ "outer._id": "12" },{ "inner._id": "123" }]
}
)
The "arrayFilters" as passed to the options for .update() or even
.updateOne(), .updateMany(), .findOneAndUpdate() or .bulkWrite() method specifies the conditions to match on the identifier given in the update statement. Any elements that match the condition given will be updated.
Because the structure is "nested", we actually use "multiple filters" as is specified with an "array" of filter definitions as shown. The marked "identifier" is used in matching against the positional filtered $[<identifier>] syntax actually used in the update block of the statement. In this case inner and outer are the identifiers used for each condition as specified with the nested chain.
This new expansion makes the update of nested array content possible, but it does not really help with the practicality of "querying" such data, so the same caveats apply as explained earlier.
You typically really "mean" to express as "attributes", even if your brain initially thinks "nesting", it's just usually a reaction to how you believe the "previous relational parts" come together. In reality you really need more denormalization.
Also see How to Update Multiple Array Elements in mongodb, since these new update operators actually match and update "multiple array elements" rather than just the first, which has been the previous action of positional updates.
NOTE Somewhat ironically, since this is specified in the "options" argument for .update() and like methods, the syntax is generally compatible with all recent release driver versions.
However this is not true of the mongo shell, since the way the method is implemented there ( "ironically for backward compatibility" ) the arrayFilters argument is not recognized and removed by an internal method that parses the options in order to deliver "backward compatibility" with prior MongoDB server versions and a "legacy" .update() API call syntax.
So if you want to use the command in the mongo shell or other "shell based" products ( notably Robo 3T ) you need a latest version from either the development branch or production release as of 3.6 or greater.
See also positional all $[] which also updates "multiple array elements" but without applying to specified conditions and applies to all elements in the array where that is the desired action.
I know this is a very old question, but I just struggled with this problem myself, and found, what I believe to be, a better answer.
A way to solve this problem is to use Sub-Documents. This is done by nesting schemas within your schemas
MainSchema = new mongoose.Schema({
array1: [Array1Schema]
})
Array1Schema = new mongoose.Schema({
array2: [Array2Schema]
})
Array2Schema = new mongoose.Schema({
answeredBy": [...]
})
This way the object will look like the one you show, but now each array are filled with sub-documents. This makes it possible to dot your way into the sub-document you want. Instead of using a .update you then use a .find or .findOne to get the document you want to update.
Main.findOne((
{
_id: 1
}
)
.exec(
function(err, result){
result.array1.id(12).array2.id(123).answeredBy.push('success')
result.save(function(err){
console.log(result)
});
}
)
Haven't used the .push() function this way myself, so the syntax might not be right, but I have used both .set() and .remove(), and both works perfectly fine.
I'm trying to do exactly what the poster in this link was trying to accomplish. I have documents with the same structure as the poster; in my documents there is an array of objects, each with many keys. I want to bring back all objects (not just the first, as you can with an $elemMatch) in that array where a key's value matches my query. I want my query's result to simply be an array of objects, where there is a key in each object that matches my query. For example, in the case of the linked question, I would want to return an array of objects where "class":"s2". I would want returned:
"FilterMetric" : [
{
"min" : "0.00",
"max" : "16.83",
"avg" : "0.00",
"class" : "s2"
},
{
"min" : "0.00",
"max" : "16.83",
"avg" : "0.00",
"class" : "s2"
}
]
I tried all the queries in the answer. The first two queries bring back an empty array in robomongo. In the shell, the command does nothing and return me to the next line. Here's a screenshot of robomongo:
On the third query in the answer, I get an unexpected token for the line where "input" is.
I'm using MongoDB version 3.0.2. It appears as if the OP was successful with the answer, so I'm wondering if there is a version issue, or if I'm approaching this the wrong way.
The only problem with the answers in that question seems to be that they're using the wrong casing for FilterMetric. For example, this works:
db.sample.aggregate([
{ "$match": { "FilterMetric.class": "s2" } },
{ "$unwind": "$FilterMetric" },
{ "$match": { "FilterMetric.class": "s2" } },
{ "$group": {
"_id": "$_id",
"FilterMetric": { "$push": "$FilterMetric" }
}}
])
Inspired by this question in Server Fault
https://serverfault.com/questions/459042/mongoexport-csv-output-array-values
I'm using mongoexport to export some collections into CSV files, however when I try to target fields which are the last members of an array I cannot get it to export correctly.
Command I'm using
mongoexport -d db -c collection -fieldFile fields.txt --csv > out.csv
One item of my collection:
{
"id": 1,
"name": "example",
"date": [
{"date": ""},
{"date": ""},
],
"status": [
"true",
"false",
],
}
I can access to the first member of my array writing the fields like the following
name
id
date.0.date
status.0
Is there a way to acess the last item of my array without knowing the lenght of the array?
Because the following doesn't work:
name
id
date.-1.date
status.-1
Any idea of the correct notation? Or if it's simply not possible?
It's not possible to reference the last element of the array without knowing the length of the array, since the notation is array_field.index where the index is in [0, length - 1]. You could use the aggregation framework to create the view of the data that you want to export, save it temporarily into a collection with $out, and then mongoexport that. For example, for your documents you could do
db.collection.aggregate([
{ "$unwind" : "$date" },
{ "$group" : { "_id" : "$_id", "date" : { "$last" : "$date" } } },
{ "$out" : "temp-for-csv" }
])
in order to get just the last date for each document and output it to the collection temp-for-csv.
You can return just the last elements in an array with the $slice projection operator, but this isn't available in aggregation and mongoexport only takes a query specification, not a projection specification, since the --fields and --fieldFile option are supposed to suffice. Might be a good feature request to ask for using a query with a projection for mongoexport.