Related
I want to track changes on MongoDB Documents. The big Challenge is that MongoDB has nested Documents.
Example
[
{
"_id": "60f7a86c0e979362a25245eb",
"email": "walltownsend#delphide.com",
"friends": [
{
"name": "Hancock Nelson"
},
{
"name": "Owen Dotson"
},
{
"name": "Cathy Jarvis"
}
]
}
]
after the update/change
[
{
"_id": "60f7a86c0e979362a25245eb",
"email": "walltownsend#delphide.com",
"friends": [
{
"name": "Daphne Kline" //<------
},
{
"name": "Owen Dotson"
},
{
"name": "Cathy Jarvis"
}
]
}
]
This is a very basic example of a highly expandable real world use chase.
On a SQL Based Database, I would suggest some sort of this solution.
The SQL way
users
_id
email
60f7a8b28db7c78b57bbc217
cathyjarvis#delphide.com
friends
_id
user_id
name
0
60f7a8b28db7c78b57bbc217
Hancock Nelson
1
60f7a8b28db7c78b57bbc217
Suarez Burt
2
60f7a8b28db7c78b57bbc217
Mejia Elliott
after the update/change
users
_id
email
60f7a8b28db7c78b57bbc217
cathyjarvis#delphide.com
friends
_id
user_id
name
0
60f7a8b28db7c78b57bbc217
Daphne Kline
1
60f7a8b28db7c78b57bbc217
Suarez Burt
2
60f7a8b28db7c78b57bbc217
Mejia Elliott
history
_id
friends_id
field
preUpdate
postUpdate
0
0
name
Hancock Nelson
Daphne Kline
If there is an update and the change has to be tracked before the next update, this would work for NoSQL as well. If there is a second Update, we have a second line in the SQL database and it't very clear. On NoSQL, you can make a list/array of the full document and compare changes during the indexes, but there is very much redundant information which hasn't changed.
Have a look at Set Expression Operators
$setDifference
$setEquals
$setIntersection
Be ware, these operators perform set operation on arrays, treating arrays as sets. If an array contains duplicate entries, they ignore the duplicate entries. They ignore the order of the elements.
In your example the update would result in
removed: [ {name: "Hancock Nelson" } ],
added: [ {name: "Daphne Kline" } ]
If the number of elements is always the same before and after the update, then you could use this one:
db.collection.insertOne({
friends: [
{ "name": "Hancock Nelson" },
{ "name": "Owen Dotson" },
{ "name": "Cathy Jarvis" }
],
updated_friends: [
{ "name": "Daphne Kline" },
{ "name": "Owen Dotson" },
{ "name": "Cathy Jarvis" }
]
})
db.collection.aggregate([
{
$set: {
difference: {
$map: {
input: { $range: [0, { $size: "$friends" }] },
as: "i",
in: {
$cond: {
if: {
$eq: [
{ $arrayElemAt: ["$friends", "$$i"] },
{ $arrayElemAt: ["$updated_friends", "$$i"] }
]
},
then: null,
else: {
old: { $arrayElemAt: ["$friends", "$$i"] },
new: { $arrayElemAt: ["$updated_friends", "$$i"] }
}
}
}
}
}
}
},
{
$set: {
difference: {
$filter: {
input: "$difference",
cond: { $ne: ["$$this", null] }
}
}
}
}
])
Help to "flatten" (to pull nested fields at same level as document's fields) a mongodb document in a query
//this is "anagrafiche" collection
[{
"name": "tizio"
,"surname": "semproni"
,"birthday": "01/02/1923"
,"home": {
"road": "via"
,"roadname": "bianca"
,"roadN": 12
,"city": "rome"
,"country": "italy"
}
},
{
"name": "caio"
,"surname": "giulio"
,"birthday": "02/03/1932"
,"home": {
"road": "via"
,"roadname": "rossa"
,"roadN": 21
,"city": "milan"
,"country": "italy"
}
},
{
"name": "mario"
,"surname": "rossi"
// birthday is not present for this document
,"home": {
"road": "via"
,"roadname": "della pace"
,"roadN": 120
,"city": "rome"
,"country": "italy"
}
}
]
my query:
db.anagrafiche.aggregate([ {$match {"home.city": "rome"}}
{$project:{"name": 1, "surname":1, <an expression to flatten the address>, "birthday": 1, "_id":0}}
]
);
expected result:
{
,"name": "tizio"
,"surname": "semproni"
,"address": "via bianca 12 rome"
,"birthday": 01/02/1923
},{
,"name": "mario"
,"surname": "rossi"
,"address": "via della pace 120 rome"
,"birthday": NULL
}
You can use $objectToArray to get nested document keys and values and then use $reduce along with $concat to concatenate values dynamically:
db.collection.aggregate([
{
$project: {
_id: 0,
name: 1,
surname: 1,
birthday: 1,
address: {
$reduce: {
input: { $objectToArray: "$home" },
initialValue: "",
in: {
$concat: [
"$$value",
{ $cond: [ { $eq: [ "$$value", "" ] }, "", " " ] },
{ $toString: "$$this.v" }
]
}
}
}
}
}
])
Mongo Playground
I need to modify a document inside an array that is inside another array.
I know MongoDB doesn't support multiple '$' to iterate on multiple arrays at the same time, but they introduced arrayFilters for that.
See: https://jira.mongodb.org/browse/SERVER-831
MongoDB's sample code:
db.coll.update({}, {$set: {“a.$[i].c.$[j].d”: 2}}, {arrayFilters: [{“i.b”: 0}, {“j.d”: 0}]})
Input: {a: [{b: 0, c: [{d: 0}, {d: 1}]}, {b: 1, c: [{d: 0}, {d: 1}]}]}
Output: {a: [{b: 0, c: [{d: 2}, {d: 1}]}, {b: 1, c: [{d: 0}, {d: 1}]}]}
Here's how the documents are set:
{
"_id" : ObjectId("5a05a8b7e0ce3444f8ec5bd7"),
"name" : "support",
"contactTypes" : {
"nonWorkingHours" : [],
"workingHours" : []
},
"workingDays" : [],
"people" : [
{
"enabled" : true,
"level" : "1",
"name" : "Someone",
"_id" : ObjectId("5a05a8c3e0ce3444f8ec5bd8"),
"contacts" : [
{
"_id" : ObjectId("5a05a8dee0ce3444f8ec5bda"),
"retries" : "1",
"priority" : "1",
"type" : "email",
"data" : "some.email#email.com"
}
]
}
],
"__v" : 0
}
Here's the schema:
const ContactSchema = new Schema({
data: String,
type: String,
priority: String,
retries: String
});
const PersonSchema = new Schema({
name: String,
level: String,
priority: String,
enabled: Boolean,
contacts: [ContactSchema]
});
const GroupSchema = new Schema({
name: String,
people: [PersonSchema],
workingHours: { start: String, end: String },
workingDays: [Number],
contactTypes: { workingHours: [String], nonWorkingHours: [String] }
});
I need to update a contact. This is what I tried using arrayFilters:
Group.update(
{},
{'$set': {'people.$[i].contacts.$[j].data': 'new data'}},
{arrayFilters: [
{'i._id': mongoose.Types.ObjectId(req.params.personId)},
{'j._id': mongoose.Types.ObjectId(req.params.contactId)}]},
function(err, doc) {
if (err) {
res.status(500).send(err);
}
res.send(doc);
}
);
The document is never updated and I get this response:
{
"ok": 0,
"n": 0,
"nModified": 0
}
What am I doing wrong?
So the arrayFilters option with positional filtered $[<identifier>] does actually work properly with the development release series since MongoDB 3.5.12 and also in the current release candidates for the MongoDB 3.6 series, where this will actually be officially released. The only problem is of course is that the "drivers" in use have not actually caught up to this yet.
Re-iterating the same content I have already placed on Updating a Nested Array with MongoDB:
NOTE Somewhat ironically, since this is specified in the "options" argument for .update() and like methods, the syntax is generally compatible with all recent release driver versions.
However this is not true of the mongo shell, since the way the method is implemented there ( "ironically for backward compatibility" ) the arrayFilters argument is not recognized and removed by an internal method that parses the options in order to deliver "backward compatibility" with prior MongoDB server versions and a "legacy" .update() API call syntax.
So if you want to use the command in the mongo shell or other "shell based" products ( notably Robo 3T ) you need a latest version from either the development branch or production release as of 3.6 or greater.
All this means is that the current "driver" implementation of .update() actually "removes" the necessary arguments with the definition of arrayFilters. For NodeJS this will be addressed in the 3.x release series of the driver, and of course "mongoose" will then likely take some time after that release to implement it's own dependencies on the updated driver, which would then no longer "strip" such actions.
You can however still run this on a supported server instance, by dropping back to the basic "update command" syntax usage, since this bypassed the implemented driver method:
const mongoose = require('mongoose'),
Schema = mongoose.Schema,
ObjectId = mongoose.Types.ObjectId;
mongoose.Promise = global.Promise;
mongoose.set('debug',true);
const uri = 'mongodb://localhost/test',
options = { useMongoClient: true };
const contactSchema = new Schema({
data: String,
type: String,
priority: String,
retries: String
});
const personSchema = new Schema({
name: String,
level: String,
priority: String,
enabled: Boolean,
contacts: [contactSchema]
});
const groupSchema = new Schema({
name: String,
people: [personSchema],
workingHours: { start: String, end: String },
workingDays: { type: [Number], default: undefined },
contactTypes: {
workingHours: { type: [String], default: undefined },
contactTypes: { type: [String], default: undefined }
}
});
const Group = mongoose.model('Group', groupSchema);
function log(data) {
console.log(JSON.stringify(data, undefined, 2))
}
(async function() {
try {
const conn = await mongoose.connect(uri,options);
// Clean data
await Promise.all(
Object.entries(conn.models).map(([k,m]) => m.remove() )
);
// Create sample
await Group.create({
name: "support",
people: [
{
"_id": ObjectId("5a05a8c3e0ce3444f8ec5bd8"),
"enabled": true,
"level": "1",
"name": "Someone",
"contacts": [
{
"type": "email",
"data": "adifferent.email#example.com"
},
{
"_id": ObjectId("5a05a8dee0ce3444f8ec5bda"),
"retries": "1",
"priority": "1",
"type": "email",
"data": "some.email#example.com"
}
]
}
]
});
let result = await conn.db.command({
"update": Group.collection.name,
"updates": [
{
"q": {},
"u": { "$set": { "people.$[i].contacts.$[j].data": "new data" } },
"multi": true,
"arrayFilters": [
{ "i._id": ObjectId("5a05a8c3e0ce3444f8ec5bd8") },
{ "j._id": ObjectId("5a05a8dee0ce3444f8ec5bda") }
]
}
]
});
log(result);
let group = await Group.findOne();
log(group);
} catch(e) {
console.error(e);
} finally {
mongoose.disconnect();
}
})()
Since that sends the "command" directly through to the server, we see the expected update does in fact take place:
Mongoose: groups.remove({}, {})
Mongoose: groups.insert({ name: 'support', _id: ObjectId("5a06557fb568aa0ad793c5e4"), people: [ { _id: ObjectId("5a05a8c3e0ce3444f8ec5bd8"), enabled: true, level: '1', name: 'Someone', contacts: [ { type: 'email', data: 'adifferent.email#example.com', _id: ObjectId("5a06557fb568aa0ad793c5e5") }, { _id: ObjectId("5a05a8dee0ce3444f8ec5bda"), retries: '1', priority: '1', type: 'email', data: 'some.email#example.com' } ] } ], __v: 0 })
{ n: 1,
nModified: 1,
opTime:
{ ts: Timestamp { _bsontype: 'Timestamp', low_: 3, high_: 1510364543 },
t: 24 },
electionId: 7fffffff0000000000000018,
ok: 1,
operationTime: Timestamp { _bsontype: 'Timestamp', low_: 3, high_: 1510364543 },
'$clusterTime':
{ clusterTime: Timestamp { _bsontype: 'Timestamp', low_: 3, high_: 1510364543 },
signature: { hash: [Object], keyId: 0 } } }
Mongoose: groups.findOne({}, { fields: {} })
{
"_id": "5a06557fb568aa0ad793c5e4",
"name": "support",
"__v": 0,
"people": [
{
"_id": "5a05a8c3e0ce3444f8ec5bd8",
"enabled": true,
"level": "1",
"name": "Someone",
"contacts": [
{
"type": "email",
"data": "adifferent.email#example.com",
"_id": "5a06557fb568aa0ad793c5e5"
},
{
"_id": "5a05a8dee0ce3444f8ec5bda",
"retries": "1",
"priority": "1",
"type": "email",
"data": "new data" // <-- updated here
}
]
}
]
}
So right "now"[1] the drivers available "off the shelf" don't actually implement .update() or it's other implementing counterparts in a way that is compatible with actually passing through the necessary arrayFilters argument. So if you are "playing with" a development series or release candiate server, then you really should be prepared to be working with the "bleeding edge" and unreleased drivers as well.
But you can actually do this as demonstrated in any driver, in the correct form where the command being issued is not going to be altered.
[1] As of writing on November 11th 2017 there is no "official" release of MongoDB or the supported drivers that actually implement this. Production usage should be based on official releases of the server and supported drivers only.
I had a similar use case. But my second level nested array doesn't have a key. While most examples out there showcase an example with arrays having a key like this:
{
"id": 1,
"items": [
{
"name": "Product 1",
"colors": ["yellow", "blue", "black"]
}
]
}
My use case is like this, without the key:
{
"colors": [
["yellow"],
["blue"],
["black"]
]
}
I managed to use the arrayfilters by ommiting the label of the first level of the array nest. Example document:
db.createCollection('ProductFlow')
db.ProductFlow.insertOne(
{
"steps": [
[
{
"actionType": "dispatch",
"payload": {
"vehicle": {
"name": "Livestock Truck",
"type": "road",
"thirdParty": true
}
}
},
{
"actionType": "dispatch",
"payload": {
"vehicle": {
"name": "Airplane",
"type": "air",
"thirdParty": true
}
}
}
],
[
{
"actionType": "store",
"payload": {
"company": "Company A",
"is_supplier": false
}
}
],
[
{
"actionType": "sell",
"payload": {
"reseller": "Company B",
"is_supplier": false
}
}
]
]
}
)
In my case, I want to:
Find all documents that have any steps with payload.vehicle.thirdParty=true and actionType=dispatch
Update the actions set payload.vehicle.thirdParty=true only for the actions that have actionType=dispatch.
My first approach was withour arrayfilters. But it would create the property payload.vehicle.thirdParty=true inside the steps with actionType store and sell.
The final query that updated the properties only inside the steps with actionType=dispatch:
Mongo Shell:
db.ProductFlow.updateMany(
{"steps": {"$elemMatch": {"$elemMatch": {"payload.vehicle.thirdParty": true, "actionType": "dispatch"}}}},
{"$set": {"steps.$[].$[i].payload.vehicle.thirdParty": false}},
{"arrayFilters": [ { "i.actionType": "dispatch" } ], multi: true}
)
PyMongo:
query = {
"steps": {"$elemMatch": {"$elemMatch": {"payload.vehicle.thirdParty": True, "actionType": "dispatch"}}}
}
update_statement = {
"$set": {
"steps.$[].$[i].payload.vehicle.thirdParty": False
}
}
array_filters = [
{ "i.actionType": "dispatch" }
]
NOTE that I'm omitting the label on the first array at the update statement steps.$[].$[i].payload.vehicle.thirdParty. Most examples out there will use both labels because their objects have a key for the array. I took me some time to figure that out.
I'm new to nodejs and currently i'm developing a Web API using nodejs, express and mongoose. Can anyone help me on following mongoose query.
var EventSchema = new Schema({
event_title: {
type: String,
required: true
},
service_order: [{
_id:{
type: String
},
service : {
type: Schema.Types.ObjectId,
ref: "Service"
}
}]
});
I want to select a entire Event document but only with selected service_order items in that array
ex :-
this is a entire Event document
{
_id: "sample_id",
name: 'some name',
"service_order": [
{
"_id": "1"
"service": {
"_id": "59c005524d9c141fe0d95f15"
},
{
"_id": "2"
"service": {
"_id": "59c005524d9c141fe0d95f18"
},
{
"_id": "3"
"service": {
"_id": "59c005524d9c141fe0d95f18"
},
{
"_id": "4"
"service": {
"_id": "59c005524d9c141fe0d95f18"
}
],
}
But I want to execute a single mongoose query which can give this as the output
{
_id: "sample_id",
name: 'some name',
"service_order": [
{
"_id": "1"
"service": {
"_id": "59c005524d9c141fe0d95f15"
},
{
"_id": "2"
"service": {
"_id": "59c005524d9c141fe0d95f18"
}
],
}
Initially I know the Event id ("sample_id") and the ids of service_orders that i want to select ( ["1","2"] ).
You can use mongodb aggregate to achieve that
don't forget to replace sample_id and ["1" , "2"] with your dynamic data
Check the code below:
db.event.aggregate([
{
$match: { _id: "sample_id" }
},
{
$unwind: { path: "$service_order" }
},
{
$match : { 'service_order._id' : { $in : [ "1" , "2"]} }
},
{
$group : {
_id: { _id: "$_id", "name": "$name"},
service_order: { $addToSet: "$service_order" },
}
},
{
$project : {
_id : "$_id._id",
name : "$_id.name",
service_order : 1
}
}
])
Given my Profile data looks like below, I want to find the profile for combination of userName and productId
and only return the profile with the respective contract for this product.
{
"firstName": "John",
"lastName": "Doe",
"userName": "john.doe#gmail.com",
"language": "NL",
"timeZone": "Europe/Amsterdam",
"contracts": [
{
"contractId": "DEMO1-CONTRACT",
"productId": "ticket-api",
"startDate": ISODate('2016-06-29T09:06:42.391Z'),
"roles": [
{
"name": "Manager",
"permissions": [
{
"activity": "ticket",
"permission": "createTicket"
},
{
"activity": "ticket",
"permission": "updateTicket"
},
{
"activity": "ticket",
"permission": "closeTicket"
}
]
}
]
},
{
"contractId": "DEMO2-CONTRACT",
"productId": "comment-api",
"startDate": ISODate('2016-06-29T10:27:45.899Z'),
"roles": [
{
"name": "Manager",
"permissions": [
{
"activity": "comment",
"permission": "createComment"
},
{
"activity": "comment",
"permission": "updateComment"
},
{
"activity": "comment",
"permission": "deleteComment"
}
]
}
]
}
]
}
I managed to find the solution how to do this from the command line. But I don't seem to find a way how to accomplish this with Morphia (latest version).
db.Profile.aggregate([
{ $match: {"userName": "john.doe#gmail.com"}},
{ $project: {
contracts: {$filter: {
input: '$contracts',
as: 'contract',
cond: {$eq: ['$$contract.productId', "ticket-api"]}
}}
}}
])
This is what I have so far. Any help is most appreciated
Query<Profile> matchQuery = getDatastore().createQuery(Profile.class).field(Profile._userName).equal(userName);
getDatastore()
.createAggregation(Profile.class)
.match(matchQuery)
.project(Projection.expression(??))
Note... meanwhile I found another solution which does not use an aggregation pipeline.
public Optional<Profile> findByUserNameAndContractQuery(String userName, String productId) {
DBObject contractQuery = BasicDBObjectBuilder.start(Contract._productId, productId).get();
Query<Profile> query =
getDatastore()
.createQuery(Profile.class)
.field(Profile._userName).equal(userName)
.filter(Profile._contracts + " elem", contractQuery)
.retrievedFields(true, Profile._contracts + ".$");
return Optional.ofNullable(query.get());
}
I finally found the best way (under assumption I only want to return max. 1 element from array) to filter embedded array.
db.Profile.aggregate([
{ $match: {"userName": "john.doe#gmail.com"}},
{ $unwind: "$contracts"},
{ $match: {"contracts.productId": "comment-api"}}
])
To match according to your first design you could try the projection settings with morphia aggregation pipeline.
Query<Profile> matchQuery = getDatastore().createQuery(Profile.class).field(Profile._userName).equal(userName);
getDatastore()
.createAggregation(Profile.class)
.match(matchQuery)
.project(Projection.expression("$filter", new BasicDBObject()
.append("input", "$contracts")
.append("as", "contract")
.append("cond", new BasicDBObject()
.append("$eq", Arrays.asList('$$contract.productId', "ticket-api")));
Also see the example written by the morphia crew around line 88 at https://github.com/mongodb/morphia/blob/master/morphia/src/test/java/org/mongodb/morphia/aggregation/AggregationTest.java.