Find mongo records with duplicated values in array field - arrays

Consider this mongo collection:
[{
"_id": {
"s": "HU",
"k": 1
},
"boxed": {
"values": [{
"s": "NL",
"k": 2
},
{
"s": "BR",
"k": 3
},
{
"s": "NL",
"k": 2
}
]
}
},
{
"_id": {
"s": "FR",
"k": 2
},
"boxed": {
"values": [{
"s": "SE",
"k": 99
}]
}
},
{
"_id": {
"s": "UA",
"k": 14
},
"boxed": {}
}
]
I'm basically trying to find the records that have duplicated boxed.values. One such example would be the first one, where NL*2 is repeated twice.
My first idea was to project the original size of the values array, use $map to turn that array of objects into an array of strings (such as $map: { input: "$boxed.values", in: { $concat: ["$$this.s", "*", "$$this.k"] } }) and then remove the duplicates out of the array of strings, so i can compare the original size with the dupe-removed one. If the size would differ, then it would mean that record has duplicates.
However, it seems that there's no way in mongo (or at least I have not found one) to easily remove duplicated values out of an array of strings.
Any ideas?

You can do something like this,
[
{
$unwind: "$boxed.values"
},
{
$group: {
_id: "$_id",
"values": {
$addToSet: "$boxed.values"
}
}
},
{
$addFields: {
"boxed.values": "$values"
}
}
]
Working Mongo playground. $addToSet is an array operation which removes duplicates

Related

MongoDB Track data changes

I want to track changes on MongoDB Documents. The big Challenge is that MongoDB has nested Documents.
Example
[
{
"_id": "60f7a86c0e979362a25245eb",
"email": "walltownsend#delphide.com",
"friends": [
{
"name": "Hancock Nelson"
},
{
"name": "Owen Dotson"
},
{
"name": "Cathy Jarvis"
}
]
}
]
after the update/change
[
{
"_id": "60f7a86c0e979362a25245eb",
"email": "walltownsend#delphide.com",
"friends": [
{
"name": "Daphne Kline" //<------
},
{
"name": "Owen Dotson"
},
{
"name": "Cathy Jarvis"
}
]
}
]
This is a very basic example of a highly expandable real world use chase.
On a SQL Based Database, I would suggest some sort of this solution.
The SQL way
users
_id
email
60f7a8b28db7c78b57bbc217
cathyjarvis#delphide.com
friends
_id
user_id
name
0
60f7a8b28db7c78b57bbc217
Hancock Nelson
1
60f7a8b28db7c78b57bbc217
Suarez Burt
2
60f7a8b28db7c78b57bbc217
Mejia Elliott
after the update/change
users
_id
email
60f7a8b28db7c78b57bbc217
cathyjarvis#delphide.com
friends
_id
user_id
name
0
60f7a8b28db7c78b57bbc217
Daphne Kline
1
60f7a8b28db7c78b57bbc217
Suarez Burt
2
60f7a8b28db7c78b57bbc217
Mejia Elliott
history
_id
friends_id
field
preUpdate
postUpdate
0
0
name
Hancock Nelson
Daphne Kline
If there is an update and the change has to be tracked before the next update, this would work for NoSQL as well. If there is a second Update, we have a second line in the SQL database and it't very clear. On NoSQL, you can make a list/array of the full document and compare changes during the indexes, but there is very much redundant information which hasn't changed.
Have a look at Set Expression Operators
$setDifference
$setEquals
$setIntersection
Be ware, these operators perform set operation on arrays, treating arrays as sets. If an array contains duplicate entries, they ignore the duplicate entries. They ignore the order of the elements.
In your example the update would result in
removed: [ {name: "Hancock Nelson" } ],
added: [ {name: "Daphne Kline" } ]
If the number of elements is always the same before and after the update, then you could use this one:
db.collection.insertOne({
friends: [
{ "name": "Hancock Nelson" },
{ "name": "Owen Dotson" },
{ "name": "Cathy Jarvis" }
],
updated_friends: [
{ "name": "Daphne Kline" },
{ "name": "Owen Dotson" },
{ "name": "Cathy Jarvis" }
]
})
db.collection.aggregate([
{
$set: {
difference: {
$map: {
input: { $range: [0, { $size: "$friends" }] },
as: "i",
in: {
$cond: {
if: {
$eq: [
{ $arrayElemAt: ["$friends", "$$i"] },
{ $arrayElemAt: ["$updated_friends", "$$i"] }
]
},
then: null,
else: {
old: { $arrayElemAt: ["$friends", "$$i"] },
new: { $arrayElemAt: ["$updated_friends", "$$i"] }
}
}
}
}
}
}
},
{
$set: {
difference: {
$filter: {
input: "$difference",
cond: { $ne: ["$$this", null] }
}
}
}
}
])

Multikey partial index not used with elemMatch

Consider the following document format which has an array field tasks holding embedded documents
{
"foo": "bar",
"tasks": [
{
"status": "sleep",
"id": "1"
},
{
"status": "active",
"id": "2"
}
]
}
There exists a partial index on key tasks.id
{
"v": 2,
"unique": true,
"key": {
"tasks.id": 1
},
"name": "tasks.id_1",
"partialFilterExpression": {
"tasks.id": {
"$exists": true
}
},
"ns": "zardb.quxcollection"
}
The following $elemMatch query with multiple conditions on the same array element
db.quxcollection.find(
{
"tasks": {
"$elemMatch": {
"id": {
"$eq": "1"
},
"status": {
"$nin": ["active"]
}
}
}
}).explain()
does not seem to use the index
"winningPlan": {
"stage": "COLLSCAN",
"filter": {
"tasks": {
"$elemMatch": {
"$and": [{
"id": {
"$eq": "1"
}
},
{
"status": {
"$not": {
"$eq": "active"
}
}
}
]
}
}
},
"direction": "forward"
}
How can I make the above query use the index? The index does seem to be used via dot notation
db.quxcollection.find({"tasks.id": "1"})
however I need the same array element to match multiple conditions which includes the status field, and the following does not seem to be equivalent to the above $elemMatch based query
db.quxcollection.find({
"tasks.id": "1",
"tasks.status": { "$nin": ["active"] }
})
The way the partial indexes work is it uses the path as a key. With $elemMatch you don't have the path explicitly in the query. If you check it with .explain("allPlansExecution") it is not even considered by the query planner.
To benefit from the index you can specify the path in the query:
db.quxcollection.find(
{
"tasks.id": "1",
"tasks": {
"$elemMatch": {
"id": {
"$eq": "1"
},
"status": {
"$nin": ["active"]
}
}
}
}).explain()
It duplicates part of the elemMatch condition, so the index will be used to get all documents containing tasks of specific id, then it will filter out documents with "active" tasks at fetch stage. I must admit the query doesn't look nice, so may be add some comments to the code with explanations.

Get frequency for multiple elements in all documents inside a collection mongodb

So heres my problem.
I am new to mongodb and have a collection which documents are saved like this:
{
"_id": {
"$oid": "60626db173b4ca321c02ee3e"
},
"year": "2021",
"name": "Book 1",
"authors": ["Joe, B", "Jessica, K"],
"createdAt": {
"$date": "2021-03-30T00:15:45.859Z"
}
},
{
"_id": {
"$oid": "60626db173b4ca321c02ee4e"
},
"year": "2021",
"authors": ["Carl, B", "Jessica, K"],
"name": "Book 2"
"createdAt": {
"$date": "2021-03-30T00:15:45.859Z"
}
},
I need to get both the frequency of all authors and the years of the books.
The expected result would be something like this (as long as i can get each element frequency it doesn't really matter how the results are returned):
{
"authors": {
"Joe, B": 1,
"Carl, B": 1,
"Jessica, K": 2
},
"year": {
"2021": 2
}
}
I've seen this thread How to count occurence of each value in array? which does the job in one array but i have no idea if its possible to adapt to get the frequency of multiple elements (year, authors) at the same time or how to do it.
I appreciate any help. Thank you.
Demo - https://mongoplayground.net/p/95JtQEThxvV
$group by year $push authors into the array get $sum count of the year occurrence, $unwind into individuals documents.
$group by authors and get $sum count of the author occurrence
$group by null to combine all documents, use $addToSet to push unique values and convert $arrayToObject to get final output in $project
$first
db.collection.aggregate([
{
$group: {
_id: { year: "$year" },
authors: { $push: "$authors" },
yearCount: { $sum: 1 }
}
},
{ $unwind: "$authors" },
{ $unwind: "$authors"},
{
$group: {
_id: { author: "$authors" },
year: { $first: "$_id.year" },
yearCount: { $first: "$yearCount" },
authors: { $push: "$authors" },
authorCount: { $sum: 1 }
}
},
{
"$group": {
_id: null,
years: {
$addToSet: { k: "$year", v: "$yearCount" }
},
authors: {
$addToSet: { k: "$_id.author", v: "$authorCount" }
}
}
},
{
$project: {
_id: 0,
years: { $arrayToObject: "$years" },
authors: { $arrayToObject: "$authors" }
}
}
])
Demo 2 - For author count grouped by year- https://mongoplayground.net/p/_elnjmknroF

Avoid empty array elements in mongo db

How to avoid empty array while filtering results while querying a collection in MongoDb
[
{
"_id": ObjectId("5d429786bd7b5f4ae4a64790"),
"extensions": {
"outcome": "success",
"docType": "ABC",
"Roll No": "1"
},
"data": [
{
"Page1": [
{
"heading": "LIST",
"content": [
{
"text": "<b>12345</b>"
},
],
}
],
"highlights": [
{
"name": "ABCD",
"text": "EFGH",
}
],
"marks": [
{
"revision": "revision 1",
"Score": [
{
"maths": "100",
"science": "40",
"history": "90"
},
{
"lab1": "25",
"lab2": "25"
}
],
"Result": "Pass"
},
{
"revision": "revision 1",
"Score": [
{
"maths": "100",
"science": "40"
},
{
"lab1": "25",
"lab2": "25"
}
],
"Result": "Pass"
}
]
}
]
}
]
I am looking for results that has only "history" marks in the score array.
I tried the following query (in mongo 3.6.10) but it returns empty score array as well the array that has history as well
db.getCollection('student_scores').find({
"data.marks.score.history": {
$not: {
$type: 10
},
$exists: true
}
},
{
"extensions.rollNo": 1,
"data.marks.score.history": 1
})
Desired output is
{
"extensions": {
"rollNo": "1"
},
"data": [
{
"marks": [
{
"Score": [
{
"history": "90"
}
]
}
]
}
]
}
I used something like the following;
db.getCollection('student_scores').aggregate([
{
$unwind: "$data"
},
{
$unwind: "$data.marks"
},
{
$unwind: "$data.marks.Score"
},
{
$match: {
"data.marks.Score.history": {
$exists: true,
$not: {
$type: 10
}
}
}
},
{
$project: {
"extensions.Roll No": 1,
"data.marks.Score.history": 1
}
},
{
$group: {
_id: "$extensions.Roll No",
history_grades: {
$push: "$data.marks.Score.history"
}
}
}
])
where I got the following result with your input (I think more readable than your expected output);
[
{
"_id": "1",
"history_grades": [
"90"
]
}
]
where _id represents "extensions.Roll No" value for any given data set.
What do you think?
check with a bigger input on mongoplayground
OK, so I still think the data design here with the Score array is a little off but here is solution that will ensure that a Score array contains only 1 entry and that entry is for a key of history. We use dotpath array diving as a trick to get to the value of history.
c = db.foo.aggregate([
{$unwind: "$data"}
,{$unwind: "$data.marks"}
,{$project: {
result: {$cond: [
{$and: [ // if
{$eq: [1, {$size: "$data.marks.Score"}]}, // Only 1 item...
// A little trick! $data.marks.Score.history will resolve to an *array*
// of the values associated with each object in $data.marks.Score (the parent
// array) having a key of history. BUT: As it resolves, if there is no
// field for that key, nothing is added to resolution vector -- not even a null.
// This means the resolved array could
// be **shorter** than the input. FOr example:
// > db.foo.insert({"x":[ {b:2}, {a:3,b:4}, {b:7}, {a:99} ]});
// WriteResult({ "nInserted" : 1 })
// > db.foo.aggregate([ {$project: {z: "$x.b", n: {$size: "$x.b"}} } ]);
// { "z" : [ 2, 4, 7 ], "n" : 3 }
// > db.foo.aggregate([ {$project: {z: "$x.a", n: {$size: "$x.a"}} } ]);
// { "z" : [ 3, 99 ], "n" : 2 }
//
// You must be careful about this.
// But we also know this resolved vector is of size 1 (see above) so we can go ahead and grab
// the 0th item and that becomes our output.
// Note that if we did not have the requirement of ONLY history, then we would not
// need the fancy $cond thing.
{$arrayElemAt: ["$data.marks.Score.history",0]}
]},
{$arrayElemAt: ["$data.marks.Score.history",0]}, // then (use value of history)
null ] } // else set null
,extensions: "$extensions" // just carry over extensions
}}
,{$match: {"result": {$ne: null} }} // only take good ones.

Query only for numbers in nested array

I am trying to get an average number of an key in a nested array inside a document, but not sure how to accomplish this.
Here is how my document looks like:
{
"_id": {
"$oid": "XXXXXXXXXXXXXXXXX"
},
"data": {
"type": "PlayerRoundData",
"playerId": "XXXXXXXXXXXXX",
"groupId": "XXXXXXXXXXXXXX",
"holeScores": [
{
"type": "RoundHoleData",
"points": 2
},
{
"type": "RoundHoleData",
"points": 13
},
{
"type": "RoundHoleData",
"points": 3
},
{
"type": "RoundHoleData",
"points": 1
},
{
"type": "RoundHoleData",
"points": 21
}
]
}
}
Now, the tricky part of this is that I only want the average of points for holeScores[0] of all documents with this playerid and this groupid.
Actually, the best solution would be collecting all documents with playerid and groupid and create a new array with the average of holeScores[0], holeScores[1], holeScores[2]... But if I only can get one array key at the time, that would be OK to :-)
Here is what I am thinking but not quit sure how to put it together:
var allScores = dbCollection('scores').aggregate(
{$match: {"data.groupId": groupId, "playerId": playerId}},
{$group: {
_id: playerId,
rounds: { $sum: 1 }
result: { $sum: "$data.scoreTotals.points" }
}}
);
Really hoping for help with this issue and thanks in advance :-)
You can use $unwind with includeArrayIndex to get index and then use $group to group by that index
dbCollection('scores').aggregate(
{
$match: { "data.playerId": "XXXXXXXXXXXXX", "data.groupId": "XXXXXXXXXXXXXX" }
},
{
$unwind: {
path: "$data.holeScores",
includeArrayIndex: "index"
}
},
{
$group: {
_id: "$index",
playerId: { $first: "data.playerId" },
avg: { $avg: "$data.holeScores.points" }
}
}
)
You can try below aggregation
db.collection.aggregate(
{ "$match": { "data.groupId": groupId, "data.playerId": playerId }},
{ "$group": {
"_id": null,
"result": {
"$sum": {
"$arrayElemAt": [
"$data.holeScores.points",
0
]
}
}
}}
)

Resources