I am new to MongoDB and trying to execute a query. I have a company collection and company IDs array. I would like to get the results where attributes.0.ccode exist and attributes.0.ccode is not empty and will be checked within the ids provided in an array( cdata)
var query = Company.find({ _id: { $in: cdata } },{ "attributes.0.ccode": { $exists: true }, $and: [ { "attributes.0.ccode": { $ne: "" } } ] }).select({"attributes": 1}).sort({});
The error I am getting is
"$err": "Can't canonicalize query: BadValue Unsupported projection option: attributes.0.ccode: { $exists: true }",
"code": 17287
I think it's a bracketing issue but can't figure it out where.
Any help is highly appreciated.
In your code { _id: { $in: cdata } } is interpreted as query, and everything else, starting from ,{ "attributes.0.ccode": { $e.. as a Projection (which field to display). Try to refactor your code so _id: {$in ...} and the rest of the query belong to the same higher - level object. Something like this:
var query = Company.find({
_id: {
$in: cdata
},
"attributes.0.ccode": {
$exists: true
},
$and: [
{
"attributes.0.ccode": {
$ne: ""
}
}
]
}).select({"attributes": 1}).sort({});
Related
I'm struggling to write a Mongo UpdateMany statement that can reference and update an object within an array.
Here I create 3 documents. Each document has an array called innerArray always containing a single object, with a single date field.
use test;
db.innerArrayExample.insertOne({ _id: 1, "innerArray": [ { "originalDateTime" : ISODate("2022-01-01T01:01:01Z") } ]});
db.innerArrayExample.insertOne({ _id: 2, "innerArray": [ { "originalDateTime" : ISODate("2022-01-02T01:01:01Z") } ]});
db.innerArrayExample.insertOne({ _id: 3, "innerArray": [ { "originalDateTime" : ISODate("2022-01-03T01:01:01Z") } ]});
I want to add a new date field, based on the original date field, to end up with this:
{ _id: 1, "innerArray": [ { "originalDateTime" : ISODate("2022-01-01T01:01:01Z"), "copiedDateTime" : ISODate("2022-01-01T12:01:01Z") } ]}
{ _id: 2, "innerArray": [ { "originalDateTime" : ISODate("2022-01-02T01:01:01Z"), "copiedDateTime" : ISODate("2022-01-02T12:01:01Z") } ]}
{ _id: 3, "innerArray": [ { "originalDateTime" : ISODate("2022-01-03T01:01:01Z"), "copiedDateTime" : ISODate("2022-01-03T12:01:01Z") } ]}
In pseudo code I am saying take the originalDateTime, run it through a function and add a related copiedDateTime value.
For my specific use-case, the function I want to run strips the timezone from originalDateTime, then overwrites it with a new one, equivalent to the Java ZonedDateTime function withZoneSameLocal. Aka 9pm UTC becomes 9pm Brussels (therefore effectively 7pm UTC). The technical justification and methodology were answered in another Stack Overflow question here.
The part of the query I'm struggling with, is the part that updates/selects data from an element inside an array. In my simplistic example, for example I have crafted this query, but unfortunately it doesn't work:
This function puts copiedDateTime in the correct place... but doesn't evaluate the commands to manipulate the date:
db.innerArrayExample.updateMany({ "innerArray.0.originalDateTime" : { $exists : true }}, { $set: { "innerArray.0.copiedDateTime" : { $dateFromString: { dateString: { $dateToString: { "date" : "$innerArray.0.originalDateTime", format: "%Y-%m-%dT%H:%M:%S.%L" }}, format: "%Y-%m-%dT%H:%M:%S.%L", timezone: "Europe/Paris" }}});
// output
{
_id: 1,
innerArray: [
{
originalDateTime: ISODate("2022-01-01T01:01:01.000Z"),
copiedDateTime: {
'$dateFromString': {
dateString: { '$dateToString': [Object] },
format: '%Y-%m-%dT%H:%M:%S.%L',
timezone: 'Europe/Paris'
}
}
}
]
}
This simplified query, also has the same issue:
b.innerArrayExample.updateMany({ "innerArray.0.originalDateTime" : { $exists : true }}, { $set: { "innerArray.0.copiedDateTime" : "$innerArray.0.originalDateTime" }});
//output
{
_id: 1,
innerArray: [
{
originalDateTime: ISODate("2022-01-01T01:01:01.000Z"),
copiedDateTime: '$innerArray.0.originalDateTime'
}
]
}
As you can see this issue looks to be separate from the other stack overflow question. Instead of being able changing timezones, it's about getting things inside arrays to update.
I plan to take this query, create 70,000 variations of it with different location/timezone combinations and run it against a database with millions of records, so I would prefer something that uses updateMany instead of using Javascript to iterate over each row in the database... unless that's the only viable solution.
I have tried putting $set in square brackets. This changes the way it interprets everything, evaluating the right side, but causing other problems:
test> db.innerArrayExample.updateMany({ "_id" : 1 }, [{ $set: { "innerArray.0.copiedDateTime" : "$innerArray.0.originalDateTime" }}]);
//output
{
_id: 1,
innerArray: [
{
'0': { copiedDateTime: [] },
originalDateTime: ISODate("2022-01-01T01:01:01.000Z")
}
]
}
Above it seems to interpret .0. as a literal rather than an array element. (For my needs I know the array only has 1 item at all times). I'm at a loss finding an example that meets my needs.
I have also tried experimenting with the arrayFilters, documented on my mongo updateMany documentation but I cannot fathom how it works with objects:
test> db.innerArrayExample.updateMany(
... { },
... { $set: { "innerArray.$[element].copiedDateTime" : "$innerArray.$[element].originalDateTime" } },
... { arrayFilters: [ { "originalDateTime": { $exists: true } } ] }
... );
MongoServerError: No array filter found for identifier 'element' in path 'innerArray.$[element].copiedDateTime'
test> db.innerArrayExample.updateMany(
... { },
... { $set: { "innerArray.$[0].copiedDateTime" : "$innerArray.$[element].originalDateTime" } },
... { arrayFilters: [ { "0.originalDateTime": { $exists: true } } ] }
... );
MongoServerError: Error parsing array filter :: caused by :: The top-level field name must be an alphanumeric string beginning with a lowercase letter, found '0'
If someone can help me understand the subtleties of the Mongo syntax and help me back on to the right path I'd be very grateful.
You want to be using pipelined updates, the issue you're having with the syntax you're using is that it does not allow the usage of aggregation operators and document field values.
Here is a quick example on how to do it:
db.collection.updateMany({},
[
{
"$set": {
"innerArray": {
$map: {
input: "$innerArray",
in: {
$mergeObjects: [
"$$this",
{
copiedDateTime: "$$this.originalDateTime"
}
]
}
}
}
}
}
])
Mongo Playground
I have a collection in mongodb with a few million documents. there is an attribute(categories) that is an array that contains all the categories that a document belongs to. I am using following query to convert the array into a comma separated string to add it to SQL server through a spoon transformation.
for example
the document has ["a","b","c",...] and i need a,b,c,.... so i can pit it in a column
categories: {
$cond: [
{ $eq: [{ $type: "$categories" }, "array"] },
{
$trim: {
input: {
$reduce: {
input: "$categories",
initialValue: "",
in: { $concat: ["$$value", ",", "$$this"] }
}
}
}
},
"$categories"
]
}
when i run the query i get the following error and i cannot figure out what the problem is.
com.mongodb.MongoQueryException: Query failed with error code 16702 and error message '$concat only supports strings, not array' on server
a few documents had this attribute as string and not array so i added a type check. but still the issue is there. any help on how to narrow down the issue will be very appreciated.
A few other attributes were the same in the same collection and this query is working fine for the rest of them.
I don't see any problem in your aggregation. It shouldn't give this error. Can you try to update your mongodb version?
However, your aggregation is not working properly reduce wasn't working . I converted it to this:
db.collection.aggregate([
{
"$project": {
categories: {
$cond: [
{
$eq: [{ $type: "$categories" }, "array"]
},
{
'$reduce': {
'input': '$categories',
'initialValue': '',
'in': {
'$concat': [
'$$value',
{ '$cond': [{ '$eq': ['$$value', ''] }, '', ', '] },
'$$this'
]
}
}
},
"$categories"
]
}
}
}
])
Edit:
So, if you have nested arrays in the categories field. We can flat our arrays with unwind stage. So if you can add these 3 stages above the $project stage. Our aggregation will work.
{
"$unwind": "$categories"
},
{
"$unwind": "$categories"
},
{
"$group": {
_id: null,
categories: {
$push: "$categories"
}
}
},
Playground
I'm currently facing quite a strange issue, i'm trying to pull from my database some data, based on a $text search and taking into account whatever permissions my user has: my data look like the following:
{
"_id" : ObjectId("5fd0e0c3233c72895e6655c9"),
"Entity" :
{
"Groups" : null,
"Name" : "Terasse"
}
}
I'm doing an aggregation query to both input the search my user queries and it's permissions values, fully formatted, the final query look like this:
db.collection.aggregate([
{
$match: {
$text: {
$search: "Terasse",
$caseSensitive: false,
$diacriticSensitive: false
}
}
},
{
$match: {
$or: [
{
"Entity.Groups": {
"$exists": false
}
},
{
"Entity.Groups": {
"$eq": null
}
},
{
"Entity.Groups": {
"$eq": []
}
},
{
$expr: {
$anyElementTrue: {
$map: {
input: "$Entity.Groups",
as: "group",
in: {
$anyElementTrue: {
$map: {
input: [
"/"
],
as: "userGroup",
in: {
$eq: [
0,
{
$indexOfBytes: [
"$$group",
"$$userGroup"
]
}
]
}
}
}
}
}
}
}
}
]
}
}
])
For a quick explanation, it first does the $text match to find the "Terasse" word in my database,
then run a second match stage to verify that my user can access this data.
My second match stage has an $or, which will first check if the data is correctly formatted before doing a special check to see if my user can access this data.
As you can see, this $or statement is checking that the Groups field of my data is: non-existing, null, or empty.
In this latter case, I would like to return this data no matter what authorization my user have and thus, not executing the very last $expr part at all
This aggregation will work perfectly fine if my Data has "Groups": [ "/" ] for example, but will fail with this error otherwise:
uncaught exception: Error: command failed: {
"ok" : 0,
"errmsg" : "$anyElementTrue's argument must be an array, but is null",
"code" : 17041,
"codeName" : "Location17041"
} : aggregate failed :
From my understanding, this error will happen IF the query will execute till the $expr part AND my Groups field is equal to non existing OR null OR empty, while it should be impossible because the $or statement should return the data as soon as it detects one of the mentionned case.
Finally, the most troubling part is that this second match stage will work perfectly with no errors at all if the first stage IS NOT a $match stage with a $text search
I am completely clueless now, is there an mongo expert that could give me a hand understanding what's happening ?
Thank you.
EDIT : as requested in comments:
this document will not work with the mentioned query
{
"_id": {
"$oid": "5fd0e0c3233c72895e6655c9"
},
"Entity": {
"Groups": null,
"Name": "Terasse"
}
}
this document will work with the mentioned query
{
"_id": {
"$oid": "5fd0e0c3233c72895e6655c9"
},
"Entity": {
"Groups": [ "/" ],
"Name": "Terasse"
}
}
also note that you cannot use mongoplayground to test this, as it requires to create a $text index before-hand (afaik, there is no way to do this in mongoplayground)
EDIT 2:
I am starting to believe that the mongo query system is quite broken when including $text stage, i've reworked the query like this to make sure that it was not due to the $or somewhat not working, and yet, it is still having the same error:
db.collection.aggregate([
{
$match: {
$text: {
$search: "Terasse",
$caseSensitive: false,
$diacriticSensitive: false
}
}
},
{
$match: {
$or: [
{
"Entity.Groups": {
"$exists": false
}
},
{
"Entity.Groups": {
"$eq": null
}
},
{
"Entity.Groups": {
"$eq": []
}
},
{
$and: [
{
"Entity.Groups": {
"$type": "array"
}
},
{
$expr: {
$anyElementTrue: {
$map: {
input: "$Entity.Groups",
as: "group",
in: {
$anyElementTrue: {
$map: {
input: [
"/test"
],
as: "userGroup",
in: {
$eq: [
0,
{
$indexOfBytes: [
"$$group",
"$$userGroup"
]
}
]
}
}
}
}
}
}
}
}
]
}
]
}
}
])
As you can see in this new query, i'm adding an $and check TO MAKE SURE THAT "Entity.Groups" is indeed an array before moving to the $anyElementTrue section and yet, the same error applies.
FINAL EDIT
Thanks to Ray's answer: I've changed my query to the following:
db.collection.aggregate([
{
$match: {
$text: {
$search: "Terasse",
$caseSensitive: false,
$diacriticSensitive: false
}
}
},
{
$addFields: {
"groupsMissing": {
$eq: [
[],
{
$ifNull: [
"$Entity.Groups",
[]
]
}
]
}
}
},
{
$match: {
$or: [
{
"groupsMissing": true
},
{
$expr: {
$anyElementTrue: {
$map: {
input: "$Entity.Groups",
as: "group",
in: {
$anyElementTrue: {
$map: {
input: [
"/test"
],
as: "userGroup",
in: {
$eq: [
0,
{
$indexOfBytes: [
"$$group",
"$$userGroup"
]
}
]
}
}
}
}
}
}
}
}
]
}
}
])
I've used another stage with the $addField as Ray mentionned but remove some of the previous/obsolete stuff, it is now working smoothly, will report if any side-effects re-occurs.
As a final note, i'm still unsure why the previous query that I've did didn't work, and why that solution does, but it seems like adding another stage to the query doing the sanitize checks and then having the second stage only checking the sanitized bool IS WORKING !
Probably it is related to the way mongo is executing the query.
I believe different stages HAS to be ran in a sequential way by mongo which is what I initially expected (though $and should also do that, by the documentation)
Having everything on a single stage is probably making mongo run the query quite differently than written in an effort to optimize it ?
That's all I can guess.
You may want to use $addFields to project some helper fields to make your life easier.
Here is the code I try to modify your version as least as possible.
db.collection.aggregate([
{
$match: {
$text: {
$search: "Terasse",
$caseSensitive: false,
$diacriticSensitive: false
}
}
},
{
$addFields: {
// flag to indicate Entity.Groups is null/missing/empty array
"groupsMissing": {
$eq: [
[],
{
$ifNull: [
"$Entity.Groups",
[]
]
}
]
},
// make Entity.Groups an empty array to avoid $anyElementTrue error
"Entity.Groups": {
$ifNull: [
"$Entity.Groups",
[]
]
}
}
},
{
$match: {
$or: [
// part of the code can be shorten
{
"groupsMissing": true
},
// the code should be the same as your version for the rest
{
$and: [
{
"Entity.Groups": {
"$type": "array"
}
},
{
$expr: {
$anyElementTrue: {
$map: {
input: "$Entity.Groups",
as: "group",
in: {
$anyElementTrue: {
$map: {
input: [
"/test"
],
as: "userGroup",
in: {
$eq: [
0,
{
$indexOfBytes: [
"$$group",
"$$userGroup"
]
}
]
}
}
}
}
}
}
}
}
]
}
]
}
}
])
First, MongoDB provides arrays for storing lists of things. Splitting strings on separators in queries is 1) less performant and 2) more difficult than it needs to be.
With that said, I do not see anything in https://docs.mongodb.com/manual/reference/operator/query/or/ saying the clauses will be evaluated in the order given. Therefore,
while it should be impossible because the $or statement should return the data as soon as it detects one of the mentionned case.
... appears to be an incorrect assumption as to how MongoDB works.
Note that https://docs.mongodb.com/manual/reference/operator/query/and/ does reference short-circuit evaluation.
I have a Schema(Case) which has an array of Sub Schema(Assessment).
My Case Schema is as follows:
const caseSchema = new mongoose.Schema({
patient: {
type: String,
default: 'Password can\'t be empty'
}
description: {
type: String,
required: 'Case description can\'t be empty'
},
assessments: [assessmentSchema]
});
My Assessment Schema is as follows
const assessmentSchema = new mongoose.Schema({
doctor: {
type: String,
required: 'Doctor can\'t be empty'
},
doa: Date,
slots: String
});
My problem here is to find all the assessment documents in the entire collection(where the doctor and doa is specified). But as these details are available only in the Sub Schema, I have no idea on how to approach this problem. And I also cannot add a doctor field to the Case Schema as there might be many doctors in a case. I'm using mongoose. I searched many places on the internet and was not able to arrive at a solution. Please help me.
I think you need something like this.
This is an aggregate where $project stage is used to pass the documents with the requested fields.
The fields that pass to the next phase are the ones that match the filter.
Into the filter we use $eq condition to check 'doctor' and 'slots' fields.
Note that you have to use $ operator, that is the positional operator.
db.collection.aggregate([
{
"$project": {
"_id": 0,
"assessments": {
"$filter": {
"input": "$assessments",
"as": "ass",
"cond": {
"$eq": [
"$$ass.doctor",
"doctor1"
],
"$eq": [
"$$ass.slots",
"slots1"
]
}
}
}
}
}
])
Here is a Playground example
Edit: With this new query you can get all fields you want.
db.collection.aggregate([
{
"$project": {
"patient": 1,
"description": 1,
"assessments": {
"$filter": {
"input": "$assessments",
"as": "as",
"cond": {
"$eq": [
"$$as.doctor",
"doctor1"
],
"$eq": [
"$$as.slots",
"slots1"
]
}
}
}
}
}
])
Note there is only a little difference.
Into $project you can decide what fields you want to show/reset/... reference $project
So adding caseSchema into query fields do the works.
I have a Mongodb document that contains an an array that is deeply imbedded inside the document. In one of my action, I would like to return the entire document but filter out the elements of that array that don't match that criteria.
Here is some simplified data:
{
id: 123 ,
vehicles : [
{name: 'Mercedes', listed: true},
{name: 'Nissan', listed: false},
...
]
}
So, in this example I want the entire document but I want the vehicles array to only have objects that have the listed property set to true.
Solutions
Ideally, I'm looking for a solution using mongo's queries (e.g. `$unwind, $elemMatch, etc...) but I'm also using mongoose so solution that uses Mongoose is OK.
You could use aggregation framework like this:
db.test312.aggregate(
{$unwind:"$vehicles"},
{$match:{"vehicles.name":"Nissan"}},
{$group:{_id:"$_id",vehicles:{$push:"$vehicles"}}}
)
You can use $addToSet on the group after unwinding and matching by listed equals true.
Sample shell query:
db.collection.aggregate([
{
$unwind: "$vehicles"
},
{
$match: {
"vehicles.listed": {
$eq: true
}
}
},
{
$group: {
_id: "$id",
vehicles: {
"$addToSet": {
name: "$vehicles.name",
listed: "$vehicles.listed"
}
}
}
},
{
$project: {
_id: 0,
id: "$_id",
vehicles: 1
}
}
]).pretty();