I have a json payload that looks like this
{
"data":{
"methods":[
[
{
"p_id":"01",
"description":"Test01",
"offline":true
}
],
[
{
"p_id":"02",
"description":"Test02",
"offline":false
}
],
[
{
"p_id":"03",
"description":"Test03",
"offline":true
}
]
]
}
}
How can I write a JSONPath expression to get the "p_id" where "offline"= false?
You can use a filter expression which selects all elements in an object or array that match the specified filter. For example, [?(#.offline === false)] will match any object if its offline property is strictly false.
So, if the object is always in the same place, you could do:
$.data.methods.*[?(#.offline === false)].p_id
Or, if you want to look for any object where offline is false and fetch p_id, you could use a recursive descent with the filter expression:
$..[?(#.offline === false)].p_id
Note: I used strict equality in my examples so it will only match with a boolean false. If you don't need/want that you could instead simply use a ! to negate the filter. E.g. [?(!#.offline)]
Related
Consider the following MongoDB collection of a few thousand Objects:
{
_id: ObjectId("xxx")
FM_ID: "123"
Meter_Readings: Array
0: Object
Date: 2011-10-07
Begin_Read: true
Reading: 652
1: Object
Date: 2018-10-01
Begin_Reading: true
Reading: 851
}
The wrong key was entered for 2018 into the array and needs to be renamed to "Begin_Read". I have a list using another aggregate of all the objects that have the incorrect key. The objects within the array don't have an _id value, so are hard to select. I was thinking I could iterate through the collection and find the array index of the errored Readings and using the _id of the object to perform the $rename on the key.
I am trying to get the index of the array, but cannot seem to select it correctly. The following aggregate is what I have:
[
{
'$match': {
'_id': ObjectId('xxx')
}
}, {
'$project': {
'index': {
'$indexOfArray': [
'$Meter_Readings', {
'$eq': [
'$Meter_Readings.Begin_Reading', True
]
}
]
}
}
}
]
Its result is always -1 which I think means my expression must be wrong as the expected result would be 1.
I'm using Python for this script (can use javascript as well), if there is a better way to do this (maybe a filter?), I'm open to alternatives, just what I've come up with.
I fixed this myself. I was close with the aggregate but needed to look at a different field for some reason that one did not work:
{
'$project': {
'index': {
'$indexOfArray': [
'$Meter_Readings.Water_Year', 2018
]
}
}
}
What I did learn was the to find an object within an array you can just reference it in the array identifier in the $indexOfArray method. I hope that might help someone else.
In Azure Data Factory v2 (ADFv2) I am having trouble passing a parameter whose value is an expression that needs evaluated at runtime.
For example, I have a set of extracts I want to download daily from the same LinkedService/Connection. I want a pipeline with a Foreach to be able to input a JSON pipeline parameter with a list of configuration for each report type (this I can do). "but" when I have one of those configuration KVPairs with value that is an expression, the expression does not seem to be evaluated.
here is an example of a Foreach parameter set that works for an SFTP LinkedService :
[ { "dirPath" : "/dirPath" ,"fileFilter" : "this_works_fine_20180307*.txt" } ]
here is an example of a Foreach parameter set that does not match the files I need to get.
(assume utcnow('yyyyMMdd') returns 20180307
[ { "dirPath" : "/dirPath" ,"fileFilter" : "this_does_NOT_work_#{utcnow('yyyyMMdd')}*.txt" } ]
This assumes that in the underlying Copy activity I am passing the dataset parameter fileFilter as
#item().fileFilter
...and in the dataset, the value of the fileFilter is an expression with value
#dataset().fileFilter
...I have also tried to wrap the argument completely as:
[ { "dirPath" : "/dirPath" ,"fileFilter" : "#toLower(concat(string('this_does_NOT_work_'),string(utcnow('yyyyMMdd')),string('*.txt') )))" } ]
...If you have suggestions/guidance, please let me know.
Thanks,
J
Try to put the fileFilter parameter directly in pipeline parameter.
Something like this will work:
[ { "dirPath" : "/dirPath" ,"fileFilter" : "this_works_fine_#{formatDateTime(utcnow(), 'yyyy')}#{formatDateTime(utcnow(), 'MM')}#{formatDateTime(utcnow(), 'dd')}*.txt" } ]
I have a model which contains an array with dates. I'm using a $gte operator as a condition to query the collection where all the elements in the array of dates are $gte a given date.
For example I have this document:
{ dates: [
ISODate("2016-10-24T22:00:00.000+0000"),
ISODate("2017-01-16T23:00:00.000+0000")]
}
When I run this query {dates: {$gte: new Date()}}, it gives me the whole document as a result. But I want a result where every single array item matches my query, not just one.
You can do this by using $not and the inversion of your comparison condition:
db.test.find({dates: {$not: {$lt: new Date()}}})
So this matches docs where it's not the case that there's a dates element with a value less than the current time; in other words, all dates values are >= the current time.
You can also use the aggregation framework with the $redact pipeline operator that allows you to proccess the logical condition with the $cond operator and uses the special operations $$KEEP to "keep" the document where the logical condition is true or $$PRUNE to "remove" the document where the condition was false.
This operation is similar to having a $project pipeline that selects the fields in the collection and creates a new field that holds the result from the logical condition query and then a subsequent $match, except that $redact uses a single pipeline stage which is more efficient.
As for the logical condition, there are Set Operators that you can use since they allow expression that perform set operations on arrays, treating arrays as sets. These couple of these operators namely the $allElementTrue and $map operators can be used as the logical condition expression as they work in such a way that if all of the elements in the array actually are $gte a specified date, then this is a true match and the document is "kept". Otherwise it is "pruned" and discarded.
Consider the following examples which demonstrate the above concept:
Populate Test Collection
db.test.insert([
{ dates: [
ISODate("2016-10-24T22:00:00.000+0000"),
ISODate("2017-01-16T23:00:00.000+0000")]
} ,
{ dates: [
ISODate("2017-01-03T22:00:00.000+0000"),
ISODate("2017-01-16T23:00:00.000+0000")]
}
])
$redact with $setEquals
db.test.aggregate([
{ "$match": { "dates": { "$gte": new Date() } } },
{
"$redact": {
"$cond": [
{
"$allElementsTrue": {
"$map": {
"input": "$dates",
"as": "date",
"in": { "$gte": [ "$$date", new Date() ] }
}
}
},
"$$KEEP",
"$$PRUNE"
]
}
}
])
Sample Output
{
"_id" : ObjectId("581899dda450d81cb7d87d3a"),
"dates" : [
ISODate("2017-01-03T22:00:00.000Z"),
ISODate("2017-01-16T23:00:00.000Z")
]
}
Another not-so elegant approach would be to use $where (as a last resort) with the Array.prototype.every() method:
db.test.find({ "$where": function(){
return this.dates.every(function(date) {
return date >= new Date();
})}
})
I currently have a simple condition query for MongoDB that goes like this:
{
"$and": [
{
"ApplicationStatus": "UW_SECOND_LEVEL"
},
{
"Application.UnderWriterDecisions.ud_cc_total_aggregate_limit": {
"$lte": 300000, "$gt":100000
}
},
{
"Application.UnderWriterDecisions.UW_Recon_Count": {
"$eq": null
}
}
]
}
I need to add another condition to the query above that checks the size of an array (DecisionReason) that I have, and make sure that it only has a size of 1:
Application.UnderwriterDecisions.DecisionReason
You can use the $size operator. Also, you don't have to use the explicit $and operator, but just separate the clauses with commas. From this page in the documentation.
MongoDB provides an implicit AND operation when specifying a comma separated list of expressions. Using an explicit AND with the $and operator is necessary when the same field or operator has to be specified in multiple expressions.
Also, you do not need to use the $eq operator, so your final query would look something like this:
db.collection.find({
"ApplicationStatus": "UW_SECOND_LEVEL",
"Application.UnderWriterDecisions.ud_cc_total_aggregate_limit":
{ "$lte": 300000, "$gt": 100000 },
"Application.UnderWriterDecisions.UW_Recon_Count": null,
"Application.UnderWriterDecisions.DecisionReason":
{ "$size": 1 }
});
For more information about the $size operator, check out this page in the documentation.
I'd like to be able to query documents where an array attribute is either null or of zero length. I've tried using $or, like
{ things : { $or : [null, { $size : 0 } ] } }
but MongoDB complains about it:
{ "$err" : "invalid operator: $or", "code" : 10068 }
It appears that MongoDB doesn't want to use $and or $or on array attributes. I thought that simply querying for { things : { $size : 0 } } would return documents that had non-null arrays of zero length as well as null arrays, but that doesn't appear to be true.
How can I issue a query to find all documents where an array attribute is null or of zero length?
You can also do this with $in:
db.test.find({things: {$in: [null, []]}})
That will also include docs with things: [null].
Another way is to use $exists on the first element of the things array:
db.test.find({'things.0': {$exists: false}})
That will find the docs where things does not have at least one element. In contrast with the above, it will still include docs with things: null or things: [], but not things: [null].
When things is null, it is not of type Array. It is of type Null which is its own data-type. $size only matches arrays. Any document where the field is something different than an array will never be matched by it.
Your syntax error is in how you use the $or operator. You don't use it for a field, you use it as a field.
db.collection.find({
$or: [
{ things: { $size: 0 } }, // 0-elements array
{ things: null } // field which is null
]
});