Trying to perform an ES query, I ran into a problem while trying to do a nested filtering of objects in an array. Our structure of data has changed from being:
"_index": "events_2015-07-08",
"_type": "my_type",
"_source":{
...
...
"custom_data":{
"className:"....."
}
}
to:
"_index": "events_2015-07-08",
"_type": "my_type",
"_source":{
...
...
"custom_data":[ //THIS CHANGED FROM AN OBJECT TO AN ARRAY OF OBJECTS
{
"key":".....",
"val":"....."
},
{
"key":".....",
"val":"....."
}
]
}
this nested filter works fine on indices that have the new data structure:
{
"nested": {
"path": "custom_data",
"filter": {
"bool": {
"must": [
{
"term":
{
"custom_data.key": "className"
}
},
{
"term": {
"custom_data.val": "SOME_VALUE"
}
}
]
}
},
"_cache": true
}
}
However, it fails when going over indices that have the older data structure, so that feature cannot be added. Ideally I'd be able to find both data structures but at this point i'd settle for a "graceful failure" i.e. just don't return results where the structure is old.
I have tried adding an "exists" filter on the field "custom_data.key", and an "exists" within "not" on the field "custom_data.className", but I keep getting "SearchParseException[[events_2015-07-01][0]: from[-1],size[-1]: Parse Failure [Failed to parse source"
There is an indices filter (and query) that you can use to perform conditional filters (and queries) based on the index that it is running against.
{
"query" : {
"filtered" : {
"filter" : {
"indices" : {
"indices" : ["old-index-1", "old-index-2"],
"filter" : {
"term" : {
"className" : "SOME_VALUE"
}
},
"no_match_filter" : {
"nested" : { ... }
}
}
}
}
}
}
Using this, you should be able to transition off of the old mapping and onto the new mapping.
Related
I have a database in MongoDB where there are entries with the following structure image
I am using a script in Python that I use to retrive information from this strucutre and then saves it to a csv.
The end goal is to remove ONLY SOME parts of the parameters structure. For instance, I want to keep parameters[0].values and parameters[1].values but not parameters[2:5].values
I am not experienced in this but the command I am using is the following
{"$unset":"exercises.parameters.values"] }
but this will remove all values from all the objects inside parameters. Since the variable has the same name amongst the objects I can't find a way to specify which ones I want.
I have also tried indexing like so
{"$unset":"exercises.parameters[2].values"] }
but that doesn't seem to work...
Help is much appreciated!
update
db.collection.update({
"_id": 1
},
{
"$unset": {
"exercises.$[].parameters.2.values": "",
"exercises.$[].parameters.3.values": "",
"exercises.$[].parameters.4.values": "",
"exercises.$[].parameters.5.values": ""
}
},
{
"multi": true
})
mongoplayground
aggregate
db.collection.aggregate([
{
$set: {
"exercises": {
"$map": {
"input": "$exercises",
"as": "item",
"in": {
"$mergeObjects": [
"$$item",
{
parameters: {
$slice: [ "$$item.parameters", 2 ]
}
}
]
}
}
}
}
}
])
mongoplayground
I am working on an express js application where I need to update a nested array.
1) Schema :
//Creating a mongoose schema
var userSchema = mongoose.Schema({
_id: {type: String, required:true},
name: String,
sensors: [{
sensor_name: {type: String, required:true},
measurements: [{time: String}]
}] });
2)
Here is the code snippet and explanation is below:
router.route('/sensors_update/:_id/:sensor_name/')
.post(function (req, res) {
User.findOneAndUpdate({_id:req.body._id}, {$push: {"sensors" :
{"sensor_name" : req.body.sensor_name , "measurements.0.time": req.body.time } } },
{new:true},function(err, newSensor) {
if (err)
res.send(err);
res.send(newSensor)
}); });
I am able to successfully update a value to the measurements array using the findOneAndUpdate with push technique but I'm failing when I try to add multiple measurements to the sensors array.
Here is current json I get if I get when I post a second measurement to the sensors array :
{
"_id": "Manasa",
"name": "Manasa Sub",
"__v": 0,
"sensors": [
{
"sensor_name": "ras",
"_id": "57da0a4bf3884d1fb2234c74",
"measurements": [
{
"time": "8:00"
}
]
},
{
"sensor_name": "ras",
"_id": "57da0a68f3884d1fb2234c75",
"measurements": [
{
"time": "9:00"
}
]
}]}
But the right format I want is posting multiple measurements with the sensors array like this :
Right JSON format would be :
{
"_id" : "Manasa",
"name" : "Manasa Sub",
"sensors" : [
{
"sensor_name" : "ras",
"_id" : ObjectId("57da0a4bf3884d1fb2234c74"),
"measurements" : [
{
"time" : "8:00"
}
],
"measurements" : [
{
"time" : "9:00"
}
]
}],
"__v" : 0 }
Please suggest some ideas regarding this. Thanks in advance.
You might want to rethink your data model. As it is currently, you cannot accomplish what you want. The sensors field refers to an array. In the ideal document format that you have provided, you have a single object inside that array. Then inside that object, you have two fields with the exact same key. In a JSON object, or mongo document in this context, you can't have duplicate keys within the same object.
It's not clear exactly what you're looking for here, but perhaps it would be best to go for something like this:
{
"_id" : "Manasa",
"name" : "Manasa Sub",
"sensors" : [
{
"sensor_name" : "ras",
"_id" : ObjectId("57da0a4bf3884d1fb2234c74"),
"measurements" : [
{
"time" : "8:00"
},
{
"time" : "9:00"
}
]
},
{
// next sensor in the sensors array with similar format
"_id": "",
"name": "",
"measurements": []
}],
}
If this is what you want, then you can try this:
User.findOneAndUpdate(
{ _id:req.body._id "sensors.sensor_name": req.body.sensor_name },
{ $push: { "sensors.0.measurements": { "time": req.body.time } } }
);
And as a side note, if you're only ever going to store a single string in each object in the measurements array, you might want to just store the actual values instead of the whole object { time: "value" }. You might find the data easier to handle this way.
Instead of hardcoding the index of the array it is possible to use identifier and positional operator $.
Example:
User.findOneAndUpdate(
{ _id: "Manasa" },
{ $push: { "sensors.$[outer].measurements": { "time": req.body.time } } }
{ "arrayFilters:" [{"outer._id": ObjectId("57da0a4bf3884d1fb2234c74")}]
);
You may notice than instead of getting a first element of the array I specified which element of the sensors array I would like to update by providing its ObjectId.
Note that arrayFilters are passed as the third argument to the update query as an option.
You could now make "outer._id" dynamic by passing the ObjectId of the sensor like so: {"outer._id": req.body.sensorId}
In general, with the use of identifier, you can get to even deeper nested array elements by following the same procedure and adding more filters.
If there was a third level nesting you could then do something like:
User.findOneAndUpdate(
{ _id: "Manasa" },
{ $push: { "sensors.$[outer].measurements.$[inner].example": { "time": req.body.time } } }
{ "arrayFilters:" [{"outer._id": ObjectId("57da0a4bf3884d1fb2234c74"), {"inner._id": ObjectId("57da0a4bf3884d1fb2234c74"}}]
);
You can find more details here in the answer written by Neil Lunn.
refer ::: positional-all
--- conditions :: { other_conditions, 'array1.array2.field_to_be_checked': 'value' }
--- updateData ::: { $push : { 'array1.$[].array2.$[].array3' : 'value_to_be_pushed' } }
I'm using the following json to find results in a Cloudant
{
"selector": {
"$and": [
{
"type": {
"$eq": "sensor"
}
},
{
"v": {
"$eq": 2355
}
},
{
"$or": [
{
"p": "#401000103"
},
{
"p": "#401000114"
}
]
},
{
"t_max": {
"$gte": 1459554894
}
},
{
"t_min": {
"$lte": 1459509591
}
}
]
},
"fields": [
"_id",
"p"
],
"limit": 200
}
If I run this againt my cloudant database I get the following error:
{
"error": "unknown_error",
"reason": "function_clause",
"ref": 3379914628
}
If I remove one the $or elements I get the results for query.
(,{"p":"#401000114"})
Also i get a result if I replace #401000114 with #401000114 I get result.
But when I want to use both element I get the error code above.
Can anybody tell what this error_reason: function_clause mean?
error_reason: function_clause means there was a problem on the server, you should probably reach out to Cloudant Support and see if they can help you with your issue.
I had contact with the Cloudant support.
This is there answer:
The issue affects Cloudant generally
It affects both mult-tenant and dedicated clusters.
There are working on the sollution.
A workaround is in the array to which the $or operator applies has two elements, you can get the correct result by repeating one of the items in the array.
I send a query from the client for docs where the currentDate is between two values attached to all docs. The docs have an expireDate and an availableDate.
How do I formulate this query? As a guess I came up with the following, which doesn't work:
{
"constant_score": {
"filter": {
"range" : {
"expiredate" : {
"lte": "2015-08-15T11:28:45.114-07:00"
},
"availabledate" : {
"gte": "2015-08-11T11:28:45.114-07:00"
}
}
}
}
}
This even looks wrong because afaik the range won't work on a compound object or an array of one-sided ranges. I guess I could add an entire new filter so one for each field. Is there a way to use one date range for two fields?
The correct way of expressing that query is using two range filters (one for each date field) and place them in another bool/must filter.
{
"query": {
"constant_score": {
"filter": {
"bool": {
"must": [
{
"range": {
"expireDate": {
"lte": "2015-08-15T11:28:45.114-07:00"
}
}
},
{
"range": {
"availabledate": {
"gte": "2015-08-11T11:28:45.114-07:00"
}
}
}
]
}
}
}
}
}
I have a document structure like
{
"_id" : ObjectId("52263922f5ebf05115bf550e"),
"Fields" : [
{
"Field" : "Lot No",
"Rules" : [ ]
},
{
"Field" : "RMA No",
"Rules" : [ ]
}
]
}
I have tried to update by using the following code to push into the Rules Array which will hold objects.
db.test.update({
"Fields.Field":{$in:["Lot No"]}
}, {
$addToSet: {
"Fields.Field.$.Rules": {
"item_name": "my_item_two",
"price": 1
}
}
}, false, true);
But I get the following error:
can't append to array using string field name [Field]
How do I do the update?
You gone too deep with that wildcard $. You match for an item in the Fields array, so you get a access on that, with: Fields.$. This expression returns the first match in your Fields array, so you reach its fields by Fields.$.Field or Fields.$.Result.
Now, lets update the update:
db.test.update({
"Fields.Field": "Lot No"
}, {
$addToSet: {
"Fields.$.Rules": {
'item_name': "my_item_two",
'price':1
}
}
}, false, true);
Please note that I've shortened the query as it is equal to your expression.