Why MongoDB aggregation is rewritten by Spring Data in a wrong way? - spring-data-mongodb

There is a collection
{
"field": {
"nested": 0
}
}
It is necessary to get a sum of nested fields. As direct operation like below is not allowed
{
"$group": {
"_id": null,
"field.nested": {
"$sum": "$field.nested"
}
}
}
It is done by the following aggregation with pre and post projection.
[
{
"$project": {
"fieldNested": "$field.nested"
}
},
{
"$group": {
"_id": null,
"fieldNested": {
"$sum": "$fieldNested"
}
}
},
{
"$project": {
"total": {
"field" : {
"nested" : "$fieldNested"
}
}
}
}
]
And it works correctly in the Mongo DB shell. The problem comes with using this aggregation in Spring Data in a json format. For example,
#Repository
public interface Sample extends MongoRepository<InData, String> {
#Aggregation({
"" +
"{" +
" '$project': {" +
" 'fieldNested': '$field.nested'" +
" }" +
"}",
"" +
"{" +
" '$group': {" +
" '_id': null," +
" 'fieldNested': {" +
" '$sum': '$fieldNested'" +
" }" +
" }" +
"}",
"" +
"{" +
" '$project': {" +
" 'total': {" +
" 'field': {" +
" 'nested': '$fieldNested'" +
" }" +
" }" +
" }" +
"}"
})
List<OutData> aggregateBy();
}
Somehow this json is rewritten by Spring Data so that the pipelines sent to the Mongo looks like the following
[
{
"$project": {
"field.nested": "$field.nested"
}
},
{
"$group": {
"_id": null,
"field.nested": {
"$sum": "$fieldNested"
}
}
},
{
"$project": {
"total": {
"field" : {
"nested" : "$fieldNested"
}
}
}
}
]
fieldNested is replaced by field.nested. As a result the aggregation fails.
Why it happens? Is there any way to avoid this rewrite by Spring Data for MongoDB (Spring Boot 2.2.13.RELEASE, MongoDB 3.6.8)?

Spring Data MongoDB is converting fieldNested to field.nested because your InData class & its collection is having fields in that hierarchy. If you want to project that nested field using #Aggregation, then project it using any other name that is not part of your InData class & its collection's fields so Spring Data won't convert it.

Related

Loop over children properties in painless script

I have the following ElasticSearch data structure for products in a webshop. It's a main product with children that have a special price and a regular price. I have omitted some extraneous info in the children to make it more clear.
{
"_index": "vue_storefront_catalog_1_product_1617378559",
"_type": "_doc",
"_source": {
"configurable_children": [
{
"price": 49.99,
"special_price": 34.99
}
{
"price": 49.99,
"special_price": null
}
]
}
Using the following mapping in Elasticsearh:
{
"vue_storefront_catalog_1_product_1614928276" : {
"mappings" : {
"properties" : {
"configurable_children" : {
"properties" : {
"price" : {
"type" : "double"
}
"special_price" : {
"type" : "double"
}
}
}
}
}
}
}
I have created a loop in a painless script to go through the children of the configurable_children. I need to do this to determine if the main product is on sale, based on the children configurable_children
boolean hasSale = false;
for(item in doc['configurable_children']) {
hasSale = true;
if (1 - (item['special_price'].value / item['price'].value) > 0.5) {
hasSale = true;
}
}
return hasSale
When I look at the results I see the following error:
"failed_shards": [{
"shard": 0,
"index": "vue_storefront_catalog_1_product_1617512844",
"node": "2EQcMMqlQgiuT5GAFPo90w",
"reason": {
"type": "script_exception",
"reason": "runtime error",
"script_stack": ["org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:90)", "org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:41)", "for(item in doc['configurable_children']) { ", " ^---- HERE"],
"script": " boolean hasSale = false; for(item in doc['configurable_children']) { hasSale = true; if (1 - (item['special_price'].value / item['price'].value) > 0.5) { hasSale = true; } } return hasSale ",
"lang": "painless",
"position": {
"offset": 42,
"start": 26,
"end": 70
},
"caused_by": {
"type": "illegal_argument_exception",
"reason": "No field found for [configurable_children] in mapping with types []"
}
}
}]
No field found for [configurable_children] in mapping with types []
Anyone knows what I'm doing wrong? It looks like the for in loop needs a different kind of data? How do I look through all of the products to determine the sale price?

Pushing to nested array inside specific element of parent array (MongoDB)

I'm rethinking how I want to structure some data which is currently being stored in the Users Collection. Previously, my server would receive messages, find the user document with the right profile.website field, and push an item into the profile.siteMessages array:
{
"_id": "hr9ck5Fis5YuvqCqP",
"profile": {
"website": "localhost",
"siteMessages": [
{
"text": "sddsd",
"createdAt": 1482001227204
},
]
}
}
Id like to change the structure to look something like the following. Instead of storing all messages, of which multiple messages could come from the same user, in a top level array in profile, I would have a profile.siteVisitors field which contains a visitorId and then the array of messages:
{
"_id": "dgfsdfdfsdf",
"emails": [
{
"address": "user2#test.com",
"verified": false
}
],
"profile": {
"website": "localhost",
"siteVisitors:" [
{
"visitorId": "74585242",
"messages": [
{
"text": "A string",
"createdAt": 1482001260853
},
{
"text": "Another string",
"createdAt": 1482001260854
}
]
},
{
"visitorId": "76672242",
"messages": [
{
"text": "A string",
"createdAt": 1482001260855
}
]
}
]
}
}
Keeping with the structure shown above, how would I query for and update the profile.siteVisitiors.messages array? Currently I query and update the Collection using something like the following:
Meteor.users.update(
{'profile.website': url},
{$push: {'profile.siteMessages': msgItem}},
function(err) {
if (err) {
console.log('whoops!' + err)
} else {
//success
}
});
How would I update the newly structured messages array? I would need to match a User documents profile.website field, match a visitorId in the profile.siteVisitors array, and then push a new element into the messages array, but I'm not sure how this would look as a MongoDB query.
EDIT I've hacked together the following which seems to work, but is very ugly. How can I improve this?
Meteor.users.update(
{"profile.website" : "localhost" , "profile.siteVisitors" : {$elemMatch: {"visitorId" : data.chirpVisitorId} } },
{$push: { "profile.siteVisitors.$.messages": {"text" : data.msg, "createdAt" : data.msg.createdAt} } },
function(err, res) {
if (err) {
console.log('failed to push ' + err)
} else {
console.log('success on new push ' + res)
if (res < 1) {
let item = {
"visitorId": data.chirpVisitorId,
"messages": [data.msg]
}
Meteor.users.update(
{"profile.website": "localhost"},
{$push: {'profile.siteVisitors': item}},
function(err, res) {
if (err) {
console.log(err)
} else {
console.log(res + " updated")
}
}
)
}
}
})
You can use the $(update) operator.
Try the following query:
db.collection.update(
{"profile.website" : "localhost" , "profile.siteVisitors" : {$elemMatch: {"visitorId" :'76672242'} } },
{ $push: { "profile.siteVisitors.$.messages": {"text" : <newText>, "createdAt" : <newCreatedDate>} } }
)
Hope this helps.

How to map json arrays in angularjs controller

I am having problem in mapping json array in angularjs, can someone please look how can i correctly map the arrays and iterate their values. In arrays icdCode requires to be dynamic field (this is the other part i need help on) how can i achieve this correctly. thanks
Way i am mapping in Controller
$scope.preAuthorizationInfo.collections.preAuthClinicalDetailInfoFormVO.active = active;
Json arrays
"preAuthDiagnosisVOs": [
{
"preauthDiagnosisId": 165,
"diagnosisVO": {
"diagnosisId": 171,
"diagnosisCode": "Provisional",
"icdCode": {
"icdCodeId": 1,
"description": "Other intestinal Escherichia coli infections",
"icdCode": "Other intestinal Escherichia coli infections",
"icdCodeChapter": "Certain infectious and parasitic diseases",
"icdCodeCode": "A04.4"
},
"active": false
},
"active": true
},
{
"preauthDiagnosisId": 166,
"diagnosisVO": {
"diagnosisId": 172,
"diagnosisCode": "differential",
"icdCode": {
"icdCodeId": 2,
"description": "Other viral enteritis",
"icdCode": "Other viral enteritis",
"icdCodeChapter": "Certain infectious and parasitic diseases",
"icdCodeCode": "A08.3"
},
"active": false
},
"active": true
}
]
},
If You want to iterate over this object , You can use nested loop
angular.forEach(your_object.preAuthDiagnosisVOs, function(value, key) {
if (angular.isObject(value)) {
angular.forEach(value, function(value1, key) {
if (angular.isObject(value1)) {
angular.forEach(value1, function(value2, key) {
if (angular.isObject(value2)) {
angular.forEach(value2, function(value3, key) {
console.log(key + " : " + value3);
});
} else {
console.log(key + " : " + value2);
}
});
} else {
console.log(key + " : " + value1);
}
});
} else {
console.log(key + " : " + value);
}
});
And if you want just that active value just use
yourObject.preAuthDiagnosisVOs[0].active or yourObject.preAuthDiagnosisVOs[1].active
Hope this helps you . Thanks
I just validate your json format it has some syntactical issue. I have corrected this.
{
"preAuthDiagnosisVOs": [{
"preauthDiagnosisId": 165,
"diagnosisVO": {
"diagnosisId": 171,
"diagnosisCode": "Provisional",
"icdCode": {
"icdCodeId": 1,
"description": "OtherintestinalEscherichiacoliinfections",
"icdCode": "OtherintestinalEscherichiacoliinfections",
"icdCodeChapter": "Certaininfectiousandparasiticdiseases",
"icdCodeCode": "A04.4"
},
"active": false
},
"active": true
}, {
"preauthDiagnosisId": 166,
"diagnosisVO": {
"diagnosisId": 172,
"diagnosisCode": "Provisional",
"icdCode": {
"icdCodeId": 2,
"description": "Otherviralenteritis",
"icdCode": "Otherviralenteritis",
"icdCodeChapter": "Certaininfectiousandparasiticdiseases",
"icdCodeCode": "A08.3"
},
"active": false
},
"active": true
}]
}
Now to access active key . preAuthDiagnosisVOs is array to access this you have to iterate with angular forEach loop. For access active key of each index just iterate and get it or to get directly use object.preAuthDiagnosisVOs[0].active and object.preAuthDiagnosisVOs[1].active.
I think it might be help you. Thank you

Updating nested array inside array mongodb [duplicate]

This question already has answers here:
Updating a Nested Array with MongoDB
(2 answers)
Closed 3 years ago.
I have the following mongodb document structure:
[
{
"_id": "04",
"name": "test service 4",
"id": "04",
"version": "0.0.1",
"title": "testing",
"description": "test",
"protocol": "test",
"operations": [
{
"_id": "99",
"oName": "test op 52222222222",
"sid": "04",
"name": "test op 52222222222",
"oid": "99",
"description": "testing",
"returntype": "test",
"parameters": [
{
"oName": "Param1",
"name": "Param1",
"pid": "011",
"type": "582",
"description": "testing",
"value": ""
},
{
"oName": "Param2",
"name": "Param2",
"pid": "012",
"type": "58222",
"description": "testing",
"value": ""
}
]
}
]
}
]
I have been able to use $elemMatch in order to update fields in operations, but when I try to do the same thing (modified) for parameters it does not seem to work. I was wondering what other approach should I look into trying in order to be able to successfully update fields in a specific parameter, looking it up by its pid.
The update code I currently have and does not work looks like this:
var oid = req.params.operations;
var pid = req.params.parameters;
collection.update({"parameters":{"$elemMatch": {"pid": pid}}},{"$set": {"parameters.$.name":req.body.name, "parameters.$.description": req.body.description,"parameters.$.oName": req.body.oName,"parameters.$.type": req.body.type} }, function(err, result) {
if (err) {
console.log('Error updating service: ' + err);
res.send({'error':'An error has occurred'});
} else {
// console.log('' + result + ' document(s) updated');
res.send(result);
}
});
MongoDB 3.6 and newer
With MongoDB 3.6 and above comes a new feature that allows you to update nested arrays by using the positional filtered $\[<identifier>\] syntax in order to match the specific elements and apply different conditions through arrayFilters in the update statement:
const { oid, pid } = req.params;
const { name, oName, description, type } = req.body;
collection.update(
{
"_id": 1,
"operations": {
"$elemMatch": {
oid, "parameters.pid": pid
}
}
},
{ "$set": {
"operations.$[outer].parameters.$[inner].name": name,
"operations.$[outer].parameters.$[inner].description": description,
"operations.$[outer].parameters.$[inner].oName": oName,
"operations.$[outer].parameters.$[inner].type": type
} },
{ "arrayFilters": [
{ "outer.oid": oid },
{ "inner.pid": pid }
] }, (err, result) => {
if (err) {
console.log('Error updating service: ' + err);
res.send({'error':'An error has occurred'});
} else {
// console.log('' + result + ' document(s) updated');
res.send(result);
}
});
For MongoDB 3.4 and older:
As #wdberkeley mentioned in his comment:
MongoDB doesn't support matching into more than one level of an array.
Consider altering your document model so each document represents an
operation, with information common to a set of operations duplicated
in the operation documents.
I concur with the above and would recommend redesigning your schema as MongoDB engine does not support multiple positional operators ( See Multiple use of the positional $ operator to update nested arrays)
However, if you know the index of the operations array that has the parameters object to be updated beforehand then the update query will be:
db.collection.update(
{
"_id" : "04",
"operations.parameters.pid": "011"
},
{
"$set": {
"operations.0.parameters.$.name": "foo",
"operations.0.parameters.$.description": "bar",
"operations.0.parameters.$.type": "foo"
}
}
)
EDIT:
If you would like to create the $set conditions on the fly i.e. something which would help you get the indexes for the objects and then modify accordingly, then consider using MapReduce.
Currently this seems to be not possible using the aggregation framework. There is an unresolved open JIRA issue linked to it. However, a workaround is possible with MapReduce. The basic idea with MapReduce is that it uses JavaScript as its query language but this tends to be fairly slower than the aggregation framework and should not be used for real-time data analysis.
In your MapReduce operation, you need to define a couple of steps i.e. the mapping step (which maps an operation into every document in the collection, and the operation can either do nothing or emit some object with keys and projected values) and reducing step (which takes the list of emitted values and reduces it to a single element).
For the map step, you ideally would want to get for every document in the collection, the index for each operations array field and another key that contains the $set keys.
Your reduce step would be a function (which does nothing) simply defined as var reduce = function() {};
The final step in your MapReduce operation will then create a separate collection operations that contains the emitted operations array object along with a field with the $set conditions. This collection can be updated periodically when you run the MapReduce operation on the original collection.
Altogether, this MapReduce method would look like:
var map = function(){
for(var i = 0; i < this.operations.length; i++){
emit(
{
"_id": this._id,
"index": i
},
{
"index": i,
"operations": this.operations[i],
"update": {
"name": "operations." + i.toString() + ".parameters.$.name",
"description": "operations." + i.toString() + ".parameters.$.description",
"type": "operations." + i.toString() + ".parameters.$.type"
}
}
);
}
};
var reduce = function(){};
db.collection.mapReduce(
map,
reduce,
{
"out": {
"replace": "operations"
}
}
);
Querying the output collection operations from the MapReduce operation will typically give you the result:
db.operations.findOne()
Output:
{
"_id" : {
"_id" : "03",
"index" : 0
},
"value" : {
"index" : 0,
"operations" : {
"_id" : "96",
"oName" : "test op 52222222222",
"sid" : "04",
"name" : "test op 52222222222",
"oid" : "99",
"description" : "testing",
"returntype" : "test",
"parameters" : [
{
"oName" : "Param1",
"name" : "foo",
"pid" : "011",
"type" : "foo",
"description" : "bar",
"value" : ""
},
{
"oName" : "Param2",
"name" : "Param2",
"pid" : "012",
"type" : "58222",
"description" : "testing",
"value" : ""
}
]
},
"update" : {
"name" : "operations.0.parameters.$.name",
"description" : "operations.0.parameters.$.description",
"type" : "operations.0.parameters.$.type"
}
}
}
You can then use the cursor from the db.operations.find() method to iterate over and update your collection accordingly:
var oid = req.params.operations;
var pid = req.params.parameters;
var cur = db.operations.find({"_id._id": oid, "value.operations.parameters.pid": pid });
// Iterate through results and update using the update query object set dynamically by using the array-index syntax.
while (cur.hasNext()) {
var doc = cur.next();
var update = { "$set": {} };
// set the update query object
update["$set"][doc.value.update.name] = req.body.name;
update["$set"][doc.value.update.description] = req.body.description;
update["$set"][doc.value.update.type] = req.body.type;
db.collection.update(
{
"_id" : oid,
"operations.parameters.pid": pid
},
update
);
};
If it is data that is changed frequently, you should flatten the structure and separate the data that changes a lot from that which does not.
If it is data that does not change often, and the entire data object is not massive, just modify the object client-side, and update the entire object.
We will try to find index of outer array(i) and inner array(j)and then update
collection.findById(04)
.then(result =>{
for(let i = 0; i<result.operations.length; i++){
if(result.operation[i]._id == "99"){
let parameters = result.operations[i].parameters;`enter code here`
for(let j = 0; j<parameters.length; j++){
if(parameters[j].pid == "011"){
console.log("i", i);
console.log("j", j);
let data = {}
data["operations." + i + ".parameters." + j + ".oName"] = updateoName
data["operations." + i + ".parameters." + j + ".name"] = updatename
data["operations." + i + ".parameters." + j + ".pid"] = updatepid
data["operations." + i + ".parameters." + j + ".description"] = updatedescription
data["operations." + i + ".parameters." + j + ".value"] = updatevalue
console.log(data)
collection.update({
"_id": "04"
},{
$set: data
})
.then(dbModel => res.json(dbModel))
}
}
}
}
})
Starting with mongo version 3.6 you can use the $[] in Conjunction with $[] to Update Nested Arrays
Update Nested Arrays in Conjunction with $[]
The $[] filtered positional operator, in conjunction with
all $[] positional operator can be used to update nested arrays.
https://docs.mongodb.com/manual/reference/operator/update/positional-filtered/#position-nested-arrays-filtered

ElasticSearch bool search using array

I would like to be able to exclude a number of nodes in my ElasticSearch query. Currently this code is working perfectly, but it will only exclude two nodes ($nodeId1 and $nodeId2).
How would I change this query so I can use an array which would contain as many node IDs as I want? Thank you.
$data_string = '{
"from" : 0, "size" : "' . $maximumResults . '",
"query": {
"filtered": {
"query": {
"match" : {
"thread.title" : {
"query" : "' . $threadTitle . '",
"operator": "or"
}
}
},
"filter": {
"bool" : {
"must" : [],
"must_not" : [
{
"term" : {"thread.discussion_id" : "' . $currentThreadId . '"}
},
{
"term" : {"thread.node" : "' . $nodeId1 . '"}
},
{
"term" : {"thread.node" : "' . $nodeId2 . '"}
}
],
"should" : []
}
}
}
}
}';
You can use the terms filter:
$data_string = '{
"from" : 0, "size" : "' . $maximumResults . '",
"query": {
"filtered": {
"query": {
"match" : {
"thread.title" : {
"query" : "' . $threadTitle . '",
"operator": "or"
}
}
},
"filter": {
"bool" : {
"must" : [],
"must_not" : [
{
"term" : {"thread.discussion_id" : "' . $currentThreadId . '"}
},
{
"terms" : {"thread.node" : ["' . $nodeId1 . '", "' . $nodeId2 . '", "' . $nodeId3 . '"}
}
],
"should" : []
}
}
}
}
}';

Resources