I'm trying to do something which should be pretty straightforward. The aggregrate pipeline allows you to pass documents through stages in sequence. For example, I can filter out some documents and pass the documents I want to the next stage. This can be accomplished with $match. Then I want to take the remaining documents and group them by a field to check for duplicates. But I am having major issues with this. This is what I have tried:
db.my_fields.aggregate(
{ '$match' : { related_code_id: { "$in" : [ BSON::ObjectId('5cddd0143ed1495e5c000008'),BSON::ObjectId('5cddd0143ed1495e5c000010')] } } },
{ '$group' : { _id: { 'field' : '$field', 'field_type_id' : '$field_type_id' }, count: { '$sum' : 1} } },
{'$match' : { count: {'$gt' : 1} } }
)
The query works fine except for the first $match. Inside the $in statement, I am getting this error:
2019-05-17T18:47:56.602-0400 E QUERY [js] SyntaxError: missing ] after element list #(shell):2:95
It is complaining about this part right here:
BSON::ObjectId('5cddd0143ed1495e5c000008'),BSON::ObjectId('5cddd0143ed1495e5c000010')
How can I resolve this issue and achieve my goal of getting this basic query working?
Converting the ObjectId to BSON isn't needed:
https://docs.mongodb.com/manual/reference/method/db.collection.find/#query-using-operators
// ...
'$match': {
related_code_id: {
"$in": [
ObjectId('5cddd0143ed1495e5c000008'),
ObjectId('5cddd0143ed1495e5c000010')
]
}
}
// ...
If you are using Mongoose as ORM, you can solve it as follows.
db.my_fields.aggregate(
{ '$match' : { related_code_id: { "$in" : [ mongoose.Types.ObjectId('5cddd0143ed1495e5c000008'),mongoose.Types.ObjectId('5cddd0143ed14 95e5c000010')] } } },
{ '$group' : { _id: { 'field' : '$field', 'field_type_id' : '$field_type_id' }, count:
{ '$sum' : 1} } },
{'$match' : { count: {'$gt' : 1} } }
)
Related
I'm new to working with elastic search so my understanding is fairly limited. For a project, using ReactJS, I need to return the latest n number of entries into a database, which changes each day. I'm currently returning all data and using slice(-n) on the frontend to solve the problem, but I appreciate this isn't ideal. This is what I'm currently posting:
return axios
.post(
`${url}/${db}/1`,
{
aggs: {
sales_over_time: {
date_histogram: {
field: "date",
calendar_interval: "day",
format: "dd-MM-yyyy",
},
},
},
},
)
.then(({ data }) => {
return data;
});
Any help is appreciated,
Thanks
You can easily sort your date_histogram aggregation result using order parameter in your elasticsearch query.
Also you can slice the result of your aggregation either by providing the size parameter in sub aggregation of type bucket_sort or by using range in filter query.
Below is the elasticsearch query to do that -
{
"aggs" : {
"sales_over_time" : {
"date_histogram" : {
"order" : {
"_key" : "desc"
},
"field" : "date",
"calendar_interval":"day",
"format" : "dd-MM-yyyy"
},
"aggs" : {
"latest7" : {
"bucket_sort" : {
"size" : 7,
"sort" : []
}
}
}
}
}
}
I have simple mapping - one string field and one string[] field.
The array of strings contains duplicate values, and I get those duplicate values in query:
{ "query" : { "term" : {"id" : "579a252585b8c5c428fa0a3c"} } }
Returns a single valid hit:
{
"id" : "579a252585b8c5c428fa0a3c",
"touches" : [ "5639abfb5cba47087e8b4571", "5639abfb5cba47087e8b4571", "5639abfb5cba47087e8b4571", "5639abfb5cba47087e8b457b", "5639abfb5cba47087e8b457b"
}
But in metric script aggregation:
"aggs": {
"path": {
"scripted_metric": {
"map_script": "_agg['result'] = doc['touches'].values"
}
}
}
retuns
"aggregations" : {
"path" : {
"value" : [ { }, {
"result" : [ "5639abfb5cba47087e8b4571", "5639abfb5cba47087e8b457b" ]
}, { }, { }, { } ]
}
}
that element is org.elasticsearch.index.fielddata.ScriptDocValues$Strings, casting it toString() returns a json-encoded 2-element array.
So, the question:
Why does ScriptDocValues$Strings return only unique array values and how to get the initial array in script aggregation?
Thanks.
UPD
I found that for numerical values (in particular floats) everything works perfect.
I have the following document:
{
"_id":"575322d9585095d9929554ba",
"Level1":{
"Level2":[
{
"Level3a":{
"Level4":{
"Level5":{
"name":"John",
"surname":"Matthew"
}
}
}
},
{
"Level3a":{
"Level4":{
"Level5":{
"name":"Emma",
"surname":"Jackson"
}
}
}
}
]
}
}
I need to insert the new name and surname at Level5. I tried the $push method but I got the error that the dotted field .. is not valid for storage:
db.names.update({ "_id":"575322d9585095d9929554ba" },
{
$push: {
"Level1.Level2":
{ $each: [ { "Level3a.Level4.Level5.name": "Greg" },
{ "Level3a.Level4.Level5.surname": "Cook" }] } } } )
It seems that push does not allow the inserting new data in hierarchical arrays or I am wrong?
There is an issue with your query as $each is used to add multiple values to same array....
As we are going to add only one entry to array we don't need to use $each
so query looks like this:
db.vnenad.update({
"_id" : "575322d9585095d9929554ba"
},
{
$push : {
"Level1.Level2" : {
"Level3a" : {
"Level4" : {
"Level5" : {
"name" : "johnny",
"surname" : "rambo"
}
}
}
}
}
})
as I was debbuging your query I decided to create variable with document to insert inside an array, then it was easier for me to adjust your query - please see below:
var doc = {
"Level3a" : {
"Level4" : {
"Level5" : {
"name" : "johnny",
"surname" : "rambo"
}
}
}
}
db.vnenad.update({
"_id" : "575322d9585095d9929554ba"
},
{
$push : {
"Level1.Level2" : doc
}
})
I am trying to query a single embedded document in an array in MongoDB. I don't know what I am doing wrong. Programmatically, I will query this document and insert new embedded documents into the currently empty trips arrays.
{
"_id" : ObjectId("564b3300953d9d51429163c3"),
"agency_key" : "DDOT",
"routes" : [
{
"route_id" : "6165",
"route_type" : "3",
"trips" : [ ]
},
{
"route_id" : "6170",
"route_type" : "3",
"trips" : [ ]
},
...
]
}
Following queries -I run in mongo shell- return empty:
db.tm_routes.find( { routes : {$elemMatch: { route_id:6165 } } } ).pretty();
db.tm_routes.find( { routes : {$elemMatch: { route_id:6165,route_type:3 } } } ).pretty();
db.tm_routes.find({'routes.route_id':6165}).pretty()
also db.tm_routes.find({'routes.route_id':6165}).count() is 0.
The following query returns every document in the array
db.tm_routes.find({'routes.route_id':'6165'}).pretty();
{
"_id" : ObjectId("564b3300953d9d51429163c3"),
"agency_key" : "DDOT",
"routes" : [
{
"route_id" : "6165",
"route_type" : "3",
"trips" : [ ]
},
{
"route_id" : "6170",
"route_type" : "3",
"trips" : [ ]
},
...
]}
but db.tm_routes.find({'routes.route_id':'6165'}).count() returns 1.
And finally, here is how I inserted data in the first place -in Node.JS-:
async.waterfall([
...
//RETRIEVE ALL ROUTEIDS FOR EVERY AGENCY
function(agencyKeys, callback) {
var routeIds = [];
var routesArr = [];
var routes = db.collection('routes');
//CALL GETROUTES FUNCTION FOR EVERY AGENCY
async.map(agencyKeys, getRoutes, function(err, results){
if (err) throw err;
else {
callback(null, results);
}
});
//GET ROUTE IDS
function getRoutes(agencyKey, callback){
var cursor = routes.find({agency_key:agencyKey});
cursor.toArray(function(err, docs){
if(err) throw err;
for(i in docs){
routeIds.push(docs[i].route_id);
var routeObj = {
route_id:docs[i].route_id,
route_type:docs[i].route_type,
trips:[]
};
routesArr.push(routeObj);
/* I TRIED 3 DIFFERENT WAYS TO PUSH DATA
//1->
collection.update({agency_key:agencyKey}, {$push:{"routes":{
'route_id':docs[i].route_id,
'route_type':docs[i].route_type,
'trips':[]
}}});
//2->
collection.update({agency_key:agencyKey}, {$push:{"routes":routeObj}});
*/
}
// 3->
collection.update({agency_key:agencyKey}, {$push:{routes:{$each:routesArr}}});
callback(null, routeIds);
});
};
},
...
var collection = newCollection(db, 'tm_routes',[]);
function newCollection(db, name, options){
var collection = db.collection(name);
if (collection){
collection.drop();
}
db.createCollection(name, options);
return db.collection(name);
}
Note: I am not using Mongoose and don't want to use if possible.
Melis,
I see what you are asking for, and what you need is help understanding how things are stored in mongodb. Things to understand:
A document is the basic unit of data for MongoDB and can be roughly compared to a row in a relational database.
A collection can be thought of as a table with a dynamic schema
So documents are stored in collections.Every document has a special _id, that is unique within a collection. What you showed us above in the following format is One document.
{
"_id" : ObjectId("564b3300953d9d51429163c3"),
"agency_key" : "DDOT",
"routes" : [
{
"route_id" : "6165",
"route_type" : "3",
"trips" : [ ]
},
{
"route_id" : "6170",
"route_type" : "3",
"trips" : [ ]
},
...
]}
If you run a query in your tm_routes collection. The find() will return each document in the collection that matches that query. Therefore when you run the query db.tm_routes.find({'routes.route_id':'6165'}).pretty(); it is returning the entire document that matches the query. Therefore this statement is wrong:
The following query returns every document in the array
If you need to find a specific route in that document, and only return that route, depending on your use, because its an array, you may have to use the $-Positional Operator or the aggregation framework.
For Node and Mongodb users using Mongoose, this is one of the ways to write the query to the above problem:
db.tm_routes.updateOne(
{
routes: {
$elemMatch: {
route_id: 6165 (or if its in a route path then **6165** could be replaced by **req.params.routeid**
}
}
},
{
$push: {
"routes.$.trips":{
//the content you want to push into the trips array goes here
}
}
}
)
my schema looks like this:
var exampleSchema = newSchema({
profile:{
experience :[{
exp : String
}]
}
});
this is the codes to update experience in profile collection:
exampleSchema.statics.experience = function (id,experience, callback){
var update = {
$push: {
'profile.experience': experience
}
}
this.findByIdAndUpdate(id,update,function(err) {
if (err) {
callback(err);
} else {
callback(null);
}
})
I was getting error like The field 'profile.experience' must be an array but is of type String in document {_id: ObjectId('5653f1d852cf7b4c0bfeb54a')}[object Object]
console.log(experience) is equal to
{ exp: 'jlkjlkjlk' }
my collection should look like this:
experience:[
{
exp : "YYYY"
},
{
exp:"xxxx"}
]
Imagine that you have this collection:
/* 1 */
{
"_id" : ObjectId("565425e862760dfe14339ba8"),
"profile" : {
"experience" : [
{
"exp" : "Experto"
}
]
}
}
/* 2 */
{
"_id" : ObjectId("565425f562760dfe14339ba9"),
"profile" : {
"experience" : {
"exp" : "Experto"
}
}
}
/* 3 */
{
"_id" : ObjectId("5654260662760dfe14339baa"),
"profile" : {
"experience" : "Experto"
}
}
If you try (update doc /* 2 */):
db.profile.update(
{ _id: ObjectId("565425f562760dfe14339ba9") },
{ $push: { "profile.experience" : { exp : "Intermediate" } } }
)
You get this error:
The field 'profile.experience' must be an array but is of type Object
in document {_id: ObjectId('565425f562760dfe14339ba9')}
And if you try (update doc /* 3 */):
db.profile.update(
{ _id: ObjectId("5654260662760dfe14339baa") },
{ $push: { "profile.experience" : { exp : "Intermediate" } } }
)
You will get:
The field 'profile.experience' must be an array but is of type String
in document {_id: ObjectId('5654260662760dfe14339baa')}
i changed Schema like this
experience : [{type:String,exp:String}],
my update object looks like this
var update = {
$push: {
'profile.experience': san.exp
}
};
san looks like this :{ exp: 'YYY' }
Inside mongoose collectionlooks like this used RoboMongo
"experience" : [
"experienced in XXX",
"YYY"
],
$push: {
'profile.experience': experience
}
Remove .exp.
First you have to check you declared your field as an array like this(look at field products):
shop = {
'name': "Apple Store",
'description': "",
'direction': "",
'contact': "",
'products':[]
}
Now if you want to add something to the field products using $push
product = {
'name': "Iphone 6",
'description': "Iphone model 6, 64GB",
'price': 700,
'count': 3
}
myquery = { "name" : "Apple Store" }
obj ={"$push":{"products":{"$each": [product]}}}
db.collection.update_one(myquery,obj)
This code is provided for PyMongo framework. To use in MongoDB directly replace update_one by update. Mongo resource
You may use $set instead of $push which might work.
$set: {
'profile.experience': experience
}
are you searching for adding multiple values into single field then use this one.
write this one your model or schema:
arrayremarks:[{remark: String}]
then write in your controller:
module.exports.addingremarks = (req, res) => {
let casenum=JSON.parse(JSON.stringify(req.body.casenum).replace(/"\s+|\s+"/g,'"'))
var rem={remark:"Suman macha"}
Inwart.update( { 'casenum': casenum },{ $push: { arrayremarks:rem} } ,function (err, inwarts) {
if (err)
return console.error(err);
res.send(inwarts);
}
)
}