Ok i am trying to implement a query, which is trying to perform regex search ( which contains an array list ) on a bunch of document
Its hard for me to explain...so I am basically coming to direct point.
There is query which works with regex array...
db.paper.find({"category": {$in: [ /xd/, /sd/, /ad/ ] }})
There is query which doesn't works with regex array...
db.paper.find({"category": {$in: [ "/xd/", "/sd/", "/ad/" ] }})
So basically what I want is remove "" sign from string array...so that i can perform below query..
var sea = [ "/xd/", "/sd/", "/ad/" ];
db.paper.find({"category": {$in: sea }});
Using $in can be fairly efficient with small arrays but not so well with huge lists since it will skip around in the index to find the matching documents, or walk through the whole collection if there isn't an index to use.
Besides using the $in with the regular expression, you could use a pipe-delimited regex pattern with the keywords list like this:
Test documents:
db.papertest.insert([
{ category: "ad bd cd" },
{ category: "dd ed fd" },
{ category: "gd hd id" },
{ category: "jd kd ld" },
{ category: "md nd od" },
{ category: "pd qd rd" },
{ category: "sd td ud" },
{ category: "vd wd xd yd zd" },
]);
The magic:
var keywords = ["xd", "sd", "ad"],
regex = keywords.join("|");
db.papertest.find({
"category": {
"$regex": regex,
"$options": "i"
}
});
The results
{ "_id" : ObjectId("56bb6f171bb4f693057c0ba4"), "category" : "ad bd cd" }
{ "_id" : ObjectId("56bb6f171bb4f693057c0baa"), "category" : "sd td ud" }
{ "_id" : ObjectId("56bb6f171bb4f693057c0bab"), "category" : "vd wd xd yd zd" }
it does not work when the double quotes are present because they are interpreted as strings instead of as RegExp objects. So to make it to work, you have to convert it to RegExp objects first in Javascript like this.
var sea = [ "xd", "sd", "ad" ]; // Note: no slashes
var regex = [];
for (var i = 0; i < sea.length; i++) {
regex[i] = new RegExp(sea[i]);
}
db.paper.find({"category": {$in: regex}});
Remember, MongoDB shell uses Javascript
It seems to be working fine for me please try this
var sea = [ "xd", "sd", "ad" ];
var regex = sea.map( function( val ){
return new RegExp( '^['+val+'].*','i' );
})
db.paper.find({"category": { $in: regex }});
For this you can add a regular expression to each item in the array, you can do it in the following way.
data = ['hoLA','Que','TAL', 'Nueva'];
data = data.map(function(v, i){return new RegExp(v, 'i')});
MyCollection.find({"thing": {$in : data}}, function(err, data){
if (err) {
console.log(err)
}else{
data.forEach(function(item){
console.log(item.nombre);
})
}
});
Slightly improved ES6 + TypeScript answer based on Meme Composer comment:
const sea: string[] = [ "xd", "sd", "ad" ];
const regex: RegExp[] = sea.map((value) => new RegExp(value));
db.paper.find({ "category": { $in: regex } });
Here is simple way to transform /.*/ style regex.
var sea = [ "/xd/", "/sd/", "/ad/" ];
var rx = [];
sea.forEach(function name(value) {
var v = value.replace(/\//ig,"");
rx.push(new RegExp(v));
});
db.paper.find({"category": {$in: rx}});
Related
is it possible to find out the query array element not in database?
example:
const query = ['aaa','bbb','ccc']
Documents in db:
[{name:'bbb'},{name:'ccc'}]
I want to find query array elements not in database:
return result should be:
['aaa']
I can't find some quickly method to do this except query each element(or batch?) in array
Any one has better method? thanks
Querying for stuff that are -missing- is always a more expensive operation, also there is no "magic" query to do it for you. I recommend using Mongo's distinct method, like so:
const queryArr = ['aaa', 'bbb', 'ccc'];
const allNames = await db.collection.distinct('name');
const notInDb = queryArr.filter(e => !allNames.includes(e));
However if you want to do it in 1 db command you could do something like this:
db.collection.aggregate([
{
$group: {
_id: null,
names: {
"$addToSet": "$name"
}
}
},
{
"$replaceRoot": {
"newRoot": {
results: {
$filter: {
input: [
"aaa",
"bbb",
"ccc"
],
as: "datum",
cond: {
$not: {
"$setIsSubset": [
[
"$$datum"
],
"$names"
]
}
}
}
}
}
}
}
])
Mongo Playground
As you can tell both approaches require you to load all the names into memory, there is no way around this, if your db's scale is too big for these approaches you will have to iterate over the query input and do it one by one.
const queryArr = ['aaa', 'bbb', 'ccc'];
for (let queryName of queryArr) {
const found = await db.collection.findOne({name: queryName})
if (!found) {
//ding
}
}
Assuming you have an index on name field this should be very efficient.
{
"_id" : ObjectId("58d9084841a6168234689aee"),
"ID" : "01",
"data" : {
"Type1" : {
"value" : "ABC",
"timestamp" : "2017-03-20 16:01:01"
},
"Type2" : {
"value" : "ccc",
"timestamp" : "2017-03-20 16:01:01"
}
}
}
I want to get timestamp of each TYPE from mongodb using queryobject using nodejs.
How to get it,Please help.
var queryObject = url.parse(req.url,true).query;
var mdb = db.collection("HISTORY").find({{'timestamp':{"$gte":queryObject.fromdate,"$lt" : queryObject.todate}},{"ID":1});
Here is my node service:
function getHistory(req,res){
try{
var queryObject = url.parse(req.url,true).query;
var index=0, resultset = [];
var db1 = db.collection("HISTORY").find({$and : [{'data.Type1.timestamp':{"$gte": new Date(queryObject.fromdate),"$lt" : new Date(queryObject.todate)}},
{'data.Type2.timestamp':{"$gte": new Date(queryObject.fromdate),"$lt" : new Date(queryObject.todate)}},
{'data.Type3.timestamp':{"$gte": new Date(queryObject.fromdate),"$lt" : new Date(queryObject.todate)}},
{'data.Type4.timestamp':{"$gte": new Date(queryObject.fromdate),"$lt" : new Date(queryObject.todate)}},
{'data.Type5.timestamp':{"$gte": new Date(queryObject.fromdate),"$lt" : new Date(queryObject.todate)}}
]},{"Ino":1,"ID":1,"data":1});
db1.count(function(err, count) {
console.log("count" , count);
db1.each(function(err, doc) {
if(doc!=null){
var valdata=doc.alarms;
var fields = [];
var queryString ="SELECT field1,NAME FROM details c inner join locdetails l on c.loc_id=l.loc_id where no='"+doc.Ino+"' limit 1;";
var dtfield1 = null;
var dtfield2 = null;
connection.query(queryString, function(err,result){
index++;
if(err){
}else{
if(result.length>0)
{
dtfield1 = result[0].field1;
dtfield2 = result[0].NAME;
if(dtfield1!=null){
for (var x in valdata) {
var dt = new Date(valdata[x].timestamp).toISOString().replace(/T/, ' ').replace(/\..+/, '');
var compareDate = new Date(dt);
if(compareDate.getTime()>=fromDate.getTime()&&compareDate.getTime()<=toDate.getTime()){
resultset.push({"Name":dtfield1,"LName":dtfield2,"Ino":doc.Ino,"ID":doc.ID,"data":x,"datav":valdata[x].value,"Timestamp":valdata[x].timestamp});
}
if(index == count){
res.writeHead(200, {
'Content-Type': 'application/json'
});
res.write(JSON.stringify(resultset));
res.end();
}
}} }}
});
} else {
}
});
});
}
catch (err) {
console.log("Exception -- ",err);
}
}
I want data should filter based on timestamp and same can be display in UI and download the displayed data.And also filter should be in UI like current day data and based on time filter also.
If you want search base on both Type1.timestamp and Type2.timestamp, you should use $and. you can change $and with $or if one matching one of them is enough
db.collection("HISTORY").find({
$and : [{'data.Type1.timestamp':{"$gte": new Date(queryObject.fromdate),"$lt" : new Date(queryObject.todate)}},
{'data.Type2.timestamp':{"$gte": new Date(queryObject.fromdate),"$lt" : new Date(queryObject.todate)}}
]
},
{ID:1}, function (err, res){
});
You can use aggregation to meet your requirements.
For your initial list which will only show the latest data values of each object, you can use the following pipeline -
[
{
$project:{
_id:1,
ID:1,
data:{ $objectToArray: "$data" }
}
},
{
$unwind:"$data"
},
{
$sort:{
"data.v.timestamp":-1
}
},
{
$group:{
_id:{
_id:"$_id",
ID:"$ID"
},
data:{
$first:"$data"
}
}
},
{
$addFields:{
data:["$data"]
}
},
{
$addFields:{
data:{ $arrayToObject: "$data" }
}
}
]
The result of the aggregation should give you an array of objects like this -
{
"_id" : ObjectId("5bc469c11f8e482416d6edb3"),
"ID" : "01",
"data" : {
"Type2" : {
"value" : "ccc",
"timestamp" : ISODate("2017-03-22T16:01:01.000+06:00")
}
}
}
As for filtering the collection to get documents with data properties that have timestamp values in the given range, the above pipeline can be modified to acquire it
[
{
$project:{
_id:1,
ID:1,
data:{ $objectToArray: "$data" }
}
},
{
$unwind:"$data"
},
{
$sort:{
"data.v.timestamp":-1
}
},
{
$match:{
"data.v.timestamp":{ $gt: "start date value", $lt: "end date value"}
}
},
{
$group:{
_id:{
_id:"$_id",
ID:"$ID"
},
data:{
$addToSet:"$data"
}
}
},
{
$project:{
_id:"$_id._id",
ID:"$_id.ID",
data:{ $arrayToObject: "$data" }
}
}
]
The result of this aggregation will produce array of objects similar to the one described above. But this time each document's data object will only have properties that match the timestamp condition.
Hope this helps. A final word of advice to you would be to rethink the schema design of the history collection. Because if simplistic date based queries are getting this difficult for you. You can only imagine what the future might hold for you with this design.
I am trying to query a single embedded document in an array in MongoDB. I don't know what I am doing wrong. Programmatically, I will query this document and insert new embedded documents into the currently empty trips arrays.
{
"_id" : ObjectId("564b3300953d9d51429163c3"),
"agency_key" : "DDOT",
"routes" : [
{
"route_id" : "6165",
"route_type" : "3",
"trips" : [ ]
},
{
"route_id" : "6170",
"route_type" : "3",
"trips" : [ ]
},
...
]
}
Following queries -I run in mongo shell- return empty:
db.tm_routes.find( { routes : {$elemMatch: { route_id:6165 } } } ).pretty();
db.tm_routes.find( { routes : {$elemMatch: { route_id:6165,route_type:3 } } } ).pretty();
db.tm_routes.find({'routes.route_id':6165}).pretty()
also db.tm_routes.find({'routes.route_id':6165}).count() is 0.
The following query returns every document in the array
db.tm_routes.find({'routes.route_id':'6165'}).pretty();
{
"_id" : ObjectId("564b3300953d9d51429163c3"),
"agency_key" : "DDOT",
"routes" : [
{
"route_id" : "6165",
"route_type" : "3",
"trips" : [ ]
},
{
"route_id" : "6170",
"route_type" : "3",
"trips" : [ ]
},
...
]}
but db.tm_routes.find({'routes.route_id':'6165'}).count() returns 1.
And finally, here is how I inserted data in the first place -in Node.JS-:
async.waterfall([
...
//RETRIEVE ALL ROUTEIDS FOR EVERY AGENCY
function(agencyKeys, callback) {
var routeIds = [];
var routesArr = [];
var routes = db.collection('routes');
//CALL GETROUTES FUNCTION FOR EVERY AGENCY
async.map(agencyKeys, getRoutes, function(err, results){
if (err) throw err;
else {
callback(null, results);
}
});
//GET ROUTE IDS
function getRoutes(agencyKey, callback){
var cursor = routes.find({agency_key:agencyKey});
cursor.toArray(function(err, docs){
if(err) throw err;
for(i in docs){
routeIds.push(docs[i].route_id);
var routeObj = {
route_id:docs[i].route_id,
route_type:docs[i].route_type,
trips:[]
};
routesArr.push(routeObj);
/* I TRIED 3 DIFFERENT WAYS TO PUSH DATA
//1->
collection.update({agency_key:agencyKey}, {$push:{"routes":{
'route_id':docs[i].route_id,
'route_type':docs[i].route_type,
'trips':[]
}}});
//2->
collection.update({agency_key:agencyKey}, {$push:{"routes":routeObj}});
*/
}
// 3->
collection.update({agency_key:agencyKey}, {$push:{routes:{$each:routesArr}}});
callback(null, routeIds);
});
};
},
...
var collection = newCollection(db, 'tm_routes',[]);
function newCollection(db, name, options){
var collection = db.collection(name);
if (collection){
collection.drop();
}
db.createCollection(name, options);
return db.collection(name);
}
Note: I am not using Mongoose and don't want to use if possible.
Melis,
I see what you are asking for, and what you need is help understanding how things are stored in mongodb. Things to understand:
A document is the basic unit of data for MongoDB and can be roughly compared to a row in a relational database.
A collection can be thought of as a table with a dynamic schema
So documents are stored in collections.Every document has a special _id, that is unique within a collection. What you showed us above in the following format is One document.
{
"_id" : ObjectId("564b3300953d9d51429163c3"),
"agency_key" : "DDOT",
"routes" : [
{
"route_id" : "6165",
"route_type" : "3",
"trips" : [ ]
},
{
"route_id" : "6170",
"route_type" : "3",
"trips" : [ ]
},
...
]}
If you run a query in your tm_routes collection. The find() will return each document in the collection that matches that query. Therefore when you run the query db.tm_routes.find({'routes.route_id':'6165'}).pretty(); it is returning the entire document that matches the query. Therefore this statement is wrong:
The following query returns every document in the array
If you need to find a specific route in that document, and only return that route, depending on your use, because its an array, you may have to use the $-Positional Operator or the aggregation framework.
For Node and Mongodb users using Mongoose, this is one of the ways to write the query to the above problem:
db.tm_routes.updateOne(
{
routes: {
$elemMatch: {
route_id: 6165 (or if its in a route path then **6165** could be replaced by **req.params.routeid**
}
}
},
{
$push: {
"routes.$.trips":{
//the content you want to push into the trips array goes here
}
}
}
)
my schema looks like this:
var exampleSchema = newSchema({
profile:{
experience :[{
exp : String
}]
}
});
this is the codes to update experience in profile collection:
exampleSchema.statics.experience = function (id,experience, callback){
var update = {
$push: {
'profile.experience': experience
}
}
this.findByIdAndUpdate(id,update,function(err) {
if (err) {
callback(err);
} else {
callback(null);
}
})
I was getting error like The field 'profile.experience' must be an array but is of type String in document {_id: ObjectId('5653f1d852cf7b4c0bfeb54a')}[object Object]
console.log(experience) is equal to
{ exp: 'jlkjlkjlk' }
my collection should look like this:
experience:[
{
exp : "YYYY"
},
{
exp:"xxxx"}
]
Imagine that you have this collection:
/* 1 */
{
"_id" : ObjectId("565425e862760dfe14339ba8"),
"profile" : {
"experience" : [
{
"exp" : "Experto"
}
]
}
}
/* 2 */
{
"_id" : ObjectId("565425f562760dfe14339ba9"),
"profile" : {
"experience" : {
"exp" : "Experto"
}
}
}
/* 3 */
{
"_id" : ObjectId("5654260662760dfe14339baa"),
"profile" : {
"experience" : "Experto"
}
}
If you try (update doc /* 2 */):
db.profile.update(
{ _id: ObjectId("565425f562760dfe14339ba9") },
{ $push: { "profile.experience" : { exp : "Intermediate" } } }
)
You get this error:
The field 'profile.experience' must be an array but is of type Object
in document {_id: ObjectId('565425f562760dfe14339ba9')}
And if you try (update doc /* 3 */):
db.profile.update(
{ _id: ObjectId("5654260662760dfe14339baa") },
{ $push: { "profile.experience" : { exp : "Intermediate" } } }
)
You will get:
The field 'profile.experience' must be an array but is of type String
in document {_id: ObjectId('5654260662760dfe14339baa')}
i changed Schema like this
experience : [{type:String,exp:String}],
my update object looks like this
var update = {
$push: {
'profile.experience': san.exp
}
};
san looks like this :{ exp: 'YYY' }
Inside mongoose collectionlooks like this used RoboMongo
"experience" : [
"experienced in XXX",
"YYY"
],
$push: {
'profile.experience': experience
}
Remove .exp.
First you have to check you declared your field as an array like this(look at field products):
shop = {
'name': "Apple Store",
'description': "",
'direction': "",
'contact': "",
'products':[]
}
Now if you want to add something to the field products using $push
product = {
'name': "Iphone 6",
'description': "Iphone model 6, 64GB",
'price': 700,
'count': 3
}
myquery = { "name" : "Apple Store" }
obj ={"$push":{"products":{"$each": [product]}}}
db.collection.update_one(myquery,obj)
This code is provided for PyMongo framework. To use in MongoDB directly replace update_one by update. Mongo resource
You may use $set instead of $push which might work.
$set: {
'profile.experience': experience
}
are you searching for adding multiple values into single field then use this one.
write this one your model or schema:
arrayremarks:[{remark: String}]
then write in your controller:
module.exports.addingremarks = (req, res) => {
let casenum=JSON.parse(JSON.stringify(req.body.casenum).replace(/"\s+|\s+"/g,'"'))
var rem={remark:"Suman macha"}
Inwart.update( { 'casenum': casenum },{ $push: { arrayremarks:rem} } ,function (err, inwarts) {
if (err)
return console.error(err);
res.send(inwarts);
}
)
}
I have an array of objects:
result = [
{ _id: 53d0dfe3c42047c81386df9d, video_id: '1' },
{ _id: 53d0dfe3c42047c81386df9e, video_id: '1' },
{ _id: 53d0dfe3c42047c81386df9f, video_id: '1' },
{ _id: 53d0dfe3c42047c81386dfa0, video_id: '2' },
{ _id: 53d0dfe3c42047c81386dfa1, video_id: '2' },
{ _id: 53d0dfe3c42047c81386dfa2, video_id: '1' },
{ _id: 53d0dfe3c42047c81386dfa3, video_id: '2' },
{ _id: 53d0dfe3c42047c81386dfa4, video_id: '1' }
]
I need to create another array, which takes video_id as the index, and contains how many times this video_id appears in the first array:
list = [
{'1' : 5},
{'2' : 4}
]
Currently, I use this code:
while (i < result.length)
{
if(list[result[i].video_id] === undefined) {
list[result[i].video_id] = 0;
}
list[result[i].video_id] = list[result[i].video_id] + 1;
i = i + 1;
}
It works, but I wonder if there is any faster and cleaner way to do so? (the real result array has over 10k elements, and I doubt >10k conditional statements are optimal...).
I am using node.js, result is from a mongoose (mongoDB) query, and I didn't see any way to get this done by mongoose itself:
var now = new Date();
//M_logs is a mongoose model
query = M_logs.where('time').gt(new Date(now.getFullYear(), 0, 1).getTime() / 1000).lt(now.getTime() / 1000).select('video_id');
(PS: I wonder if this isn't more a Code Review question, please tell me if I am off-topic so I can migrate the question).
EDIT:
To answer to Juan Carlos Farah:
S_logs = new mongoose.Schema({
user_ip : String,
user_id : String,
user_agent : String,
canal_id : String,
theme_id : String,
video_id : String,
osef : String,
time : Number,
action: String,
is_newuser : String,
operator : String,
template : String,
catalogue : String,
referer : String,
from : String,
osef1 : String
});
M_logs = mongoose.model('logs', S_logs);
You can do this using the aggregation framework. The idea is to do something as follows:
Match the documents you are looking for. Based on your current query, I understand it would be documents where time is between new Date(now.getFullYear(), 0, 1).getTime() / 1000 and now.getTime() / 1000.
Group the matched documents by video_id and keep track of their count.
Optionally sort by _id, which would be equivalent to the original video_id.
The following is in mongo shell syntax:
var now = new Date();
db.M_logs.aggregate([
{
"$match" : {
"time" : {
"$gt" : new Date(now.getFullYear(), 0, 1).getTime() / 1000,
"$lt" : now.getTime() / 1000
}
}
},
{
"$group" : {
"_id" : "$video_id",
"count" : { "$sum" : 1 }
}
},
{
"$sort" : { "_id" : 1 }
}
]);
If this works for you, you can easily implement it in Mongoose or Node.js driver syntax. Note that the aggregation framework returns a cursor, which you can iterate through to populate your array.
EDIT:
Using the Node.js driver, you can access the results from the aggregation query in the callback function. Something as follows:
...
, function(err, result) {
console.dir(result);
db.close();
}
Note that the Mongoose syntax for aggregation queries is slightly different.
Example:
Model.aggregate([ <QUERY> ]).exec( <CALLBACK> );
For more information, consult the documentation here.
I would suggest that you use aggregation framework to count number of documents. It will be significantly faster than iterating all your documents and counting them.
Using mongoose you can do it like this:
var now = new Date();
var startTime = new Date(now.getFullYear(), 0, 1).getTime() / 1000):
var endTime = now.getTime() / 1000;
M_logs.aggregate([
// filter the documents you're looking for
{"$match" : { "time" : {"$gt": startTime, "$lt": endTime}}},
// group by to get the count for each video_id
{"$group" : {"_id" : "$video_id", "count" : {"$sum" : 1}}},
// make the output more explanatory; this part is optional
{"$project" : { "video_id" : "$_id", "count" : "$count", _id : 0}}
]).exec(function(err, docs){
if (err) console.err(err);
console.log(docs);
});
The output of the docs will be:
[ { count: 4, video_id: '2' }, { count: 5, video_id: '1' } ]
use
var list = {};
result.forEach(function (el) {
list[el.video_id] = (list[el.video_id] || 0) + 1;
});
the resuling list will look something like this:
var list = {
'1': 5,
'2': 4
};