I have an array of objects:
result = [
{ _id: 53d0dfe3c42047c81386df9d, video_id: '1' },
{ _id: 53d0dfe3c42047c81386df9e, video_id: '1' },
{ _id: 53d0dfe3c42047c81386df9f, video_id: '1' },
{ _id: 53d0dfe3c42047c81386dfa0, video_id: '2' },
{ _id: 53d0dfe3c42047c81386dfa1, video_id: '2' },
{ _id: 53d0dfe3c42047c81386dfa2, video_id: '1' },
{ _id: 53d0dfe3c42047c81386dfa3, video_id: '2' },
{ _id: 53d0dfe3c42047c81386dfa4, video_id: '1' }
]
I need to create another array, which takes video_id as the index, and contains how many times this video_id appears in the first array:
list = [
{'1' : 5},
{'2' : 4}
]
Currently, I use this code:
while (i < result.length)
{
if(list[result[i].video_id] === undefined) {
list[result[i].video_id] = 0;
}
list[result[i].video_id] = list[result[i].video_id] + 1;
i = i + 1;
}
It works, but I wonder if there is any faster and cleaner way to do so? (the real result array has over 10k elements, and I doubt >10k conditional statements are optimal...).
I am using node.js, result is from a mongoose (mongoDB) query, and I didn't see any way to get this done by mongoose itself:
var now = new Date();
//M_logs is a mongoose model
query = M_logs.where('time').gt(new Date(now.getFullYear(), 0, 1).getTime() / 1000).lt(now.getTime() / 1000).select('video_id');
(PS: I wonder if this isn't more a Code Review question, please tell me if I am off-topic so I can migrate the question).
EDIT:
To answer to Juan Carlos Farah:
S_logs = new mongoose.Schema({
user_ip : String,
user_id : String,
user_agent : String,
canal_id : String,
theme_id : String,
video_id : String,
osef : String,
time : Number,
action: String,
is_newuser : String,
operator : String,
template : String,
catalogue : String,
referer : String,
from : String,
osef1 : String
});
M_logs = mongoose.model('logs', S_logs);
You can do this using the aggregation framework. The idea is to do something as follows:
Match the documents you are looking for. Based on your current query, I understand it would be documents where time is between new Date(now.getFullYear(), 0, 1).getTime() / 1000 and now.getTime() / 1000.
Group the matched documents by video_id and keep track of their count.
Optionally sort by _id, which would be equivalent to the original video_id.
The following is in mongo shell syntax:
var now = new Date();
db.M_logs.aggregate([
{
"$match" : {
"time" : {
"$gt" : new Date(now.getFullYear(), 0, 1).getTime() / 1000,
"$lt" : now.getTime() / 1000
}
}
},
{
"$group" : {
"_id" : "$video_id",
"count" : { "$sum" : 1 }
}
},
{
"$sort" : { "_id" : 1 }
}
]);
If this works for you, you can easily implement it in Mongoose or Node.js driver syntax. Note that the aggregation framework returns a cursor, which you can iterate through to populate your array.
EDIT:
Using the Node.js driver, you can access the results from the aggregation query in the callback function. Something as follows:
...
, function(err, result) {
console.dir(result);
db.close();
}
Note that the Mongoose syntax for aggregation queries is slightly different.
Example:
Model.aggregate([ <QUERY> ]).exec( <CALLBACK> );
For more information, consult the documentation here.
I would suggest that you use aggregation framework to count number of documents. It will be significantly faster than iterating all your documents and counting them.
Using mongoose you can do it like this:
var now = new Date();
var startTime = new Date(now.getFullYear(), 0, 1).getTime() / 1000):
var endTime = now.getTime() / 1000;
M_logs.aggregate([
// filter the documents you're looking for
{"$match" : { "time" : {"$gt": startTime, "$lt": endTime}}},
// group by to get the count for each video_id
{"$group" : {"_id" : "$video_id", "count" : {"$sum" : 1}}},
// make the output more explanatory; this part is optional
{"$project" : { "video_id" : "$_id", "count" : "$count", _id : 0}}
]).exec(function(err, docs){
if (err) console.err(err);
console.log(docs);
});
The output of the docs will be:
[ { count: 4, video_id: '2' }, { count: 5, video_id: '1' } ]
use
var list = {};
result.forEach(function (el) {
list[el.video_id] = (list[el.video_id] || 0) + 1;
});
the resuling list will look something like this:
var list = {
'1': 5,
'2': 4
};
Related
I have read as many articles as I can on how to correctly update an element in an array in a MongoDB (such as this one: Mongoose, update values in array of objects), and thought I had followed all the advice, but I am still getting this wrong, and would be very grateful if someone could spot my error, as I've been trying to debug this for hours!
The problem I am specifically having is that the findOneAndUpdate call seems to be just updating the first element in the array of "itineraryItems", no matter whether it matches my query for a specific element or not.
Data in my user collection (2 array elements in the itineraryItems array on the user document):
db.users.find({_id: ObjectId("5dd65ce7998d626a2c71a547"), {itineraryItems: 1})
{ "_id" : ObjectId("5dd65ce7998d626a2c71a547"),
"itineraryItems" : [
{ "_id" : ObjectId("5e3d5a301b65f3f9fd1621f8"),
"iType" : "event",
"startDate" : ISODate("2020-02-07T11:00:00Z"),
"endDate" : ISODate("2020-02-07T13:00:00Z"),
"includeTravelTime" : false,
"travelTimeMinutes" : 0,
"item" : null,
"event" : ObjectId("5dea66c182d9ac6fb4c6f36e")
},
{ "_id" : ObjectId("5e3d5a341b65f3f9fd1621ff"),
"iType" : "item",
"startDate" : ISODate("2020-02-07T11:00:00Z"),
"endDate" : ISODate("2020-02-07T13:00:00Z"),
"includeTravelTime" : false,
"travelTimeMinutes" : 0,
"item" : ObjectId("5e29df801f026697b71f7f48"),
"event" : null
}
]
}
My query building function:
Note that queries that satisfy the first if statement (i.e. I pass in a known id for the element in the array) seem to work fine. It is the other 2 cases that seem to fail.
function getUpdateItineraryElementQuery(user, id, type, startDate, endDate, includeTravelTime, travelTimeMinutes, itemId) {
let query = {};
if (
(id!=null) &&
(id!='not_set')
) {
query = {
_id: user._id,
'itineraryItems._id': mongoose.Types.ObjectId(id)
};
} else {
if (type==='event') {
query = {
_id: user._id,
'itineraryItems.iType': type,
'itineraryItems.startDate': startDate,
'itineraryItems.endDate': endDate,
'itineraryItems.includeTravelTime': includeTravelTime,
'itineraryItems.travelTimeMinutes': travelTimeMinutes,
'itineraryItems.event': mongoose.Types.ObjectId(itemId)
};
} else {
if (type==='item') {
query = {
_id: user._id,
'itineraryItems.iType': type,
'itineraryItems.startDate': startDate,
'itineraryItems.endDate': endDate,
'itineraryItems.includeTravelTime': includeTravelTime,
'itineraryItems.travelTimeMinutes': travelTimeMinutes,
'itineraryItems.item': mongoose.Types.ObjectId(itemId)
};
}
}
}
return query;
}
My Mongoose call to findOneAndUpdate:
User.findOneAndUpdate(
query,
{
'itineraryItems.$.startDate': dtNewStartDate,
'itineraryItems.$.endDate': dtNewEndDate,
'itineraryItems.$.includeTravelTime': newIncludeTravelTime,
'itineraryItems.$.travelTimeMinutes': newTravelTimeMinutes
// eslint-disable-next-line no-unused-vars
}, (err, doc) => {
if (err) {
// not found?
res.sendStatus(404).end();
} else {
// ok
res.sendStatus(200).end();
}
}
);
Thanks a lot if you can tell me what I am doing wrong!
You need to match your subdocument using $elemMatch if you want to update only the subdocument that matches all the conditions
Like this,
{
_id: user._id,
'itineraryItems':{
$elemMatch:{
iType: type,
startDate: startDate,
endDate: endDate,
includeTravelTime: includeTravelTime,
travelTimeMinutes: travelTimeMinutes,
item: mongoose.Types.ObjectId(itemId)
}
}
}
Ok i am trying to implement a query, which is trying to perform regex search ( which contains an array list ) on a bunch of document
Its hard for me to explain...so I am basically coming to direct point.
There is query which works with regex array...
db.paper.find({"category": {$in: [ /xd/, /sd/, /ad/ ] }})
There is query which doesn't works with regex array...
db.paper.find({"category": {$in: [ "/xd/", "/sd/", "/ad/" ] }})
So basically what I want is remove "" sign from string array...so that i can perform below query..
var sea = [ "/xd/", "/sd/", "/ad/" ];
db.paper.find({"category": {$in: sea }});
Using $in can be fairly efficient with small arrays but not so well with huge lists since it will skip around in the index to find the matching documents, or walk through the whole collection if there isn't an index to use.
Besides using the $in with the regular expression, you could use a pipe-delimited regex pattern with the keywords list like this:
Test documents:
db.papertest.insert([
{ category: "ad bd cd" },
{ category: "dd ed fd" },
{ category: "gd hd id" },
{ category: "jd kd ld" },
{ category: "md nd od" },
{ category: "pd qd rd" },
{ category: "sd td ud" },
{ category: "vd wd xd yd zd" },
]);
The magic:
var keywords = ["xd", "sd", "ad"],
regex = keywords.join("|");
db.papertest.find({
"category": {
"$regex": regex,
"$options": "i"
}
});
The results
{ "_id" : ObjectId("56bb6f171bb4f693057c0ba4"), "category" : "ad bd cd" }
{ "_id" : ObjectId("56bb6f171bb4f693057c0baa"), "category" : "sd td ud" }
{ "_id" : ObjectId("56bb6f171bb4f693057c0bab"), "category" : "vd wd xd yd zd" }
it does not work when the double quotes are present because they are interpreted as strings instead of as RegExp objects. So to make it to work, you have to convert it to RegExp objects first in Javascript like this.
var sea = [ "xd", "sd", "ad" ]; // Note: no slashes
var regex = [];
for (var i = 0; i < sea.length; i++) {
regex[i] = new RegExp(sea[i]);
}
db.paper.find({"category": {$in: regex}});
Remember, MongoDB shell uses Javascript
It seems to be working fine for me please try this
var sea = [ "xd", "sd", "ad" ];
var regex = sea.map( function( val ){
return new RegExp( '^['+val+'].*','i' );
})
db.paper.find({"category": { $in: regex }});
For this you can add a regular expression to each item in the array, you can do it in the following way.
data = ['hoLA','Que','TAL', 'Nueva'];
data = data.map(function(v, i){return new RegExp(v, 'i')});
MyCollection.find({"thing": {$in : data}}, function(err, data){
if (err) {
console.log(err)
}else{
data.forEach(function(item){
console.log(item.nombre);
})
}
});
Slightly improved ES6 + TypeScript answer based on Meme Composer comment:
const sea: string[] = [ "xd", "sd", "ad" ];
const regex: RegExp[] = sea.map((value) => new RegExp(value));
db.paper.find({ "category": { $in: regex } });
Here is simple way to transform /.*/ style regex.
var sea = [ "/xd/", "/sd/", "/ad/" ];
var rx = [];
sea.forEach(function name(value) {
var v = value.replace(/\//ig,"");
rx.push(new RegExp(v));
});
db.paper.find({"category": {$in: rx}});
I am trying to query a single embedded document in an array in MongoDB. I don't know what I am doing wrong. Programmatically, I will query this document and insert new embedded documents into the currently empty trips arrays.
{
"_id" : ObjectId("564b3300953d9d51429163c3"),
"agency_key" : "DDOT",
"routes" : [
{
"route_id" : "6165",
"route_type" : "3",
"trips" : [ ]
},
{
"route_id" : "6170",
"route_type" : "3",
"trips" : [ ]
},
...
]
}
Following queries -I run in mongo shell- return empty:
db.tm_routes.find( { routes : {$elemMatch: { route_id:6165 } } } ).pretty();
db.tm_routes.find( { routes : {$elemMatch: { route_id:6165,route_type:3 } } } ).pretty();
db.tm_routes.find({'routes.route_id':6165}).pretty()
also db.tm_routes.find({'routes.route_id':6165}).count() is 0.
The following query returns every document in the array
db.tm_routes.find({'routes.route_id':'6165'}).pretty();
{
"_id" : ObjectId("564b3300953d9d51429163c3"),
"agency_key" : "DDOT",
"routes" : [
{
"route_id" : "6165",
"route_type" : "3",
"trips" : [ ]
},
{
"route_id" : "6170",
"route_type" : "3",
"trips" : [ ]
},
...
]}
but db.tm_routes.find({'routes.route_id':'6165'}).count() returns 1.
And finally, here is how I inserted data in the first place -in Node.JS-:
async.waterfall([
...
//RETRIEVE ALL ROUTEIDS FOR EVERY AGENCY
function(agencyKeys, callback) {
var routeIds = [];
var routesArr = [];
var routes = db.collection('routes');
//CALL GETROUTES FUNCTION FOR EVERY AGENCY
async.map(agencyKeys, getRoutes, function(err, results){
if (err) throw err;
else {
callback(null, results);
}
});
//GET ROUTE IDS
function getRoutes(agencyKey, callback){
var cursor = routes.find({agency_key:agencyKey});
cursor.toArray(function(err, docs){
if(err) throw err;
for(i in docs){
routeIds.push(docs[i].route_id);
var routeObj = {
route_id:docs[i].route_id,
route_type:docs[i].route_type,
trips:[]
};
routesArr.push(routeObj);
/* I TRIED 3 DIFFERENT WAYS TO PUSH DATA
//1->
collection.update({agency_key:agencyKey}, {$push:{"routes":{
'route_id':docs[i].route_id,
'route_type':docs[i].route_type,
'trips':[]
}}});
//2->
collection.update({agency_key:agencyKey}, {$push:{"routes":routeObj}});
*/
}
// 3->
collection.update({agency_key:agencyKey}, {$push:{routes:{$each:routesArr}}});
callback(null, routeIds);
});
};
},
...
var collection = newCollection(db, 'tm_routes',[]);
function newCollection(db, name, options){
var collection = db.collection(name);
if (collection){
collection.drop();
}
db.createCollection(name, options);
return db.collection(name);
}
Note: I am not using Mongoose and don't want to use if possible.
Melis,
I see what you are asking for, and what you need is help understanding how things are stored in mongodb. Things to understand:
A document is the basic unit of data for MongoDB and can be roughly compared to a row in a relational database.
A collection can be thought of as a table with a dynamic schema
So documents are stored in collections.Every document has a special _id, that is unique within a collection. What you showed us above in the following format is One document.
{
"_id" : ObjectId("564b3300953d9d51429163c3"),
"agency_key" : "DDOT",
"routes" : [
{
"route_id" : "6165",
"route_type" : "3",
"trips" : [ ]
},
{
"route_id" : "6170",
"route_type" : "3",
"trips" : [ ]
},
...
]}
If you run a query in your tm_routes collection. The find() will return each document in the collection that matches that query. Therefore when you run the query db.tm_routes.find({'routes.route_id':'6165'}).pretty(); it is returning the entire document that matches the query. Therefore this statement is wrong:
The following query returns every document in the array
If you need to find a specific route in that document, and only return that route, depending on your use, because its an array, you may have to use the $-Positional Operator or the aggregation framework.
For Node and Mongodb users using Mongoose, this is one of the ways to write the query to the above problem:
db.tm_routes.updateOne(
{
routes: {
$elemMatch: {
route_id: 6165 (or if its in a route path then **6165** could be replaced by **req.params.routeid**
}
}
},
{
$push: {
"routes.$.trips":{
//the content you want to push into the trips array goes here
}
}
}
)
I have a big collection of songs and want to get most played songs per week, in a array. as example:
{
"_id" : {
"title" : "demons savaites hitas",
"name" : "imagine dragons"
},
"value" : {
"weeks" : [
{
"played" : 56,
"week" : 9,
"year" : 2014
}
]
}
}
It sometimes becomes:
{
"_id" : {
"title" : "",
"name" : "top 15"
},
"value" : {
"played" : 1,
"week" : 8,
"year" : 2014
}
}
The collection which i get the data from is named songs and new fields get added all the time when a songs get added. No unique artistnames or songtitles and every document in the collection looks like this:
{
"_id" : ObjectId("530536e3d4ca1a783342f1c8"),
"week" : 8,
"artistname" : "City Shakerz",
"songtitle" : "Love Somebody (Summer 2012 Mix Edit)",
"year" : 2014,
"date" : ISODate("2014-02-19T22:57:39.926Z")
}
I now want to do a mapreduce which add the new week to the array. It now overwrites it.
I also noted when trying to change to a array, not all the played get counted, with the new mapreduce.
The new mapreduce not working, with weeks:
map = function () {
if (this.week == 9 && this.year == 2014) emit({title:this.songtitle.toLowerCase(), name:this.artistname.toLowerCase()}, {played:1, week:this.week, year:this.year});
}
reduce = function(k, values) {
var result = {};
result.weeks = new Array();
var object = {played:0, week: 0, year: 0};
values.forEach(function(value) {
object.played += value.played;
object.week = value.week;
object.year = value.year;
});
result.weeks.push(object);
return result;
}
db.songs.mapReduce(map,reduce,{out: {reduce:"played2"}})
This is the old one i'm using with is a new field in the collection per week and song:
map = function () {
if (this.week == 10 && this.year == 2014) emit({title:this.songtitle.toLowerCase(), name:this.artistname.toLowerCase(), week:this.week, year:this.year}, {count:1});
}
reduce = function(k, values) {
var result = {count: 0,};
values.forEach(function(value) {
result.count += value.count;
});
return result;
}
db.songs.mapReduce(map,reduce,{out: {merge:"played"}})
I get the information fro the toplist right now from played2 like this:
db.played2.find({'_id.week': 9,'_id.year': 2014}).sort(array("value.count" => -1)).limit(50)
Above line can include any typo because i use mongoclient for php and needed to change it to javascript syntax for you.
What am I doing wrong?
I found out that I could do mapreduce as the code snippet above and then just get this week in a query and another one for previous week and do simple double for with a if to update this week with previous week place.
I made the script in python, which i run also for my mapreduce as a cronjob. As example:
if len(sys.argv) > 1 and sys.argv[1] is not None:
week = int(sys.argv[1])
else:
week = (datetime.date.today().isocalendar()[1]) - 1
year = datetime.date.today().year
previous_week = week - 1
client = MongoClient()
db = client.db
played = db.played
print "Updating it for week: " + str(week)
previous = played.find({"_id.week": previous_week, "_id.year": year}).sort("value.count", -1).limit(50)
thisweek = played.find({"_id.week": week, "_id.year": year}).sort("value.count", -1).limit(50)
thisplace = 1
for f in thisweek:
previous.rewind() # Reset second_collection_records's iterator
place = 1
if previous.count() > 0:
checker = bool(1)
for s in previous:
if s["_id"]["name"] == f["_id"]["name"] and s["_id"]["title"] == f["_id"]["title"]:
result = played.update({"_id.week": f["_id"]["week"], "_id.year": f["_id"]["year"], "_id.title": f["_id"]["title"], "_id.name": f["_id"]["name"]}, {"$set": {"place.previous_week":place, "place.this_week":thisplace}})
checker = bool(0)
print result
place = place + 1
if checker is True:
result = played.update({"_id.week": f["_id"]["week"], "_id.year": f["_id"]["year"], "_id.title": f["_id"]["title"], "_id.name": f["_id"]["name"]}, {"$set": {"place.previous_week":0, "place.this_week":thisplace}})
print result
else:
result = played.update({"_id.week": f["_id"]["week"], "_id.year": f["_id"]["year"], "_id.title": f["_id"]["title"], "_id.name": f["_id"]["name"]}, {"$set": {"place.previous_week":0, "place.this_week":thisplace}})
print result
thisplace = thisplace + 1
print "done."
This seems to work very good. Hopefully mongodb adds support to just update a field or anything in mapreduce to add information to a document without overwrite it.
I'm taking a stab at the structure of your collection based on your input fields, but I don't think mapReduce is the tool you want. Your apparent desired output can be achieved using aggregate :
db.collection.aggregate([
// Match a specific week and year if you want - remove if you want all
{ "$match": { "year": inputYear, "week": inputWeek } },
// Group to get the total number of times played
{ "$group": {
"_id": {
"title": { "$toLower": "$songtitle" },
"name": { "$toLower": "$artistname" },
"week": "$week",
"year": "$year"
},
played: { "$sum": 1 }
}},
// Sort the results by the most played in the range
{ "$sort": { "year": -1, "week": -1, "played": -1 } },
// Optionally limit to the top 15 results
{ "$limit": 15 }
])
That basically is what you appear to be trying to do. So this sums up the "number of appearances" as the number of times played. Then we take the additional steps of sorting the results, and optionally (if you can live with looking for one week at a time) limits the results to a set number. Those last two steps you won't get with mapReduce.
If you are ultimately looking for the "top ten" for each week, as a single query result, then you can look at this for a discussion (and methods to achieve) what we call the "topN" results problem.
I have a schema like following : -
var P = {
s : [{
data : [],
sec : mongoose.Schema.Types.ObjectId
}]
};
Now I want to find only the object of section not entire the row. Like If I pass sec value I want only the value of s.data of that sec object.
example : -
{ s : [
{
data : [],
sec : '52b9830cadaa9d273200000d'
},{
data : [],
sec : '52b9830cadaa9d2732000005'
}
]
}
Result should be look like -
{
data : [],
sec : '52b9830cadaa9d2732000005'
}
I do not want all entire row. Is it possible? If yes, then please help me.
You can use the $elemMatch projection operator to limit an array field like s to a single, matched element, but you can't remove the s level of your document using find.
db.test.find({}, {_id: 0, s: {$elemMatch: {sec: '52b9830cadaa9d2732000005'}}})
outputs:
{
"s": [
{
"data": [ ],
"sec": "52b9830cadaa9d2732000005"
}
]
}
You can always get the value of some field by using find(). For example in your case:
db.collectionName.find({},{s.data:1})
So the first bracket is to apply any condition or query, and in the second bracket you have to define the field as 1(to fetch only those fields value).
Please check http://docs.mongodb.org/manual/reference/method/db.collection.find for more information.
Let me know if it solves your problem.
Not into Mongo or db but working with Pure JavaSript skills here is the Solution as you mentioned Node.js which would do the execution task of the below.
Schema
var P = { s : [
{
data : [],
sec : '52b9830cadaa9d273200000d'
},{
data : [],
sec : '52b9830cadaa9d2732000005'
}
]
};
Search Method Code
var search = function (search_sec){
for (var i=0; i<(P.s.length);i++){
var pointer = P.s[i].sec;
var dataRow = P.s[i];
if((pointer) === search_sec ){
console.log(dataRow);
}
}
};
Here is How you can call - search('search_id');
For example input :
search('52b9830cadaa9d2732000005');
Output:
[object Object] {
data: [],
sec: "52b9830cadaa9d2732000005"
}
Working Demo here - http://jsbin.com/UcobuVOf/1/watch?js,console