I have a very large list of users (in the millions) stored in a database (Mongo in our case). Each user has a score associated with it. The score is constantly changing based on actions of the user.
I want to be able to query for the position of each user when the list is sorted by score.
Right now, I have an index on the score. I query for all users, sort them by score, and then iterate through to find the position of the user I want.
var users = User.find({})
.sort({
score: -1
});
var position = 1;
for (
let user = await users.next(); user !== null; user = await users.next()
) {
if (user === targetUser) {
return position;
}
position++;
}
Sorting is a slow operation but from my understanding the index orders it already, so this is only as expensive as fetching all the records.
I can maybe increase the speed of finding the element by implementing a binary search since the list is sorted.
The bigger problem is storing the list in memory (I keep running into heap out of memory errors in Node).
What is the right database + data structure to achieve what I want?
You may leverage $setWindowFields and $rank available starting from MongoDB v5.0+.
db.collection.aggregate([
{
"$setWindowFields": {
"partitionBy": null,
"sortBy": {
"score": -1
},
"output": {
"rank": {
$rank: {}
}
}
}
},
{
$match: {
_id: <target user's identifier>
}
}
])
Here is the Mongo playground for your reference.
#barrypicker I tried this and it is slightly faster, but still very slow (on the order of 20 seconds). But it may solve my memory issue problem I think?
// Get target user
let target = await User.findOne({
_id: targetId
});
// Find number of users with lower score without bringing all of them into memory
let lowerScoreCount = await User.count({
score: { $lt: target.score }
});
// Find all users with same score
let sameScoreSignups = await User.findOne({
score: target.score
});
// Iterate over users with same score till you find your user and add this to the lower score count to get the position of the user in the sorted list
let position = lowerScoreCount;
sameScoreSignups.forEach((user) => {
if (user._id === targetId) {
return position;
}
position++;
});
return position;
I have two arrays of users and beams
struct User {
var id: Int
var jobTitle: String
}
struct Beams {
var isSelected = false
var id: Int
var userIds: [Int]
}
If the user selects a particular beam, all the users which are assigned to that beam will be selected of the first array. The user can select multiple beams and i am getting the data from api response, and i have to send the selected users in a post call. so i have to select all the selected users from an array of beams and on the basis of filtered ids, send that array in a post call. initially i am filtering all the beams which are selected like this
beams.filter { $0.isSelected }.compactMap { $0.userIDS }
which gives me an array of [Int]. These are the userIds which have to be sent. I somehow can't figure out how will i select these particular ids from my array of users which contains other attributes as well. contains can be used to filter one element but not a sequence. and even if i filter or take an intersection of these ids and my userArray. i am still left with a set or an array of ids. from which i'd have to generate my array, i want to keep all the other attributes as well. Any help in the right direction would be appreciated. I have something of this nature in my mind
let selectedBeamsIds = beams.filter { $0.isSelected }.compactMap { $0.userIDS }
users.append(selectedBeamsIds.compactMap { getUniqueUsersForBeams(ids: $0) } )
private func getUniqueUsersForBeams(ids: [Int]) -> [User] {
let ower = users.reduce(into: [User]()) { (result,
user) in
let filteredUserIds = ids.filter { $0 == user.id }
//result.append(use)
}
}
I also need to do this with another array of jobTitles which is like
struct JobTitle {
var jobTitle: String
var isSelected = false
}
this one is straight forward as if the users's jobTitle matches with the selected items of [String] array then select that user to be sent.
Try,
let selectedIds = beams.filter { (beams) -> Bool in
beams.isSelected
}.flatMap { (beams) -> [Int] in
beams.userIds
}
let filteredUsers = users.filter { (user) -> Bool in
selectedIds.contains(user.id)
}
1 ) Filter out selected beams
2 ) Collect all the userIds in the selected beams
3 ) Filter users by checking if the id is present in the previously collected ids.
My code stays in the second for forever, testing the same category every step and decrementing every time.
I have two arrays, one of them is called categoriesToUpdate and is a list of category ids (string values) for categories that I have to update, and the other is called categories, containing all the actual category data I'm working with.
I have to test if the id value for a category that I have to update is the same as the database and if it is, decrement the attribute position of its object and update the database. But it is infinitely decrementing the bank.
let newCategory;
let name;
let position;
for(let id of categoriesToUpdate) {
for(let cat of categories) {
if(id === cat.id) {
position = cat.category.category.lastPosition - 1;
name = cat.category.category.categoryName;
newCategory = {
category: {
categoryName: name,
lastPosition: position,
}
}
cRef.child(id).update(newCategory);
}
}
}
Examples of the two arrays:
categoriesToUpdate = ['-lGz4j77...', '-uEbKO3...', ...]
and
categories = [
{
category: {
category: {
categoryName: "name",
lastPosition: "number",
}
},
id: "category id";
},
{
...
}
]
it is difficult to explain how I get the arrays, but basically, categoriesToUpdate is an array of ids that I add to my logic, I have to do update in each of these categoriesand categories is an array that has all categories of the database, comes from Firebase.
let id of categoriesToUpdate. I'm assuming categoriesToUpdate is an array of Category objects. So id is actually a Category object. To compare it should be id.id === cat.id.
Also you can try filter instead of a second loop.
Something like
var matched = categories.filter(c => c.id === id.id)[0];
Then compare matched. Nested loops are hard to read, imo.
I have a multiple search field. I want to be available to loop through an array of properties and filter out the correct data that it corresponds too and display the data in the table I have it working but in a different way. Please look at old code and new code. New code does not work and new code is using for loops to match properties and they applying the correct filter. Each item hits the .filter() function. So 9 items will hit it 9 times to do the filtering on each item. This is a reusable component.
Old Code
// filter is a get and set, depending on the items it will run this code; I have 9 items and each item will hit .filter(item)
.filter((item: Array<any>) => {
// item looks like this:
item = {"description": "item name", "unit": "56", "ourPrice": 200, "notes": "lorem", "seller": "lorem ipsum", "manufacturer":"lorem ipsum"};
let SelectedItem = item.description.toLowerCase();
// this.filter is an array of values i type in the search field
// I do this for each property in its own .filter() and it works and if it is number value I do it based on number .
return SelectedItem.indexOf(this.filter.description.toLowerCase());
}).filter((item: Array<any>) => {
// add so on
});
New Code
.filter((item: Array<any>) => {
var _item;
// this returns just a string of values but this is only shown here for visual but not written here
this._sortProperties = {"description", "unit", "ourPrice", "notes", "seller", "manufacturer" };
for(let _i = 0; _i < this._sortProperties.length; _i++) {
let _filterData = item[this._sortProperties[_i]];
if(typeof _filterData === 'string' || _filterData instanceof String) {
_item = _filterData.toLowerCase();
_item.indexOf(this.filter[this._sortProperties[_i]].toLowerCase()) !== -1;
} else if(isNaN(_filterData) === true) {
_item = _filterData;
_item !== -1;
}
}
return _item;
});
I have a big collection of songs and want to get most played songs per week, in a array. as example:
{
"_id" : {
"title" : "demons savaites hitas",
"name" : "imagine dragons"
},
"value" : {
"weeks" : [
{
"played" : 56,
"week" : 9,
"year" : 2014
}
]
}
}
It sometimes becomes:
{
"_id" : {
"title" : "",
"name" : "top 15"
},
"value" : {
"played" : 1,
"week" : 8,
"year" : 2014
}
}
The collection which i get the data from is named songs and new fields get added all the time when a songs get added. No unique artistnames or songtitles and every document in the collection looks like this:
{
"_id" : ObjectId("530536e3d4ca1a783342f1c8"),
"week" : 8,
"artistname" : "City Shakerz",
"songtitle" : "Love Somebody (Summer 2012 Mix Edit)",
"year" : 2014,
"date" : ISODate("2014-02-19T22:57:39.926Z")
}
I now want to do a mapreduce which add the new week to the array. It now overwrites it.
I also noted when trying to change to a array, not all the played get counted, with the new mapreduce.
The new mapreduce not working, with weeks:
map = function () {
if (this.week == 9 && this.year == 2014) emit({title:this.songtitle.toLowerCase(), name:this.artistname.toLowerCase()}, {played:1, week:this.week, year:this.year});
}
reduce = function(k, values) {
var result = {};
result.weeks = new Array();
var object = {played:0, week: 0, year: 0};
values.forEach(function(value) {
object.played += value.played;
object.week = value.week;
object.year = value.year;
});
result.weeks.push(object);
return result;
}
db.songs.mapReduce(map,reduce,{out: {reduce:"played2"}})
This is the old one i'm using with is a new field in the collection per week and song:
map = function () {
if (this.week == 10 && this.year == 2014) emit({title:this.songtitle.toLowerCase(), name:this.artistname.toLowerCase(), week:this.week, year:this.year}, {count:1});
}
reduce = function(k, values) {
var result = {count: 0,};
values.forEach(function(value) {
result.count += value.count;
});
return result;
}
db.songs.mapReduce(map,reduce,{out: {merge:"played"}})
I get the information fro the toplist right now from played2 like this:
db.played2.find({'_id.week': 9,'_id.year': 2014}).sort(array("value.count" => -1)).limit(50)
Above line can include any typo because i use mongoclient for php and needed to change it to javascript syntax for you.
What am I doing wrong?
I found out that I could do mapreduce as the code snippet above and then just get this week in a query and another one for previous week and do simple double for with a if to update this week with previous week place.
I made the script in python, which i run also for my mapreduce as a cronjob. As example:
if len(sys.argv) > 1 and sys.argv[1] is not None:
week = int(sys.argv[1])
else:
week = (datetime.date.today().isocalendar()[1]) - 1
year = datetime.date.today().year
previous_week = week - 1
client = MongoClient()
db = client.db
played = db.played
print "Updating it for week: " + str(week)
previous = played.find({"_id.week": previous_week, "_id.year": year}).sort("value.count", -1).limit(50)
thisweek = played.find({"_id.week": week, "_id.year": year}).sort("value.count", -1).limit(50)
thisplace = 1
for f in thisweek:
previous.rewind() # Reset second_collection_records's iterator
place = 1
if previous.count() > 0:
checker = bool(1)
for s in previous:
if s["_id"]["name"] == f["_id"]["name"] and s["_id"]["title"] == f["_id"]["title"]:
result = played.update({"_id.week": f["_id"]["week"], "_id.year": f["_id"]["year"], "_id.title": f["_id"]["title"], "_id.name": f["_id"]["name"]}, {"$set": {"place.previous_week":place, "place.this_week":thisplace}})
checker = bool(0)
print result
place = place + 1
if checker is True:
result = played.update({"_id.week": f["_id"]["week"], "_id.year": f["_id"]["year"], "_id.title": f["_id"]["title"], "_id.name": f["_id"]["name"]}, {"$set": {"place.previous_week":0, "place.this_week":thisplace}})
print result
else:
result = played.update({"_id.week": f["_id"]["week"], "_id.year": f["_id"]["year"], "_id.title": f["_id"]["title"], "_id.name": f["_id"]["name"]}, {"$set": {"place.previous_week":0, "place.this_week":thisplace}})
print result
thisplace = thisplace + 1
print "done."
This seems to work very good. Hopefully mongodb adds support to just update a field or anything in mapreduce to add information to a document without overwrite it.
I'm taking a stab at the structure of your collection based on your input fields, but I don't think mapReduce is the tool you want. Your apparent desired output can be achieved using aggregate :
db.collection.aggregate([
// Match a specific week and year if you want - remove if you want all
{ "$match": { "year": inputYear, "week": inputWeek } },
// Group to get the total number of times played
{ "$group": {
"_id": {
"title": { "$toLower": "$songtitle" },
"name": { "$toLower": "$artistname" },
"week": "$week",
"year": "$year"
},
played: { "$sum": 1 }
}},
// Sort the results by the most played in the range
{ "$sort": { "year": -1, "week": -1, "played": -1 } },
// Optionally limit to the top 15 results
{ "$limit": 15 }
])
That basically is what you appear to be trying to do. So this sums up the "number of appearances" as the number of times played. Then we take the additional steps of sorting the results, and optionally (if you can live with looking for one week at a time) limits the results to a set number. Those last two steps you won't get with mapReduce.
If you are ultimately looking for the "top ten" for each week, as a single query result, then you can look at this for a discussion (and methods to achieve) what we call the "topN" results problem.