In my chat app (+20m users for example) i need to get last updated chats but there's no optimal solution.
Chats
{
id: 1,
title: 'My Group',
updated_at: 137974654
},
{
id: 2,
title: 'Gamers',
updated_at: 137973654
}
Members
{
chat_id: 1,
user_id: 'A'
},
{
chat_id: 2,
user_id: 'B'
}
I don't want to embed members because a user can join thousands of chats so document size is beyond 16MB.
The maximum BSON document size is 16 megabytes.
A potential problem with the embedded document pattern is that it can lead to large documents, especially if the embedded field is unbounded. In this case, you can use the subset pattern to only access data which is required by the application, instead of the entire set of embedded data.
https://docs.mongodb.com/manual/tutorial/model-embedded-one-to-many-relationships-between-documents/#subset-pattern
My solution but not efficient
I first tried to get a list of chat_ids then query chats:
const members = Members.find({ user_id: 'A' }).project({ chat_id: 1 });
const chat_ids = members.map(item => item.chat_id);
const chats = Chats.find({
id: {
$in: chat_ids
}
}).sort({ updated_at: -1 }).limit(20);
But as i said, What if a user has joined +100000 chats? so the first query is too large + the last query is too slow.
What should i do? Do i need a relational database?
Making some assumptions that you are only interested in user_id: 'A', and keeping in mind that I am a MongoDB noob, and I only tested this with the example data you provided, does this do what you want?
db.members.aggregate([
{
$match: {
user_id: "A"
}
},
{
$lookup: {
from: "chats",
localField: "chat_id",
foreignField: "id",
as: "thechats"
}
},
{
"$unwind": "$thechats"
},
{
"$replaceRoot": {
"newRoot": "$thechats"
}
},
{
"$sort": {
updated_at: -1
}
},
{
"$limit": 20
}
])
Try it at mongoplayground.net.
Related
I am trying to do an update many.
The problem that I am encountering is getting the length of an array and seeing if it is less than or equal to another number being passed in from the params.
This is what I have done so far. But I feel this is a bad way of doing it. if anyone has a better way of doing it, please give your input. trying to improve :)
const updateStatus = (materialId, attempts) => {
return db.collection.updateMany(
{
_id,
[`data.${attempts}`]: { $exists: true },
status: { $in: ['outstanding', 'overdue'] },
},
{ status: 'restricted' }
)
}
this is the data I am trying to update many on
[{
_id: 1,
status: 'outstanding',
data:[],
},
{
_id: 2,
status: 'overdue',
data:[{passed: true},{passed: false}],
}
{
_id: 3,
status: 'outstanding',
data:[{passed: true},{passed: false}, {passed: false}],
}
]
In my collection of users I have the following
{
_id: ObjectId('whatever user id'),
movies: [
{
_id: ObjectId('whatever id of this movie'),
name: 'name of this movie',
actors: [
{
_id: ObjectId('whatever id of this actor'),
name: 'name of this actor'
}
]
}
]
}
So in my users collection I want to be able to query for a actor by the user.id, pet.id, and the actor.id
I want to return the actor somewhat like this...
actor: {
fields...
}
I tried the following...
const actor = await User.findById(req.user.id, {
movies: {
$elemMatch: {
_id: req.params.movie_id,
actors: {
$elemMatch: {
_id: req.params.actor_id,
},
},
},
},
});
I have tried other things but can't seem to get it to work. I saw that you can maybe use aggregate but I am not sure how to query that while using the ids I have at my disposal.
I was able to figure it out by using aggregate. I was using this before but it seems that I needed to cast my ids with mongoose.Types.ObjectId so a simple req.user.id would not work.
In order to get my answer I did...
const user = await User.aggregate([
{ $match: { _id: mongoose.Types.ObjectId(req.user.id) } },
{ $unwind: '$movies' },
{ $match: { 'movies._id': mongoose.Types.ObjectId(req.params.movie_id) } },
{ $unwind: '$movies.actors' },
{
$match: {
'movies.actors._id': mongoose.Types.ObjectId(req.params.actor_id),
},
},
]);
This did not return data in the following format...
actor: {
fields...
}
but returns it instead like this...
user: {
movies: {
actor: {
fields...
}
},
otherFields...
}
then sending the response back...
res.status(200).json({
status: 'success',
data: {
actor
}
})
gives that format I wanted. However, I would still want to know how to just get the data actor without getting the full document
I have an aggregate function that returns people in a collection:
const getById = ({ id }) => {
return Project.aggregate([
{ $match: { _id: Types.ObjectId(id) } },
{
$lookup: {
from: "members",
localField: "_id",
foreignField: "project_id",
as: "members"
}
},
])
.then(data => {
const [project] = data;
console.log(project) // see below
return {
id: project._id,
...project
};
})
.catch(err => console.log(err));
}
If I return the data from this I get the following:
// Server response
{ _id: 5e2f57b577a8ce59c79e74af,
title: 'ok',
user_id: 5e2dc7961e6b840c315b5a03,
__v: 0,
members:
[ { _id: 5e447683b4f732cc9c4a9531,
name: 'Karl Taylor',
email: 'karl#queuey.dev',
project_id: 5e2f57b577a8ce59c79e74af,
position: 1,
__v: 0 },
{ _id: 5e45be128ed96a5eaef5d13e,
name: 'John Smith',
email: 'john#queuey.dev',
project_id: 5e2f57b577a8ce59c79e74af,
position: 2,
__v: 0 } ] }
However, when I query from the frontend using Apollo GraphQL, the id is null. (But it works on other items, as id is a getter for id but this does not happen on aggregate functions).
What is the best practice to map the id to the correct value? I would normally just use array.map but I feel like that might be overkill if I have too many members (at which point I would probably break this out to do pagination, but that's a different story.)
This is the response from frontend
// Client response
"project": {
"id": "5e2f57b577a8ce59c79e74af",
"title": "ok",
"members": [
{
"id": null, // <-- Notice here is null
"name": "Karl Taylor",
"email": "karl#queuey.dev",
"__typename": "Member"
},
{
"id": null, // <-- Notice here is null
"name": "John Smith",
"email": "john#queuey.dev",
"__typename": "Member"
}
],
"__typename": "Project"
}
This question here is similar, however, I do not believe it is a duplicate because we are querying different data. (the answer does not solve my question.)
I need to be able to return id otherwise cached redirects will not work.
I ran into this problem as well when I try to use aggregations.
simply you can you both ID's(id and _id) but it's not good thing.
what I use is to iterate the data
if the returning data of the query is not array use this
return {
...res._doc,
id: res._id,
}
but the returning data is an array you can add id to the response variable like below using forEach
res.forEach(element => {
element.id = element._id
});
return res;
Details
I develop survey application with express and struggle with some getting of data.
The case:
you can get all surveys by "GET /surveys". And every survey doc has to contains hasVoted:mongoose.Bool and optionsVote:mongoose.Map if the user has voted for the survey. (SurveySchema is bellow)
you can vote for survey by "POST /surveys/vote"
you can see the results of any survey only if you vote for it
new Schema({
question: {
type: mongoose.Schema.Types.String,
required: true,
},
options: {
type: [{
type: mongoose.Schema.Types.String,
required: true,
}]
},
optionsVote: {
type: mongoose.Schema.Types.Map,
of: mongoose.Schema.Types.Number,
},
votesCount: {
type: mongoose.Schema.Types.Number,
},
votes: {
type: [{
user: {
type: mongoose.Schema.Types.ObjectId,
ref: 'User',
},
option: mongoose.Schema.Types.Number,
}]
},
})
Target:
So the target of the question is how to add fields hasVoted and optionsVote if there is "Vote" sub document in votes array where user===req.user.id ?
I believe you got the idea so if you have an idea how to change the schema to achieve the desired result I'm open!
Example:
Data:
[{
id:"surveyId1
question:"Question",
options:["op1","op2"],
votes:[{user:"userId1", option:0}]
votesCount:1,
optionsVote:{"0":1,"1":0}
},{
id:"surveyId2
question:"Question",
options:["op1","op2"],
votes:[{user:"userId2", option:0}]
votesCount:1,
optionsVote:{"0":1,"1":0}
}]
Route handler:
Where req.user.id='userId1' and then make the desired query.
The result
[{ // Voted for this survey
id:"surveyId1
question:"Question",
options:["op1","op2"],
votes:[{user:"userId1", option:0}]
votesCount:1,
optionsVote:{"0":1,"1":0},
hasVoted:true,
},{ // No voted for this survey
id:"surveyId2
question:"Question",
options:["op1","op2"],
votesCount:1,
}]
In MongoDB, you can search for sub document as follows
//Mongodb query to search for survey filled by a user
db.survey.find({ 'votes.user': myUserId })
So with this when you can get results only where user has voted, do you really need hasVoted field?
To have optionsVote field, first I would prefer schema of optionsVote as {option: "a", count:1}. You can choose any of the following approach.
A. manage to update optionsVote field at the time of update by incrementing the count of the voted option when you POST /survey/vote.
B. Another approach would be to calculate the optionsVote based on votes entries at the time of GET /survey. You can do this via aggregate
//Mongodb query to get optionsVote:{option: "a", count:1} from votes: { user:"x", option:"a"}
db.survey.aggregate([
{ $unwind: "$votes" },
{ $group: {
"_id": { "id": "_id", "option": "$votes.option" },
optionCount: { $sum: 1 }
}
},
{
$group: { "_id": "$_id.id" },
optionsVote: { $push : { option: "$_id.option", count: "$optionCount" } },
votes: { $push : '$votes'}
}
])
//WARNING: I haven't tested this query, this is just to show the approach -> group based on votes.option and count all votes for that option for each document and then create optionsVote field by pushing all option with their count using $push into the field `optionsVote`
I recommend approach A because I assume POST operations would be quite less than GET operations. Also it's easier to implement. Having said that, keeping query in B handy will help you with sanity check.
I have a set of data in MongoDB with parse-server in the following format-
Rating => objectId, user<_User>, rating...
_User => objectId, gender<m|f|nb|na>
I have been trying to group the data based on the user's gender to find out how many male, female, non-binary or N/A users have rated. user field in a pointer reference to _User. I am using the following aggregate pipeline.
const pipeline = [
{
lookup: {
from: '_User',
localField: 'user',
foreignField: 'objectId',
as: 'user'
}
},
{
unwind: { path: '$user' }
},
{
group: {
objectId: '$user.gender',
count: {
$sum: 1
}
}
}
]
const data = await new Query('Rating').aggregate(pipeline)
Result =>
[
{
"count": 54,
"objectId": "na"
},
{
"count": 405,
"objectId": null
},
{
"count": 27,
"objectId": "f"
},
{
"count": 540,
"objectId": "m"
}
],
However, returned data count doesn't match with actual data. The actual database has only 27 ratings with 1 f, 2 na, 24 m.
For MongoDB developers, objectId is equavalent to _id.
I am a novice to aggregation framework. What am I doing wrong?
Server Environment-
parse-server: 3.2.3
mongodb: 4.0.2
It is tricky because you need to understand how Parse Server stores the data inside the MongoDB. The following query should solve your problem:
const query = new Parse.Query('Rating');
const pipeline = [
{
project: {
objectId: 1,
userId: { $substr: ['$_p_user', '_User$'.length, -1] }
}
},
{
lookup: {
from: '_User',
localField: 'userId',
foreignField: '_id',
as: 'user'
}
},
{
unwind: { path: '$user' }
},
{
group: {
objectId: '$user.gender',
count: {
$sum: 1
}
}
}
];
return await query.aggregate(pipeline, { useMasterKey: true });