I have a simple data structure of users and events. I am wanting to find all users who are attending the same events that the logged in user is. Users can attend multiple events.
My data is setup as follows
{
events:{
123:{
name: 'event1',
users:{
9876: true,
7564: true
}
}
},
users:{
9876:{
name: 'John',
events:{
123: true
}
},
7564:{
name: 'Peter',
events:{
123: true
}
}
}
}
I have the following code to achieve this, I was just wondering if I am on the right path and if my data structure is correct for this type of query (Firebaseref is an Angular factory)
FirebaseRef.child("users/" + authData.uid + "/events").orderByChild('displayName').once("value", function (snap) {
snap.forEach(function (event) {
FirebaseRef.child("events/" + event.key() + "/users").once("value", function (userSnap) {
userSnap.forEach(function (user) {
FirebaseRef.child("users/" + user.key()).once("value", function (realUserSnap) {
if (realUserSnap.key() != authData.uid) {
//This is a user who attends the same event
}
});
});
});
});
});
I would probably change that outermost query from a once('value' to an on('child_added' (and its other child_* siblings). The main advantage is that you're monitoring/synchronizing the data, instead of retrieving is just once. An added advantage is that it will remove the need for your first forEach.
Aside from that, this looks pretty common. The inner calls need to be once's, because you only want them to execute once. Most people get nervous because of the number of the number of on calls that will happen. But Firebase's data retrieval has little overhead after the initial websocket connection has been set up, so this typically performs pretty well.
There are lots of "should", "typically" and "may" in this answer, since the only way to be certain is for you to actually:
verify that the code functionally does what your application requires
measure the performance of the code in the conditions that you expect your users to encounter
If you do run into higher-than-expected latency, you could consider denormalizing the data a bit further. For example: you could keep the user's name in each event, where you now store true. With that, you would need to look up each user's name.
Related
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 10 months ago.
Improve this question
Im going to create a achievement system in Mongodb. But im not sure how i would format/store it in the database.
As of the users should have a progress (on each achievement they would have some progress value stored), im really confused what would be the best way to perform this, and without having an performence issue.
what should i do?, cause i dont know, what i had in mind, was maybe something like:
Should i store each achievement in an unique row in a Achievement collection, and an user array within that row, containing object with userid and achievement progress?
Would i then get an performance issue when its 1000+ achievements, that is beeing checked fairy often?
or should i do something else?
example schema for the option above:
{
name:{
type:String,
default:'Achievement name'
},
users:[
{
userid:{
type:String,
default:' users id here'
},
progress:{
type:Number,
default:0
}
}
]
}
Even though the question is specifically about the database design, I will give a solution for the tracking/awarding logic as well to establish more accurate context for the db design.
I would store the achievements progress separately from the already awarded achievements for cleaner tracking and discovery.
The whole logic is event based and has multiple layers of event handling. This gives you TONS of flexibility on how you track your data and also gives you a pretty good mechanism to track history. Basically, you can look at it as a form of logging.
Of course, your system design and contracts are highly dependent on the information you're gonna be tracking and its complexity. A simple progress field may not suffice for each case(you might want to track something more complex, not a simple number between X and Y). There is also the case of tracking data which updates quite frequently(as distance travelled in games, for example). You didn't give any context on the topic of your achievement system so we're gonna stick with a generic solution. It's just a couple of things that you should take a note about as it will affect the design.
Okay, so, let's start from the top and track the entire flow for a tracked piece of data and its eventual achievement progress. Let's say we're tracking consecutive days of user login and we're gonna award him with an achievement when he reaches [10].
Note that everything below is just a pseudo-code.
So, let's say today is [8th of July, 2017]. For now, our User entity looks like this:
User: {
id: 7;
trackingData: {
lastLogin: 7 of July, 2017 (should be full DateTime object, but using this for brevity),
consecutiveDays: 9
},
achievementProgress: [
{
achievementID: 10,
progress: 9
}
],
achievements: []
}
And our achievements collection contains the following entity:
Achievement: {
id: 10,
name: '10 Consecutive Days',
rewardValue: 10
}
The user tries to login(or visit the site). The application handler takes note of that and after handling the login logic fires an event of type ACTION:
ACTION_EVENT = {
type: ACTION,
name: USER_LOGIN,
payload: {
userID: 7,
date: 8 of July, 2017 (should be full DateTime object, but using this for brevity)
}
}
We have an ActionHandler which listens for events of type ACTION:
ActionHandler.handleEvent(actionEvent) {
subscribersMap = Map<eventName, handlers>;
subscribersMap[actionEvent.name].forEach(subscriber => subscriber.execute(actionEvent.payload));
}
subscribersMap gives us a collection of handlers that should respond to each specific action(this should resolve to USER_LOGIN for us). In our case we can have 1 or 2 that concern themselves with updating the user tracking information of lastLogin and consecutiveDays tracking properties in the user entity. The handlers in our case will update the tracking information and fire new events further down the line.
Once again, for brevity, we're gonna incorporate both into one:
updateLoginHandler: function(payload) {
user = db.getUser(payload.userID);
let eventType;
let eventValue;
if (date - user.trackingData.lastLogin > 1 day) {
user.trackingData = 1;
eventType = 'PROGRESS_RESET';
eventValue = 1;
}
else {
const newValue = user.trackingData.consecutiveDays + 1;
user.trackingData.consecutiveDays = newValue;
eventType = 'PROGRESS_INCREASE';
eventValue = newValue;
}
user.trackingData.lastLogin = payload.date;
/* DISPATCH NEW EVENT OF TYPE ACHIEVEMENT_PROGRESS */
AchievementProgressHandler.dispatch({
type: ACHIEVEMENT_PROGRESS
name: eventType,
payload: {
userID: payload.userID,
achievmentID: 10,
value: eventValue
}
});
}
Here, PROGRESS_RESET have the same contract as the PROGRESS_INCREASE but have a different semantic meaning and I would keep them separate for history/tracking purposes. If you wish, you can combine them into a single PROGRESS_UPDATE event.
Basically, we update the tracked fields that are dependent on the lastLogin date and fire a new ACHIEVEMENT_PROGRESS event which should be handled by a separate handler with the same pattern(AchievementProgressHandler). In our case:
ACHIEVEMENT_PROGRESS_EVENT = {
type: ACHIEVEMENT_PROGRESS,
name: PROGRESS_INCREASE
payload: {
userID: 7,
achievementID: 10,
value: 10
}
}
Then, in AchievementProgressHandler we follow the same pattern:
AchievementProgressHandler: function(event) {
achievementCheckers = Map<achievementID, achievementChecker>;
/* update user.achievementProgress code */
switch(event.name): {
case 'PROGRESS_INCREASE':
achievementCheckers[event.payload.achievementID].execute(event.payload);
break;
case 'PROGRESS_RESET':
...
}
}
achievementCheckers contains a checker function for each specific achievement that decides if the achievement has reached its desired value(a progress of 100%) and should be awarded. This enables us to handle all kinds of complex cases. If you only track a single X out of Y scenario, you can share the function between all achievements.
The handler basically does this:
achievementChecker: function(payload) {
achievementAwardHandler;
achievement = db.getAchievement(payload.achievementID);
if (payload.value >= achievement.rewardValue) {
achievementAwardHandler.dispatch({
type: ACHIEVEMENT_AWARD,
name: ACHIEVEMENT_AWARD,
payload: {
userID: payload.userID,
achievementID: achievementID,
awardedAt: [current date]
}
});
/* Here you can clear the entry from user.achievementProgress as you no longer need it. You can also move this inside the achievementAwardHandler. */
}
}
We once again dispatch an event and use an event handler - achievementAwardHandler. You can skip the event creation step and award the achievement to the user directly but we keep it consistent with the whole history logging flow.
An added benefit here is that you can use the handler to defer the achievement awarding to a specific later time thus effectively batching awarding for multiple users, which serve a couple of purposes including performance enhancement.
Basically, this pseudo code handles the flow from [a user action] to [achievement rewarding] with all intermediate steps included. It's not set in stone, you can modify it as you like but all in all, it gives you a clean separation of concerns, cleaner entities, it's performant, let's you add complex checks and handlers which are easy to reason about while in the same time provide a great history log of the user overall progress.
Regarding the DB schema entities, I would suggest the following:
User: {
id: any;
trackingData: {},
achievementProgress: {} || [],
achievements: []
}
Where:
trackingData is an object that contains everything you're willing
to track about the user. The beauty is that properties here are
independent from achievement data. You can track whatever and eventually use it for achievement purposes.
achievementProgress: a map of <key: achievementID, value: data> or
an array containing the current progress for each achievement.
achievements: an array of awarded achievements.
and Achievement:
Achievement: {
id: any,
name: any,
rewardValue: any (or any other field/fields. You have complete freedom to introduce any kind of tracking with the approach above),
users?: [
{
userID: any,
awardedAt: date
}
]
}
users is a collection of users who have been rewarded the given achievement. This is optional and is here only if you have the use for it and query for this data frequently.
What you might be looking for is a Badge style implementation. Just like Stack Overflow rewards it's users with badges for specific achievements.
Method 1: You can have flags in the user profile for each badge. Since you're doing it in NoSQL database, you just have to set a flag for each badge.
const badgeSchema = new mongoose.Schema({
badgeName: {
type: String,
required: true,
},
badgeDescription: {
type: String,
required: true,
}
});
const userSchema = new mongoose.Schema({
userName: {
type: String,
required: true,
},
badges: {
type: [Object],
required: true,
}
});
If your application architecture is event based, you can trigger awarding badges to users. And that operation is just inserting Badge object with progress in User badges array.
{
badgeId: ObjectId("602797c8242d59d42715ba2c"),
progress: 10
}
Update operation will be to find and update the badges array with progress percentage number
And while displaying user achievements on user interface, you can just loop over badges array to show the badges this user has achieved and their progress with it.
Method 2: Have a separate mongo collection for Badge and User Mapping. Whenever a user achieves a badge you insert a record in that collection. It will be one to one mapping of user _id and badge _id and progress value. But as the table will grow huge you will need to do indexing to efficiently query user and badge mapping.
You will have to do analysis on best approach according to your specific use case.
MongoDB is flexible enough to allow teams develop applications quickly, and involve their model with litter friction as the application needs it. In cases where you need a robust model from day one, theirs is a flexible methodology that can guide you through the process of modeling your data.
The methodology is composed of:
Workload: This stage is about gathering as much information as possible to understand your data. This will allow you formulate assumptions about, you data size the operations that will be performance against it (reads and writes), quantify operations and qualify operations.
You can get this by:
Scenarios
Prototype
Production Logs & Stats (if you are migrating).
Relationships: Identify the relationship between the different entities in your data, quantify those relationships and apply embedding or linking. In general you should prefer embedding by default, but remember that arrays should not grow without bound (6 Rules of Thumb for MongoDB Schema Design: Part 3).
Patterns: Apply schema design patterns. Take a look at Building with Patterns: A Summary, it presents a matrix that highlights the pattern that could be useful for a given use case.
Finally, the goal of this methodology is help you create a model, that can scale and perform well under stress.
If you design the achievement schema like this:
{
name: {
type: String,
default: "Achievement name",
},
userid: {
type: String,
default: " users id here",
},
progress: {
type: Number,
default: 0,
},
}
}
When an achievement is gained you just add another entry
for getting achievements Map-Reduce is a good candidate for running map reduce on the database. you can run them on a less regular basis, using them for offline computation of the data that you want.
based on documentation you can do like the following photo
i'm building an Gmail-like email browser client app prototype and i need a little help/advice structuring my React/Flux app. I decided to use pure Flux to get a better idea of how it works.
It's a simple email client with a list of letters, grouped by folders and tags and an ability to add letters to favorites.
So, i have a LettersStore containing an array of letters. The single letter data object looks something like this
{
id: 0,
new: true, //unread
checked: false,
starred: false,
folder: "inbox", //could be 'sent', 'spam', 'drafts'
sender: "Sender Name",
subject: "Re:",
snippet: "Hello there, how are you...",
body: "Hello there, how are you doing, Mike?",
date: "02.19.2016 16:30",
tags:["personal", "urgent"]
}
So what i'm trying to achieve is to let users navigate through folders (inbox, sent, drafts, spam) and filters (starred, tag, etc.)
In both folders and filters there has to be a way to select (check) some/all letters. The view state depends on how many letters are selected (the Select-all checkbox update, just like on Gmail). When the user selects a letter, the Flux action is being triggered and the state of the app updates.
The controller-view on top of the app does all the calls to the LettersStore public methods and passes the data down as props, but i'm not sure, what public methods the LettersStore should have. Currently it has:
emitChange()
addChangeListener()
removeChangeListener()
getAll() //returns array
areSomeLettersInFolderChecked(folderName) //returns bool
areAllLettersInFolderChecked(folderName) //returns bool
countNewLettersInAllFolders() //returns object
This works ok with folders, but when it comes to filters, it doesn't make sense anymore, since a starred letter is in some folder, and i feel like it's not the right thing to add specific methods like areSomeLettersInFilterChecked(filterType) etc.
Also, just like in Gmail, there has to be a way to select letter in the "Starred" filter, which belongs to the "Inbox" folder, then navigate to "Inbox" folder and keep that letter selected.
Maybe i should move the areSomeLettersInFolderChecked-like stuff to the component level?
I'm sure here has to be a proper way of doing it. Thanks in advance!
Rather than trying to encapsulate all the possible states and filters into your letter objects, keep it dumb. Normalize it and use supporting data structures to represent the other characteristics.
I'd strip it down to just the following properties:
{
id: 0,
sender: "Sender Name",
subject: "Re:",
snippet: "Hello there, how are you...",
body: "Hello there, how are you doing, Mike?",
date: "02.19.2016 16:30",
tags:["personal", "urgent"]
}
Your LetterStore can stay the same, or alternatively you could use an object or map to store letters against their id's for quick lookups later.
Now we need to represent the properties we removed from the message.
We can use individual sets to determine whether a message belongs to the new, checked and starred categories.
For instance, to star a message, just add it's id to the starred set.
var starred = new Set();
starred.add(message.id);
You can easily check whether a message is starred later on.
function isStarred(message) {
return starred.has(message.id);
}
The pattern would be the same for checked and unread.
To represent folders you probably want to use a combination of objects and sets.
var folders = {
inbox: new Set(),
sent: new Set(),
spam: new Set(),
drafts: new Set()
}
Simplifying your structures into these sets makes designing queries quite easy. Here are some examples of the methods you talked about implemented with sets.
function checkAll() {
messages.forEach(function(message) {
checked.add(message.id);
});
return checked;
}
function isChecked(message) {
return checked.has(message.id);
}
function inFolder(name, message) {
return folders[name].has(message.id);
}
// is message checked and in inbox
if(isChecked(message) && inFolder('inbox', message)) {
// do something
}
It becomes easy to construct complex queries, simply by checking whether messages belong to multiple sets.
I'm following the Meteor To-Do App tutorial with Angular integration and am learning about filtering collections. I've been able to implement a simple filter on a collection for the app I'm working on by following the principles in the tutorial, but now I'm stuck trying to figure out how to add multiple queries to the filter.
In the example, you can view incomplete tasks by toggling a checkbox. This is implemented in the controller by watching $scope.hideCompleted for changes and passing it as a Mongo query to filter the Meteor collection.
Watcher
$scope.$watch('hideCompleted', function() {
if ($scope.hideCompleted)
$scope.query = {checked: {$ne: true}};
else
$scope.query = {};
});
Collection filter
$scope.tasks = $meteor.collection(function() {
return Tasks.find($scope.getReactively('query'), {sort: {createdAt: -1}})
});
How do I make the query support multiple filters? For example, say I've chosen to extend the example and have ranked each to-do item by priority. I then would have an an input field for the user to filter the collection by priority, whose value is bound to $scope.priority. Now, if I wanted to filter the to-do list by incomplete and priority=$scope.priority tasks, I understand the Mongo query would need to be something along the lines of Tasks.find({ $and: [{ checked: {$ne: true} },{ priority: $scope.priority }]},{ sort: { createdAt: -1 } }).
In my app, I've been able to make two watchers properly track changes to two scope variables, analogous to my example with $scope.hideCompleted and $scope.priority, but I don't know how to take the next step to merge the queries when filtering the collection. I've also tinkered around a little with this package, since I'll eventually hope to be able to filter and sort by many criteria, but I didn't get too far with it before switching to the concepts I've described here.
I'd appreciate any help with this. Thank you!
This would be my approach:
$meteor.autorun($scope, function() {
// uncomment subscribe after you've got to that point
// $scope.$meteorSubscribe('yourSubscription').then(function() {
$scope.tasks = $scope.$meteorCollection(function() {
return Tasks.find({
checked: $scope.getReactively('model.hideCompleted'),
priority: $scope.getReactively('model.priority')
}, { sort: { createdAt: -1 } });
});
// });
});
A couple of things here:
Once you have removed autopublish you can uncomment the $scope.$meteorSubscribe method and replace "yourSubscription" with the name of your actual subscription.
$meteor.autorun will fire every time any getReactively variable changes.
$scope.$meteorSubscribe and $scope.$meteorCollection are favored as they will remove the subscriptions and object/collection when the scope is destroyed.
If you have any issues then perhaps I can setup a demo for you to look at.
Well, I guess I was a lot closer than I had expected, so I'll answer my question and share what I did to implement multiple filters with regards to the hypothetical extension of the to-do app.
I made hideCompleted and priority scope variables into properties of a scope object model, and used a single watcher with the argument true at the end to check for object equality (for any changes to model or its properties). I then generated $scope.query by stitching together "sub-queries." I've added the code below.
This seems to be working fine for now, but I'm not sure if it's the best solution, so I will continue experimenting, and I will update my answer if I find anything better. I'd be interested in any other approaches, though!
Watcher
var initQuery=true;
var subQueries=[];
$scope.$watch('model', function() {
if (!initQuery){
subQueries=[];
if ($scope.model.hideCompleted)
subQueries.push({checked: {$ne: true}});
if ($scope.model.priority)
subQueries.push({priority: $scope.model.priority});
$scope.query = { $and: subQueries};
} else {
initQuery = false;
$scope.query = {};
}
}, true);
Filter Collections (unchanged)
$scope.tasks = $meteor.collection(function() {
return Tasks.find($scope.getReactively('query'), {sort: {createdAt: -1}})
});
I've been working with react/flux for a few weeks now and while I feel like I've got a pretty good handle on everything from async loading to updating props/states/etc, one thing that is still bothering me is how to handle save states.
For example, when loading data, I just have an isLoading boolean parameter in my store that gets passed to my components. But when I try and post an updated object to the server, it's trivial to:
fire the update action
display a "save in progress" state
but figuring out the result of the update action seems to be way more difficult.
Probably the most applicable post I've seen on this is in Fluxxor's async data guide, but their solution (adding/modifying a status property on the object) feels error-prone to me.
onAddBuzz: function(payload) {
var word = {id: payload.id, word: payload.word, status: "ADDING"};
this.words[payload.id] = word;
this.emit("change");
},
onAddBuzzSuccess: function(payload) {
this.words[payload.id].status = "OK";
this.emit("change");
},
onAddBuzzFail: function(payload) {
this.words[payload.id].status = "ERROR";
this.words[payload.id].error = payload.error;
this.emit("change");
}
Is there a better way to manage save states or is adding a status property to the object the best way?
I recommend keeping your "model stores" and "ui stores" separate, or at least accessed via different cursor positions in the same store. So, in your case you'd have one store or branch for your "word model" and then another store or branch for "word status."
While this adds some complexity in the form of breaking up logic across stores and reacting twice to the AddBuzz action, it ends up reducing (more) complexity by confining model store updates to true changes in model data and managing ui states separately.
EDIT
This is what Relay will more-or-less be doing, keeping persisted data in a separate self-managed store, leaving custom stores for nothing but ui state. Some other libraries like https://github.com/Yomguithereal/baobab also recommend this approach. The idea is that these are fundamentally different kinds of state. One is domain-specific persisted data and the other is ui-specific ephemeral application state.
It might look something like this:
model_store.js:
onAddBuzz: function(payload) {
var word = {id: payload.id, word: payload.word};
this.words[payload.id] = word;
this.emit("change");
}
ui_store.js:
onAddBuzz: function(payload) {
this.wordStates[payload.id] = 'ADDING';
this.emit("change");
}
my_view_controller.js:
// listen to both stores and pass down both words and wordStates to your views
my_word_view.js:
...
render: function() {
var word = this.props.word,
wordState = this.props.wordState;
...
}
If you don't want to emit two change events, have one waitFor the other and emit the change only from the second one.
I've found all kinds of useful posts/info about $q, $q.all, and chaining in general, but I haven't found any examples of my exact issue, so I'm wondering whether what I want to do is even possible. I've got a series of calls I need to make, and each is dependent on the previous BUT there's a step in the middle where the series splits into several parallels. The gist of it goes something like this:
call #1: get keyword (based on inputs)
call #2: with keyword, get list of items
call #3: with item ID#s, get item profiles
call #4: with data in profiles, get item images
(I know, I know, four calls for just one is ridiculous, let alone sixteen calls total, but this is for a prototype. I don't have to be ultra-speedy. I just need to prove I can get to that final step with the existing data/calls.)
Anyway, this means that #1 is one call for all... and then as of #2, it splits up and I have to do #2, #3, and #4 for each different ID# I got from #1's return. I've mocked up a plunk using jsonplaceholder. It should retrieve the first set, then do the next two calls with the retrieved IDs, one loop per ID.
I've tried a simple for loop (which Plunker tells me I can't do w/a function in it). I've tried an angular.forEach (inside a $scope.watch and outside it), but that doesn't seem to get me anything at all:
angular.forEach($scope.resultsb, function(value, key) {
console.log(key + ': ' + value);
$http.get('http://jsonplaceholder.typicode.com/users?id='+ value)
.then(function(res2){
$scope.data2 = res2.data;
var obj = {
title: $scope.results1[i].title,
id: $scope.data2[i].id,
username: $scope.data2.username
};
$scope.results2.push(obj);
I think the forEach just isn't kicking in properly b/c it can't really run until after I get $scope.results1 (the first step's results), but can I chain an $http call to a loop?
You can see the full thing (in its current version, which may change since I continue to beat at it) at http://plnkr.co/edit/CjXt7E?p=preview. Is what I'm trying even possible, or do I have to do this long-hand?
Your for each will run even without response as it is Asynchronous, instead make a function and call it when you get the callback in .then()
Edited your plunker: http://plnkr.co/edit/mxvvCB?p=preview
like this:
$scope.Call = function(value) {
if(typeof value[$scope.counter] !== 'undefined'){
value = value[$scope.counter].id;
var i =$scope.counter;
console.log(value+' ' +i);
$http.get('http://jsonplaceholder.typicode.com/users?id='+ value)
.then(function(res2){console.log(res2);
$scope.data2 = res2.data;
var obj = {
title: $scope.results1[i].title,
id: $scope.data2[0].id,
username: $scope.data2.username
};
$scope.results2.push(obj);
$scope.counter++;
$scope.Call($scope.resultsb);
});
}
};
I hope it is what you are trying to achieve.
I GOT IT TO WORK.
This might be the worst case of function-use ever, but it works, and I get results. Not entirely -- I think some of the calls time out, so I figured out I need to ask for, like, 6, and then I get at least the 4 results I want. Which is probably indicative of the fact that this desperately calls for a single endpoint instead of doing the logic on the frontend, but it works enough for a prototype, so I'm happy (for now).
In my controller, I get the first set of IDs, and using a simple for-loop, I assign a loop-id to each. Then I call getUser(), with num (to connect the info to the first set of results), and id (to connect to the actual user I'm tracking). Here's the function w/ its internal chains:
function getUser(num, id) {
$http.get('http://jsonplaceholder.typicode.com/users?id='+ id)
.then(function(res2){
$scope.data2 = res2.data;
if ($scope.data2[0].username !== 'undefined') {
var obj = {
num: $scope.results1[num].num,
id: $scope.results1[num].id,
title: $scope.results1[num].title,
username: $scope.data2[0].username
};
$scope.results2.push(obj);
}
return $http.get('http://jsonplaceholder.typicode.com/comments?id=' + id);
}).then(function(res3){
if ($scope.results2[num].username !== 'undefined') {
$scope.data3 = res3.data;
var obj = {
num: num,
id: $scope.results2[num].id,
title: $scope.results2[num].title,
username: $scope.results2[num].username,
email: $scope.data3[0].email
};
$scope.results3.push(obj);
}
});
}
You can see the entire thing at http://plnkr.co/edit/CjXt7E?p=preview -- it shows the output of each step (results1, results2, results3). Hopefully I won't have to do something this crazy again, but just in case, maybe this will help someone. Or maybe it'll cause someone to leap up and tell me everything I did wrong, but that'd be fine, too. Gotta learn somehow. :)