Data Modeling Best Practices in Firebase/AngularFire - angularjs

I'm developing an application in Firebase for the first time and was curious how I should model the data between two objects, a user and a post. I come from more of a relational db background and was curious not only how this would be done in nonrelational DBs but specifically how to set up a relationship between two objects in Firebase.
For example, my application has many Users, and each user creates many Posts.
User {
firstName: String,
lastname: String,
userName: String
}
Post {
title: String,
content: String,
date: Date,
writtenBy: [User object?]
}
How should I structure these two objects in Firebase so that a Post belongs to a User, but all Posts can be queried for regardless of User, and both User and Post objects can be edited without disrupting the other object's data and/or relationship?
And how should I create new "relational" objects via firebase:
sync.$set({userA: {
firstname: "Billy",
lastName: "Bob",
userName: "BillyBob",
Posts: {
// .....
}
}
});
Thanks!

Firebase is built with performance in mind. This is the reason you have to design data structures differently, normalization is your enemy in most cases. Every object in Firebase can be accessed by URL, and you should always keep this in mind.
There are still many ways of designing the data structures, it depends on what queries do you want to execute. If one of the queries is to be able to display all messages (I believe a number of latest messages would be the most common use case), but at the same time you want to be able to show messages per user than one of the possible data structures could look like this:
User {
userId(assigned by Firebase automatically) {
firstName: String,
lastname: String,
userName: String
}
}
Post {
User {
userId(matching userId in the User object) {
postId(assigned by Firebase for every new post automatically) {
title: String,
content: String,
date: Date,
writtenBy: String, userName or userId (this is not really needed, but may keep it for easier data access)
}
}
}
}
Then you can change any user data without triggering data change events in Posts, like in your example, (which would be extremely heavy if you have large number of messages).
You can get all messages independently of user:
var postListRef = new Firebase(URL);
var lastPostQuery = postListRef.child("Post").limit(500);
You can also use startAt() and endAt() quesries https://www.firebase.com/docs/web/api/query/limit.html
As a drawback - you have to unpack every message in the for loop if you need to show only messages, but I would expect you would show user info as well, so it should be ok.
If you want to listen for just one user messages, it's very simple and fast:
var postListRef = new Firebase(URL);
var lastPostQuery = postListRef.child("Post/User").child(userId);
And Angular/AngularFire has great support for this kind of data structures.

I am also new to Firebase, I would recommend the following structure.
Users: {
userID: {
firstName: String,
lastname: String,
userName: String,
posts: {
postID1:true,
postID2:true
}
Posts: {
postID1:{
title: String,
content: String,
date: Date,
writtenBy: userID
}
}
It allows you to get the latest posts without having to through any users. Plus you can get all the post made by any user.

Related

How to push an object in array field using Firebase DB

I want to push an object in array field of Firebase database.
I've tried,
const friendObj = {
first_name: item.first_name,
last_name: item.last_name,
username: item.username,
phoneNumber: item.phoneNumber,
email: item.email,
}
db.collection('users')
.doc('documentId')
.set({ friends: friendObj })
I want to add an object in friends array of below table.
My firebase table looks like below
Can anyone please help me, I'm having difficulty while add value in DB.
Firebase doesn't actually have a data type as arrays, but you have few alternatives to store data as an array using an alternative (like sets) and there are methods specifically to work with those.
Check this link for an implementation
https://stackoverflow.com/a/40055996/11945277

Mongodb schema best storage of Achievement system [closed]

Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 10 months ago.
Improve this question
Im going to create a achievement system in Mongodb. But im not sure how i would format/store it in the database.
As of the users should have a progress (on each achievement they would have some progress value stored), im really confused what would be the best way to perform this, and without having an performence issue.
what should i do?, cause i dont know, what i had in mind, was maybe something like:
Should i store each achievement in an unique row in a Achievement collection, and an user array within that row, containing object with userid and achievement progress?
Would i then get an performance issue when its 1000+ achievements, that is beeing checked fairy often?
or should i do something else?
example schema for the option above:
{
name:{
type:String,
default:'Achievement name'
},
users:[
{
userid:{
type:String,
default:' users id here'
},
progress:{
type:Number,
default:0
}
}
]
}
Even though the question is specifically about the database design, I will give a solution for the tracking/awarding logic as well to establish more accurate context for the db design.
I would store the achievements progress separately from the already awarded achievements for cleaner tracking and discovery.
The whole logic is event based and has multiple layers of event handling. This gives you TONS of flexibility on how you track your data and also gives you a pretty good mechanism to track history. Basically, you can look at it as a form of logging.
Of course, your system design and contracts are highly dependent on the information you're gonna be tracking and its complexity. A simple progress field may not suffice for each case(you might want to track something more complex, not a simple number between X and Y). There is also the case of tracking data which updates quite frequently(as distance travelled in games, for example). You didn't give any context on the topic of your achievement system so we're gonna stick with a generic solution. It's just a couple of things that you should take a note about as it will affect the design.
Okay, so, let's start from the top and track the entire flow for a tracked piece of data and its eventual achievement progress. Let's say we're tracking consecutive days of user login and we're gonna award him with an achievement when he reaches [10].
Note that everything below is just a pseudo-code.
So, let's say today is [8th of July, 2017]. For now, our User entity looks like this:
User: {
id: 7;
trackingData: {
lastLogin: 7 of July, 2017 (should be full DateTime object, but using this for brevity),
consecutiveDays: 9
},
achievementProgress: [
{
achievementID: 10,
progress: 9
}
],
achievements: []
}
And our achievements collection contains the following entity:
Achievement: {
id: 10,
name: '10 Consecutive Days',
rewardValue: 10
}
The user tries to login(or visit the site). The application handler takes note of that and after handling the login logic fires an event of type ACTION:
ACTION_EVENT = {
type: ACTION,
name: USER_LOGIN,
payload: {
userID: 7,
date: 8 of July, 2017 (should be full DateTime object, but using this for brevity)
}
}
We have an ActionHandler which listens for events of type ACTION:
ActionHandler.handleEvent(actionEvent) {
subscribersMap = Map<eventName, handlers>;
subscribersMap[actionEvent.name].forEach(subscriber => subscriber.execute(actionEvent.payload));
}
subscribersMap gives us a collection of handlers that should respond to each specific action(this should resolve to USER_LOGIN for us). In our case we can have 1 or 2 that concern themselves with updating the user tracking information of lastLogin and consecutiveDays tracking properties in the user entity. The handlers in our case will update the tracking information and fire new events further down the line.
Once again, for brevity, we're gonna incorporate both into one:
updateLoginHandler: function(payload) {
user = db.getUser(payload.userID);
let eventType;
let eventValue;
if (date - user.trackingData.lastLogin > 1 day) {
user.trackingData = 1;
eventType = 'PROGRESS_RESET';
eventValue = 1;
}
else {
const newValue = user.trackingData.consecutiveDays + 1;
user.trackingData.consecutiveDays = newValue;
eventType = 'PROGRESS_INCREASE';
eventValue = newValue;
}
user.trackingData.lastLogin = payload.date;
/* DISPATCH NEW EVENT OF TYPE ACHIEVEMENT_PROGRESS */
AchievementProgressHandler.dispatch({
type: ACHIEVEMENT_PROGRESS
name: eventType,
payload: {
userID: payload.userID,
achievmentID: 10,
value: eventValue
}
});
}
Here, PROGRESS_RESET have the same contract as the PROGRESS_INCREASE but have a different semantic meaning and I would keep them separate for history/tracking purposes. If you wish, you can combine them into a single PROGRESS_UPDATE event.
Basically, we update the tracked fields that are dependent on the lastLogin date and fire a new ACHIEVEMENT_PROGRESS event which should be handled by a separate handler with the same pattern(AchievementProgressHandler). In our case:
ACHIEVEMENT_PROGRESS_EVENT = {
type: ACHIEVEMENT_PROGRESS,
name: PROGRESS_INCREASE
payload: {
userID: 7,
achievementID: 10,
value: 10
}
}
Then, in AchievementProgressHandler we follow the same pattern:
AchievementProgressHandler: function(event) {
achievementCheckers = Map<achievementID, achievementChecker>;
/* update user.achievementProgress code */
switch(event.name): {
case 'PROGRESS_INCREASE':
achievementCheckers[event.payload.achievementID].execute(event.payload);
break;
case 'PROGRESS_RESET':
...
}
}
achievementCheckers contains a checker function for each specific achievement that decides if the achievement has reached its desired value(a progress of 100%) and should be awarded. This enables us to handle all kinds of complex cases. If you only track a single X out of Y scenario, you can share the function between all achievements.
The handler basically does this:
achievementChecker: function(payload) {
achievementAwardHandler;
achievement = db.getAchievement(payload.achievementID);
if (payload.value >= achievement.rewardValue) {
achievementAwardHandler.dispatch({
type: ACHIEVEMENT_AWARD,
name: ACHIEVEMENT_AWARD,
payload: {
userID: payload.userID,
achievementID: achievementID,
awardedAt: [current date]
}
});
/* Here you can clear the entry from user.achievementProgress as you no longer need it. You can also move this inside the achievementAwardHandler. */
}
}
We once again dispatch an event and use an event handler - achievementAwardHandler. You can skip the event creation step and award the achievement to the user directly but we keep it consistent with the whole history logging flow.
An added benefit here is that you can use the handler to defer the achievement awarding to a specific later time thus effectively batching awarding for multiple users, which serve a couple of purposes including performance enhancement.
Basically, this pseudo code handles the flow from [a user action] to [achievement rewarding] with all intermediate steps included. It's not set in stone, you can modify it as you like but all in all, it gives you a clean separation of concerns, cleaner entities, it's performant, let's you add complex checks and handlers which are easy to reason about while in the same time provide a great history log of the user overall progress.
Regarding the DB schema entities, I would suggest the following:
User: {
id: any;
trackingData: {},
achievementProgress: {} || [],
achievements: []
}
Where:
trackingData is an object that contains everything you're willing
to track about the user. The beauty is that properties here are
independent from achievement data. You can track whatever and eventually use it for achievement purposes.
achievementProgress: a map of <key: achievementID, value: data> or
an array containing the current progress for each achievement.
achievements: an array of awarded achievements.
and Achievement:
Achievement: {
id: any,
name: any,
rewardValue: any (or any other field/fields. You have complete freedom to introduce any kind of tracking with the approach above),
users?: [
{
userID: any,
awardedAt: date
}
]
}
users is a collection of users who have been rewarded the given achievement. This is optional and is here only if you have the use for it and query for this data frequently.
What you might be looking for is a Badge style implementation. Just like Stack Overflow rewards it's users with badges for specific achievements.
Method 1: You can have flags in the user profile for each badge. Since you're doing it in NoSQL database, you just have to set a flag for each badge.
const badgeSchema = new mongoose.Schema({
badgeName: {
type: String,
required: true,
},
badgeDescription: {
type: String,
required: true,
}
});
const userSchema = new mongoose.Schema({
userName: {
type: String,
required: true,
},
badges: {
type: [Object],
required: true,
}
});
If your application architecture is event based, you can trigger awarding badges to users. And that operation is just inserting Badge object with progress in User badges array.
{
badgeId: ObjectId("602797c8242d59d42715ba2c"),
progress: 10
}
Update operation will be to find and update the badges array with progress percentage number
And while displaying user achievements on user interface, you can just loop over badges array to show the badges this user has achieved and their progress with it.
Method 2: Have a separate mongo collection for Badge and User Mapping. Whenever a user achieves a badge you insert a record in that collection. It will be one to one mapping of user _id and badge _id and progress value. But as the table will grow huge you will need to do indexing to efficiently query user and badge mapping.
You will have to do analysis on best approach according to your specific use case.
MongoDB is flexible enough to allow teams develop applications quickly, and involve their model with litter friction as the application needs it. In cases where you need a robust model from day one, theirs is a flexible methodology that can guide you through the process of modeling your data.
The methodology is composed of:
Workload: This stage is about gathering as much information as possible to understand your data. This will allow you formulate assumptions about, you data size the operations that will be performance against it (reads and writes), quantify operations and qualify operations.
You can get this by:
Scenarios
Prototype
Production Logs & Stats (if you are migrating).
Relationships: Identify the relationship between the different entities in your data, quantify those relationships and apply embedding or linking. In general you should prefer embedding by default, but remember that arrays should not grow without bound (6 Rules of Thumb for MongoDB Schema Design: Part 3).
Patterns: Apply schema design patterns. Take a look at Building with Patterns: A Summary, it presents a matrix that highlights the pattern that could be useful for a given use case.
Finally, the goal of this methodology is help you create a model, that can scale and perform well under stress.
If you design the achievement schema like this:
{
name: {
type: String,
default: "Achievement name",
},
userid: {
type: String,
default: " users id here",
},
progress: {
type: Number,
default: 0,
},
}
}
When an achievement is gained you just add another entry
for getting achievements Map-Reduce is a good candidate for running map reduce on the database. you can run them on a less regular basis, using them for offline computation of the data that you want.
based on documentation you can do like the following photo

Where to store a reference to other data models (in mongoDB) for best performance

In my project I have users and circles. Circles can have multiple users and a user can be in multiple circles. Lastly there are events. Each event can have multiple users in one circle. Later, events will get a lot of content, so there will be a lot of stuff to load (images, comments, etc.).
I was thinking that these would be a good data models:
User = {
_id: "uuid",
name: "string",
password: "string",
circles: [Circle._id],
}
Event = {
_id: "uuid",
name: "string",
location: "string",
circles:Circle._id,
participants: [User._id],
}
Circle = {
_id: "uuid",
name: "string"
}
Once the user logs in, he/she selects one of his circles, users and events in that circle will be displayed.
An API with these data models (I think) would mean to get the users and events from one circle, the database has to search through all users and events and check check if they are in that circle. With a lot of users and events, I think this might not be the most efficient way?
So I was thinking of putting the user and events into arrays of the circle like this:
User = {
_id: "uuid",
name: "string",
password: "string",
}
Event = {
_id: "uuid",
name: "string",
location: "string",
participants: [User._id],
}
Circle = {
_id: "uuid",
name: "string",
users:[User._id],
events:[Event._id]
}
Now, when the user selects the circle, the circle loads slower, because the users and events have to be loaded first. But I was thinking, that searching for users and events would now be faster. Is this the correct approach/thinking? Would it make sense to keep a reference to the specific circle ids in the User and Event data model?
If you want to use mongoDb to its full strength, I strongly recommend denormalising your data.
If you normalize your data, you might have to use $lookup to club multiple collections. Even if you save up on your harddisk, you will end up with relatively heavier computation.
Assuming that an application generally has 90% of hits as reads and 10% as writes, it makes sense to model your data in read friendly way. Hence highly denormalize your data untill its really necessary to create references to other collection. Optimizations can be later achieved by indexing and caching, but give below scema a thought.
User = {
_id: "uuid",
name: "string",
password: "string",
circles: ["circle1","circle2"],
events : ["event1","event2"]
}
Event = {
_id: "uuid",
name: "string",
location: "string"
}
Circle = {
_id: "uuid",
name: "string"
}
Try and know your queries beforehand, archiving most of your data in User collection. circles and events field in User collection can also be an array of objects [{},{}] if there are more properties to be stored.
I am certain that the more collections you club, the more complicated your queries will get and the computation will also be more.
I wont recommend storing userId's in circle or event collections as users may grow over time and you dont want to endup with a collection that has a document with one field storing thousands of array elements. On the contrary a user can be a part of 100's of circles and events, and if we store this data in User collection, it becomes quite easy to query and manage.
Long story short : Do not treat a nosql db as a relational db. It will never fit in. Model your database keeping your future queries in mind. Highly denormalize your data to make your read simpler i.e avoid references.

Meteor inserting into a collection schema with array elements

Hi I created a SimpleSchema for a Mongo collection which has a variable number of sub-documents called measurables. Unfortunately it's been a while since I've done this and I can't remember how to insert into this type of schema! Can someone help me out?
The schema is as follows:
const ExerciseTemplates = new Mongo.Collection('ExerciseTemplates');
const ExerciseTemplateSchema = new SimpleSchema({
name: {
type: String,
label: 'name',
},
description: {
type: String,
label: 'description',
},
createdAt: {
type: Date,
label: 'date',
},
measurables: {
type: Array,
minCount: 1,
},
'measurables.$': Object,
'measurables.$.name': String,
'measurables.$.unit': String,
});
ExerciseTemplates.attachSchema(ExerciseTemplateSchema);
The method is:
Meteor.methods({
addNewExerciseTemplate(name, description, measurables) {
ExerciseTemplates.insert({
name,
description,
createdAt: new Date(),
measurables,
});
},
});
The data sent by my form for measurables is an array of objects.
The SimpleSchema docs seem to be out of date. If I use the example they show with measurables: type: [Object] for an array of objects. I get an error that the the type can't be an array and I should set it to Array.
Any suggestions would be awesome!!
Many thanks in advance!
edit:
The measurable variable contains the following data:
[{name: weight, unit: kg}]
With the schema above I get no error at all, it is silent as if it was successful, but when I check the db via CLI I have no collections. Am I doing something really stupid? When I create a new meteor app, it creates a Mongo db for me I assume - I'm not forgetting to actually create a db or something dumb?
Turns out I was stupid. The schema I posted was correct and works exactly as intended. The problem was that I defined my schema and method in a file in my imports directory, outside both client and server directories. This methods file was imported into the file with the form that calls the method, and therefore available on the client, but not imported into the server.
I guess that the method was being called on the client as a stub so I saw the console.log firing, but the method was not being called on the server therefore not hitting the db.
Good lesson for me regarding the new recommended file structure. Always import server side code in server/main.js!!! :D
Thanks for your help, thought I was going to go mad!

React/Flux app data structure

i'm building an Gmail-like email browser client app prototype and i need a little help/advice structuring my React/Flux app. I decided to use pure Flux to get a better idea of how it works.
It's a simple email client with a list of letters, grouped by folders and tags and an ability to add letters to favorites.
So, i have a LettersStore containing an array of letters. The single letter data object looks something like this
{
id: 0,
new: true, //unread
checked: false,
starred: false,
folder: "inbox", //could be 'sent', 'spam', 'drafts'
sender: "Sender Name",
subject: "Re:",
snippet: "Hello there, how are you...",
body: "Hello there, how are you doing, Mike?",
date: "02.19.2016 16:30",
tags:["personal", "urgent"]
}
So what i'm trying to achieve is to let users navigate through folders (inbox, sent, drafts, spam) and filters (starred, tag, etc.)
In both folders and filters there has to be a way to select (check) some/all letters. The view state depends on how many letters are selected (the Select-all checkbox update, just like on Gmail). When the user selects a letter, the Flux action is being triggered and the state of the app updates.
The controller-view on top of the app does all the calls to the LettersStore public methods and passes the data down as props, but i'm not sure, what public methods the LettersStore should have. Currently it has:
emitChange()
addChangeListener()
removeChangeListener()
getAll() //returns array
areSomeLettersInFolderChecked(folderName) //returns bool
areAllLettersInFolderChecked(folderName) //returns bool
countNewLettersInAllFolders() //returns object
This works ok with folders, but when it comes to filters, it doesn't make sense anymore, since a starred letter is in some folder, and i feel like it's not the right thing to add specific methods like areSomeLettersInFilterChecked(filterType) etc.
Also, just like in Gmail, there has to be a way to select letter in the "Starred" filter, which belongs to the "Inbox" folder, then navigate to "Inbox" folder and keep that letter selected.
Maybe i should move the areSomeLettersInFolderChecked-like stuff to the component level?
I'm sure here has to be a proper way of doing it. Thanks in advance!
Rather than trying to encapsulate all the possible states and filters into your letter objects, keep it dumb. Normalize it and use supporting data structures to represent the other characteristics.
I'd strip it down to just the following properties:
{
id: 0,
sender: "Sender Name",
subject: "Re:",
snippet: "Hello there, how are you...",
body: "Hello there, how are you doing, Mike?",
date: "02.19.2016 16:30",
tags:["personal", "urgent"]
}
Your LetterStore can stay the same, or alternatively you could use an object or map to store letters against their id's for quick lookups later.
Now we need to represent the properties we removed from the message.
We can use individual sets to determine whether a message belongs to the new, checked and starred categories.
For instance, to star a message, just add it's id to the starred set.
var starred = new Set();
starred.add(message.id);
You can easily check whether a message is starred later on.
function isStarred(message) {
return starred.has(message.id);
}
The pattern would be the same for checked and unread.
To represent folders you probably want to use a combination of objects and sets.
var folders = {
inbox: new Set(),
sent: new Set(),
spam: new Set(),
drafts: new Set()
}
Simplifying your structures into these sets makes designing queries quite easy. Here are some examples of the methods you talked about implemented with sets.
function checkAll() {
messages.forEach(function(message) {
checked.add(message.id);
});
return checked;
}
function isChecked(message) {
return checked.has(message.id);
}
function inFolder(name, message) {
return folders[name].has(message.id);
}
// is message checked and in inbox
if(isChecked(message) && inFolder('inbox', message)) {
// do something
}
It becomes easy to construct complex queries, simply by checking whether messages belong to multiple sets.

Resources