They provide the example below in the Firebase documentation for data organization.
What happens when there are eventually 10,000 users signed up, but I decide to add another group? You'd have to write something to add the group with every existing user attached to that group as well as add every existing user to the new group entry? Is that right?
// An index to track Ada's memberships
{
"users": {
"alovelace": {
"name": "Ada Lovelace",
// Index Ada's groups in her profile
"groups": {
// the value here doesn't matter, just that the key exists
"techpioneers": true,
"womentechmakers": true
}
},
...
},
"groups": {
"techpioneers": {
"name": "Historical Tech Pioneers",
"members": {
"alovelace": true,
"ghopper": true,
"eclarke": true
}
},
...
I'm building a little game with React-Native and using Realtime Database in Firebase for my database. I am building a customize section, where a user can purchase to unlock more customizations. If they purchase an item, it will set effect/{purchasedId}/members/{uid}/({isPurchased:true}) which will then display in my CustomizedSection with getCustomizablesForUserByUserId.
If I want to add more effects down the line, will I just have to add a new effect and attach every members ID to it?
Thanks!
What happens when there are eventually 10,000 users signed up, but I decide to add another group? You'd have to write something to add the group with every existing user attached to that group as well as add every existing user to the new group entry? Is that right?
That is indeed correct. This process is known as backfilling, and is a normal (and sometimes quite large) concern when using schemaless NoSQL databases.
Related
I've got a verification bot for my server with linked roles. I've currently got it to remove all roles manually one by one, but of course this is inefficient and it only works for about 5/6 roles before stopping for a few seconds and continuing. What I'd like to try is some sort of discUser.removeRoles kind of thing, if that's possible.
Or is there a way to only try removing a role if the person has it? My code just does discuser.removeRole for every binded rank.
UPDATE
I got a notification about this question, so wanted to update it with a new solution for anyone else who finds this:
Create a table of your role ids. (e.g var giveThese = [])
guildMember.roles.add(giveThese,"Reason / Description"
For removing, you can replace roles.add with roles.remove
From what I understand from the question you're looping through every member of the guild and removing the role from each one of them.
To me, the most efficient way to do it is to take from the role the list of the members that have it (with Role.members) and then looping through that list.
You can do something like this:
let roleID = '1234...'
let role = guild.roles.fetch(roleID).then(role => {
role.members.forEach(member => member.roles.remove(roleID))
})
This is the most efficient way I can think for doing that since Discord currently has no way of "bulk removing" roles from users.
I was able to figure out a solution by looping over an external json file that holds all of the roles' data.
In the main file where you are trying to remove the roles, use a for in loop that loops over the json file containing all of the role names. Within the loop put the remove role method in there.
Here's an example of the .json file:
[
{
"role_name": "very slightly red",
"color": "#ffcccc",
"color_tier": "vs"
},
{
"role_name": "very slightly orange",
"color": "#ffedcc",
"color_tier": "vs"
},
{
"role_name": "very slightly yellow",
"color": "#ffffcc",
"color_tier": "vs"
}
]
And here's the code that removes the roles in bulk:
const vs_json = require("../vs_colors.json");
for (var key in vs_json) {
if (vs_json.hasOwnProperty(key)) {
console.log(
key +
" -> " +
vs_json[key].role_name
);
memberData.roles.remove(
getRole(
vs_json[key].role_name
)
);
}
}
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 10 months ago.
Improve this question
Im going to create a achievement system in Mongodb. But im not sure how i would format/store it in the database.
As of the users should have a progress (on each achievement they would have some progress value stored), im really confused what would be the best way to perform this, and without having an performence issue.
what should i do?, cause i dont know, what i had in mind, was maybe something like:
Should i store each achievement in an unique row in a Achievement collection, and an user array within that row, containing object with userid and achievement progress?
Would i then get an performance issue when its 1000+ achievements, that is beeing checked fairy often?
or should i do something else?
example schema for the option above:
{
name:{
type:String,
default:'Achievement name'
},
users:[
{
userid:{
type:String,
default:' users id here'
},
progress:{
type:Number,
default:0
}
}
]
}
Even though the question is specifically about the database design, I will give a solution for the tracking/awarding logic as well to establish more accurate context for the db design.
I would store the achievements progress separately from the already awarded achievements for cleaner tracking and discovery.
The whole logic is event based and has multiple layers of event handling. This gives you TONS of flexibility on how you track your data and also gives you a pretty good mechanism to track history. Basically, you can look at it as a form of logging.
Of course, your system design and contracts are highly dependent on the information you're gonna be tracking and its complexity. A simple progress field may not suffice for each case(you might want to track something more complex, not a simple number between X and Y). There is also the case of tracking data which updates quite frequently(as distance travelled in games, for example). You didn't give any context on the topic of your achievement system so we're gonna stick with a generic solution. It's just a couple of things that you should take a note about as it will affect the design.
Okay, so, let's start from the top and track the entire flow for a tracked piece of data and its eventual achievement progress. Let's say we're tracking consecutive days of user login and we're gonna award him with an achievement when he reaches [10].
Note that everything below is just a pseudo-code.
So, let's say today is [8th of July, 2017]. For now, our User entity looks like this:
User: {
id: 7;
trackingData: {
lastLogin: 7 of July, 2017 (should be full DateTime object, but using this for brevity),
consecutiveDays: 9
},
achievementProgress: [
{
achievementID: 10,
progress: 9
}
],
achievements: []
}
And our achievements collection contains the following entity:
Achievement: {
id: 10,
name: '10 Consecutive Days',
rewardValue: 10
}
The user tries to login(or visit the site). The application handler takes note of that and after handling the login logic fires an event of type ACTION:
ACTION_EVENT = {
type: ACTION,
name: USER_LOGIN,
payload: {
userID: 7,
date: 8 of July, 2017 (should be full DateTime object, but using this for brevity)
}
}
We have an ActionHandler which listens for events of type ACTION:
ActionHandler.handleEvent(actionEvent) {
subscribersMap = Map<eventName, handlers>;
subscribersMap[actionEvent.name].forEach(subscriber => subscriber.execute(actionEvent.payload));
}
subscribersMap gives us a collection of handlers that should respond to each specific action(this should resolve to USER_LOGIN for us). In our case we can have 1 or 2 that concern themselves with updating the user tracking information of lastLogin and consecutiveDays tracking properties in the user entity. The handlers in our case will update the tracking information and fire new events further down the line.
Once again, for brevity, we're gonna incorporate both into one:
updateLoginHandler: function(payload) {
user = db.getUser(payload.userID);
let eventType;
let eventValue;
if (date - user.trackingData.lastLogin > 1 day) {
user.trackingData = 1;
eventType = 'PROGRESS_RESET';
eventValue = 1;
}
else {
const newValue = user.trackingData.consecutiveDays + 1;
user.trackingData.consecutiveDays = newValue;
eventType = 'PROGRESS_INCREASE';
eventValue = newValue;
}
user.trackingData.lastLogin = payload.date;
/* DISPATCH NEW EVENT OF TYPE ACHIEVEMENT_PROGRESS */
AchievementProgressHandler.dispatch({
type: ACHIEVEMENT_PROGRESS
name: eventType,
payload: {
userID: payload.userID,
achievmentID: 10,
value: eventValue
}
});
}
Here, PROGRESS_RESET have the same contract as the PROGRESS_INCREASE but have a different semantic meaning and I would keep them separate for history/tracking purposes. If you wish, you can combine them into a single PROGRESS_UPDATE event.
Basically, we update the tracked fields that are dependent on the lastLogin date and fire a new ACHIEVEMENT_PROGRESS event which should be handled by a separate handler with the same pattern(AchievementProgressHandler). In our case:
ACHIEVEMENT_PROGRESS_EVENT = {
type: ACHIEVEMENT_PROGRESS,
name: PROGRESS_INCREASE
payload: {
userID: 7,
achievementID: 10,
value: 10
}
}
Then, in AchievementProgressHandler we follow the same pattern:
AchievementProgressHandler: function(event) {
achievementCheckers = Map<achievementID, achievementChecker>;
/* update user.achievementProgress code */
switch(event.name): {
case 'PROGRESS_INCREASE':
achievementCheckers[event.payload.achievementID].execute(event.payload);
break;
case 'PROGRESS_RESET':
...
}
}
achievementCheckers contains a checker function for each specific achievement that decides if the achievement has reached its desired value(a progress of 100%) and should be awarded. This enables us to handle all kinds of complex cases. If you only track a single X out of Y scenario, you can share the function between all achievements.
The handler basically does this:
achievementChecker: function(payload) {
achievementAwardHandler;
achievement = db.getAchievement(payload.achievementID);
if (payload.value >= achievement.rewardValue) {
achievementAwardHandler.dispatch({
type: ACHIEVEMENT_AWARD,
name: ACHIEVEMENT_AWARD,
payload: {
userID: payload.userID,
achievementID: achievementID,
awardedAt: [current date]
}
});
/* Here you can clear the entry from user.achievementProgress as you no longer need it. You can also move this inside the achievementAwardHandler. */
}
}
We once again dispatch an event and use an event handler - achievementAwardHandler. You can skip the event creation step and award the achievement to the user directly but we keep it consistent with the whole history logging flow.
An added benefit here is that you can use the handler to defer the achievement awarding to a specific later time thus effectively batching awarding for multiple users, which serve a couple of purposes including performance enhancement.
Basically, this pseudo code handles the flow from [a user action] to [achievement rewarding] with all intermediate steps included. It's not set in stone, you can modify it as you like but all in all, it gives you a clean separation of concerns, cleaner entities, it's performant, let's you add complex checks and handlers which are easy to reason about while in the same time provide a great history log of the user overall progress.
Regarding the DB schema entities, I would suggest the following:
User: {
id: any;
trackingData: {},
achievementProgress: {} || [],
achievements: []
}
Where:
trackingData is an object that contains everything you're willing
to track about the user. The beauty is that properties here are
independent from achievement data. You can track whatever and eventually use it for achievement purposes.
achievementProgress: a map of <key: achievementID, value: data> or
an array containing the current progress for each achievement.
achievements: an array of awarded achievements.
and Achievement:
Achievement: {
id: any,
name: any,
rewardValue: any (or any other field/fields. You have complete freedom to introduce any kind of tracking with the approach above),
users?: [
{
userID: any,
awardedAt: date
}
]
}
users is a collection of users who have been rewarded the given achievement. This is optional and is here only if you have the use for it and query for this data frequently.
What you might be looking for is a Badge style implementation. Just like Stack Overflow rewards it's users with badges for specific achievements.
Method 1: You can have flags in the user profile for each badge. Since you're doing it in NoSQL database, you just have to set a flag for each badge.
const badgeSchema = new mongoose.Schema({
badgeName: {
type: String,
required: true,
},
badgeDescription: {
type: String,
required: true,
}
});
const userSchema = new mongoose.Schema({
userName: {
type: String,
required: true,
},
badges: {
type: [Object],
required: true,
}
});
If your application architecture is event based, you can trigger awarding badges to users. And that operation is just inserting Badge object with progress in User badges array.
{
badgeId: ObjectId("602797c8242d59d42715ba2c"),
progress: 10
}
Update operation will be to find and update the badges array with progress percentage number
And while displaying user achievements on user interface, you can just loop over badges array to show the badges this user has achieved and their progress with it.
Method 2: Have a separate mongo collection for Badge and User Mapping. Whenever a user achieves a badge you insert a record in that collection. It will be one to one mapping of user _id and badge _id and progress value. But as the table will grow huge you will need to do indexing to efficiently query user and badge mapping.
You will have to do analysis on best approach according to your specific use case.
MongoDB is flexible enough to allow teams develop applications quickly, and involve their model with litter friction as the application needs it. In cases where you need a robust model from day one, theirs is a flexible methodology that can guide you through the process of modeling your data.
The methodology is composed of:
Workload: This stage is about gathering as much information as possible to understand your data. This will allow you formulate assumptions about, you data size the operations that will be performance against it (reads and writes), quantify operations and qualify operations.
You can get this by:
Scenarios
Prototype
Production Logs & Stats (if you are migrating).
Relationships: Identify the relationship between the different entities in your data, quantify those relationships and apply embedding or linking. In general you should prefer embedding by default, but remember that arrays should not grow without bound (6 Rules of Thumb for MongoDB Schema Design: Part 3).
Patterns: Apply schema design patterns. Take a look at Building with Patterns: A Summary, it presents a matrix that highlights the pattern that could be useful for a given use case.
Finally, the goal of this methodology is help you create a model, that can scale and perform well under stress.
If you design the achievement schema like this:
{
name: {
type: String,
default: "Achievement name",
},
userid: {
type: String,
default: " users id here",
},
progress: {
type: Number,
default: 0,
},
}
}
When an achievement is gained you just add another entry
for getting achievements Map-Reduce is a good candidate for running map reduce on the database. you can run them on a less regular basis, using them for offline computation of the data that you want.
based on documentation you can do like the following photo
I want to have the users in the database structured in a way that makes it easier for a human to read and manage. Using the users email address as the property name instead of the User ID:
Users:
"Users" : {
"emailaddress#domain.com":{
"id": "DK66qu2dfUHt4ASfy36sdfYHS9fh",
"name": "A Display Name",
"groups": {
"moderators": true,
"users": true
}
},
{...}
}
So that if I have a list of users in a group, they can be read as a list of emails and not a list of user IDs.
Groups Such as:
"Groups": {
"moderators":{
"name": "moderator",
"members": {
"emailaddress#domain.com": true,
"emailaddress2#domain.com": true
}
}
}
Groups Instead of:
"Groups": {
"moderators":{
"name": "moderator",
"members": {
"DK66qu2dfUHt4ASfy36sdfYHS9fh": true,
"K2fkHYQDFOge3Hw7SjRaGP3N2sdo": true
}
}
}
However, using rules to verify a property of the user (such as their group), would require me to maintain two list of users, one like the list above, and another essentially a table of key-value pairs of ID's and email addresses so I can get the users email address from their uid.
Pseudo-code rule: Users[UsersKeyVal[auth.uid]].groups.moderator == true
With firebase, what would be considered the most acceptable practice? What are the pros and cons of both?
Please do not store user data under their email address! This will be BIG TROUBLE later.
Your users node should follow the 'standard' Firebase design pattern
users
uid_0
name:
gender:
etc
uid_1
name:
gender:
etc
The bottom line is that in general, it's best to disassociate the dynamic data stored in the node from the key of the node.
Why?
Suppose you build a complex structure with all kinds of links and references to frank#mycoolcompany.com and then #mycoolcompany.com gets acquired by #mynotsocoolcompany.com. Well, you will then have to go in and rebuild every reference to franks's email in the entire database. ugh.
Then what if there are 100 or 1000 users #mycoolcompany.com! Ouch.
If you disassociate the data, like my per above suggested structure, you just change the email address within the node and everything else... just works!
PLEASE, read this answer on stack overflow - written by a Firebaser and addresses your question
Firebase data structure and url
In my opinion there is no problem with your data structure.
According to the Doc
This is a necessary redundancy for two-way relationships. It allows you to quickly and efficiently fetch your members memberships
Also using the generated UId from firebase or your custom Id (here your e-mail) doesn't change the way firebase works. You just have to make sure your e-mail are unique.
I'm creating a data structure for Firebase and AngularFire consisting of Users, Posts, and Comments. I was under the impression that the key/id for users would be the username, and that the key/id for comments and posts would be the auto-generated firebase key.
I've been working my through the angularfire documentation and am confused about the auto-generated keys (name()) that is added to an object when the $push() method is used.
Looking at some examples on the firebase website I see that an example of a Users object does not have the auto-generated key -- the key for an individual user is the username -- but at the same time a key is added whenever you add an object to the array via $push
My question is:
1) Should I always be using the firebase auto-generated keys? And if not, then how do I add a new user since $push() automatically creates the key, and $set() would reset all of my users?
2) What is the relationship between $id and name()?
Example Data
From https://www.firebase.com/docs/web/guide/saving-data.html
The docs show the following Users object:
{
"users": {
"alanisawesome": {
"date_of_birth": "June 23, 1912",
"full_name": "Alan Turing"
},
"gracehop": {
"date_of_birth": "December 9, 1906",
"full_name": "Grace Hopper"
}
}
}
How would I add more users without resetting my current users with $set() or adding the angularfire id with push()?
And then a Posts object with the generated id:
{
"posts": {
"-JRHTHaIs-jNPLXOQivY": {
"author": "gracehop",
"title": "Announcing COBOL, a New Programming Language"
},
"-JRHTHaKuITFIhnj02kE": {
"author": "alanisawesome",
"title": "The Turing Machine"
}
}
}
Thanks very much.
The short answer: you probably don't want to use push to store your users.
If you're getting your key from another source, like a uid from Simple Login, you will almost certainly want to use the uid to organize your users and their data in your firebase.
This is because, your users' ongoing sessions always provide you with that same uid which you can use to look up their user data and their stuff.
And you can safely use set in this case without resetting all of your users if you set based on that known user id.
But what I think you're getting at is, So in general, when do you set vs push?
A typical blog might look something like this in Firebase:
{
'users' : {
// uid from Simple Login, that you used with set()
'google-1234' : {
'displayName' : 'Jane Smith',
...
}
, ...
},
'posts' : {
// a blog post ID you pick and use for set()
'blog-post-id-i-use-in-the-url' : {
'title' : 'Blog Post Title',
'contents' : 'Four score and seven...'
}, ...
}
'postComments' {
'blog-post-id-i-use-in-the-url' : {
// Firebase generated ID done with push()
'_fe31ca1' : {
// uid from simple login (assuming comments require auth)
'commenterUserId': 'google-5678',
'commentBody': 'cats back for everyone!'
} ... other comments ...
}
}
}
In this example we use set when inserting new users and posts because we get a good unique ID from another source. These IDs are good because they allow us to easily recall the content later based on that ID.
We use push for comments, though. We don't have a good ID from another source, and order does matter, so we let Firebase generate a key for us. This works out OK because most of the time we're working with comments relative to an entry, so we can just grab them all as needed.
Following what mimmming said, I found a solution to this.
Have your add user function take an id as a parameter. this will be the authData.uid for the user you want to save.
Then append that id to the firebase link to make a new user using set.
Any other user you add using set will not wipe this since it is an entire new branch of your database under users. No firebase unique id too.
$scope.addUSer = function(id){
//pass the id in, andd append it to the end of your url link
var usersRef = new Firebase("https//<your fire base>.firebaseio.com/Users/"+id);
usersRef.set($scope.newUserData);
};
Ok Im starting out fresh with Firebase. I've read this: https://www.firebase.com/docs/data-structure.html and I've read this: https://www.firebase.com/blog/2013-04-12-denormalizing-is-normal.html
So I'm suitably confused as one seems to contradict the other. You can structure your data hierarchically, but if you want it to be scalable then don't. However that's not the actual problem.
I have the following structure (please correct me if this is wrong) for a blog engine:
"authors" : {
"-JHvwkE8jHuhevZYrj3O" : {
"userUid" : "simplelogin:7",
"email" : "myemail#domain.com"
}
},
"posts" : {
"-JHvwkJ3ZOZAnTenIQFy" : {
"state" : "draft",
"body" : "This is my first post",
"title" : "My first blog",
"authorId" : "-JHvwkE8jHuhevZYrj3O"
}
}
A list of authors and a list of posts. First of all I want to get the Author where the userUid equals my current user's uid. Then I want to get the posts where the authorId is the one provided to the query.
But I have no idea how to do this. Any help would be appreciated! I'm using AngularFire if that makes a difference.
Firebase is a NoSQL data store. It's a JSON hierarchy and does not have SQL queries in the traditional sense (these aren't really compatible with lightning-fast real-time ops; they tend to be slow and expensive). There are plans for some map reduce style functionality (merged views and tools to assist with this) but your primary weapon at present is proper data structure.
First of all, let's tackle the tree hierarchy vs denormalized data. Here's a few things you should denormalize:
lists you want to be able to iterate quickly (a list of user names without having to download every message that user ever wrote or all the other meta info about a user)
large data sets that you view portions of, such as a list of rooms/groups a user belongs to (you should be able to fetch the list of rooms for a given user without downloading all groups/rooms in the system, so put the index one place, the master room data somewhere else)
anything with more than 1,000 records (keep it lean for speed)
children under a path that contain 1..n (i.e. possibly infinite) records (example chat messages from the chat room meta data, that way you can fetch info about the chat room without grabbing all messages)
Here's a few things it may not make sense to denormalize:
data you always fetch en toto and never iterate (if you always use .child(...).on('value', ...) to fetch some record and you display everything in that record, never referring to the parent list, there's no reason to optimize for iterability)
lists shorter than a hundred or so records that you always as a whole (e.g. the list of groups a user belongs to might always be fetched with that user and would average 5-10 items; probably no reason to keep it split apart)
Fetching the author is as simple as just adding the id to the URL:
var userId = 123;
new Firebase('https://INSTANCE.firebaseio.com/users/'+userId);
To fetch a list of posts belonging to a certain user, either maintain an index of that users' posts:
/posts/$post_id/...
/my_posts/$user_id/$post_id/true
var fb = new Firebase('https://INSTANCE.firebaseio.com');
fb.child('/my_posts/'+userId).on('child_added', function(indexSnap) {
fb.child('posts/'+indexSnap.name()).once('value', function(dataSnap) {
console.log('fetched post', indexSnap.name(), dataSnap.val());
});
});
A tool like Firebase.util can assist with normalizing data that has been split for storage until Firebase's views and advanced querying utils are released:
/posts/$post_id/...
/my_posts/$user_id/$post_id/true
var fb = new Firebase('https://INSTANCE.firebaseio.com');
var ref = Firebase.util.intersection( fb.child('my_posts/'+userId), fb.child('posts') );
ref.on('child_added', function(snap) {
console.log('fetched post', snap.name(), snap.val();
});
Or simply store the posts by user id (depending on your use case for how that data is fetched later):
/posts/$user_id/$post_id/...
new Firebase('https://INSTANCE.firebaseio.com/posts/'+userId).on('child_added', function(snap) {
console.log('fetched post', snap.name(), snap.val());
});