The situation might be a bit complex, let's say I have a shopping cart related to items (items has some other relations with other entities) in a relational DB, it does not have any problem of query the shopping cart with all the other related entities by specifying them.
Now I am trying to move the shopping cart to Redis for high performance, if the shopping cart is created, with the items and the other relations will be saved to Redis.
await client.hSet(`shopping-cart:id:${cartId}`, {shoppingCartObj});
shoppingCartObj:
{
cartId: string,
items: [
item1: {
maker:{},
},
item2: {
maker:{},
},
item3: {
maker:{},
},
]
}
My question is the items and the items other related entities are still in DBMS like Postgres. They are still editable for users, like the maker. If the shopping cart is read from Redis always, they are out dated. Any strategy to maintain?
I can think of only maintain the shopping cart and the item relation in Redis with id links like cartId with itemId, if needed query again with the itemId.
Or load the items in the Redis as a copy, but will need to update if the user edited it.
Any other suggestions?
Related
Let's say i have a query to fetch a collection of movies:
useQuery(['movies'], getMovies)
Now, if I want to re-fetch only one movie instead of all I can write something like this:
useQuery(['movies', movieId], () => getMovie(movieId))
Problem is that I use a different query key and that it will duplicate the data. I'll have the movie in my cache twice.
So, what's the react-query way of updating a single item of a fetched collection? All components that use useQuery(['movies']) should be updated automatically when the single item was fetched.
I think there are two ways:
like you described with a new key. yes, the data will be "twice" in the cache, but oftentimes, the "list" data has a different structure than the "detail" data. movies-list might just have id and name, while the details have a description etc. as well. If you do that, make sure to give them both the same prefix ('movies'), so that you can utilize the fuzzy matching when invalidating: queryClient.invalidateQueries(['movies']) will then invalidate the list and all details, which is nice.
you can use the select option to build a detail query on:
const useMovies = (select) => useQuery(['movies'], getMovies, { select })
const useMovie = (id) => useMovies(movies => movies.find(movie => movie.id === id))
with that, you can do useMovie(5) and thus will only subscribe to the movie with id 5. This is really nice and also render optimized - if the movie with id 4 updates, the component that subscribes to movie with id 5 will not re-render.
the "drawback" is of course that for the data / network perspective, there is only one query - the list query, so everytime you do background refetches, the whole list will be queried from the backend. But it's still a nice approach to avoid data duplication in the cache, and it makes optimistic updates easier as well, because you only have to write to update one cache entry.
I use Firestore as my back-end. I have designed my data for the shopping cart like this:
cart--
cartid--
product--
id: 'dsdasd',
title: 'New Dress',
price: 10,
images: 'https://someurl/image.jpg
totalPrice: 50,
amount: 5
But I am not sure to use it. I'd like to hear some suggestions for this situation. I built it in React.js. How can I design my shopping cart data with Firestore? What should I use Firebase or Firestore?
I'd personally use Firestore, but it also depends on your needs and your app's requirements. You can read more about differences between the Realtime DB and Firestore here.
Talking about your data model, I'm not sure about your actual implementation (it's not clear from the question), but you may need two different collections: products and carts.
Every cart document will have a field (array/object) with all the products IDs in that cart. This way you can achieve data consistency, fetching product data from the actual product document every time you need it (for example displaying the cart).
The totalPricefield in the cart is probably unnecessary and leads to data inconsistency, because you need to update it when products in the cart change. It might be better to display the total price in the front end with a simple sum.
Intro
I have a FireStore database similar to a social media db, with 3 collections Users, Events, and EventUpdates. My goal is to create a feed with eventUpdates created by me and my friends. So i have to expand my database with friendship connections. But i struggle with 3 problems, and hopefully somebody here can push me in the right direction to solve these.
Problem/Question 1:
I added username and user image to the EventUpdate model so it's easier to query. I've heard denormalise is the way to go in a NoSQL database. But if a user updates his user image, i've to update all eventUpdates created by that user. Sounds like something you don't wanne do. But is there a better way to do this?
Problem/Question 2:
How can i create a data structure that is optimised for performing the following query: get eventUpdates from me and my friends ordered by date.
Problem/Question 3:
How to store likes? I can keep a counter in a eventUpdate. But this becomes a problem when i denormalise eventUpdates (see current solution underneath EDIT)..
Data structure example .
{
"users": {
"1": { "name": "Jack", "imageUrl": "http://lorempixel.nl" }
},
"events": {
"A": {
"name": "BeerFestival",
"date": "2018/09/05",
"creatorId": "1"
}
},
"eventUpdates": {
"1": {
"timestamp": "13243543",
"creatorId: "1",
"creatorName": "Jack",
"creatorImageUrl": "http://lorempixel.nl",
"eventId": "A",
"message": "Lorem ipsum"
}
}
}
EDIT
OK, after some trial and error i ended up with the following structure. This structure seems work, but my problem with this solution is that i need to make a lot of write calls to update a single eventUpdate because of all the copies in each feed (1000 followers means 1000 copies). And it looks like i need to do that a lot.
I would like for example to add a like button to each event update. This trigger an update on all EventUpdate copies. For me it looks like firebase is not suited for my project and i'm thinking of replacing it with a SQL DB, or can anyone here change my mind with a better solution?
{
"users": {
"user1": { "name": "Jack",
"imageUrl": "http://lorempixel.nl",
"followers": ["user1"]
}
},
"feeds": {
"user1": {
"eventUpdates": {
"1": {
"timestamp": "13243543",
"creatorId: "1",
"eventId": "A",
"message": "Lorem ipsum"
}
},
"following": {
"user1": {
"name": "Jack",
"imageUrl": "http://lorempixel.nl",
"followers": ["user1"]
}
}
},
"events": {
"A": {
"name": "BeerFestival",
"date": "2018/09/05",
"creatorId": "1"
}
}
}
I added username and user image to the EventUpdate model so it's easier to query. I've heard denormalise is the way to go in a NoSQL database.
That's right, denormalization and is a common practice when it comes to Firebase. If you are new to NoQSL databases, I recommend you see this video, Denormalization is normal with the Firebase Database for a better understanding. It is for Firebase realtime database but same rules apply to Cloud Firestore.
But if a user updates his user image, i've to update all eventUpdates created by that user. Sounds like something you don't wanne do. But is there a better way to do this?
Yes, that's also correct. You need to update all the places where that image exists. Because you have chosen google-cloud-firestore as a tag, I recommend you see my answer from this post because in case of many write operations, Firestore might be a little costly. Please also see Firestore pricing plans.
Regarding Firestore, instead of holding an entire object you can only hold a reference to a picture. In this case, there is nothing that you need to update. It's always a trade between these two techniques and unfortunately there is no way between. You either hold objects or only references to objects. For that, please see my answer from this post.
How can i create a data structure that is optimised for performing the following query: get eventUpdates from me and my friends ordered by date.
As I see, your schema is more a Firebase realtime database schema more than a Cloud Firestore. And to answer to your question, yes you can create. So talking about Firestore, you can create a collection named eventUpdates that can hold eventUpdate objects and to query it according to a timestamp, a query like this is needed:
FirebaseFirestore rootRef = FirebaseFirestore.getInstance();
CollectionReference eventUpdatesRef = rootRef.collection("eventUpdates");
Query query = eventUpdatesRef.orderBy("timestamp", Query.Direction.ASCENDING);
But please note that the timestamp field should be of type Date and not long. Please also take a look at my answer from this post, to see how you can add a date property in a Cloud Firestore database.
How to store likes? I can keep a counter in a eventUpdate. But this becomes a problem when i denormalise eventUpdates (see current solution underneath EDIT)
You can simply add likes but I recommend you see the last part of my answer from this post. So you might consider adding that count in a Firebase realtime database rather than in Cloud Firestore. Both databases work very well together.
This structure seems work, but my problem with this solution is that i need to make a lot of write calls to update a single eventUpdate because of all the copies in each feed (1000 followers means 1000 copies). And it looks like i need to do that a lot.
You might also take a look at my answer from this post.
For me it looks like firebase is not suited for my project and i'm thinking of replacing it with a SQL DB, or can anyone here change my mind with a better solution?
I don't think this way. There are many apps out there that have the exact mechanism as yours and are working very well.
If you want your feed items to be in sync with the real users data (new profile image when the user changes it for example) you can simply store the user ID in the eventUpdate document. This way you don't have to keep them in sync manually, and every time you have to display the item in the feed you could simply fetch user data, and easily query many eventUpdates on userId and created_at fields ( assuming you have them ).
To implement likes in your feed the solution depends on a bunch of things like traffic volume.
The simplest way is to update a likes field with a transaction, but Firestore has a maximum updates frequency on a single document of 1 second. Plus, a transaction can easily fail if more than 5 transactions are trying to update the same document.
To implement a more solid likes system take a look at this page from the official Firebase docs.
Firestore has a different approach to the NoSQL world. Once you know the data you will use (as You already do) there are some very important points about what architecture the data will have. And It depends a lot about how the data grows, what kind of queries you will need and how often you will use them. Some cases You can create a root collection that aggregates data and queries might be easier.
There is a great video from Firebase Channel that might help. Check it out!
How to Structure Your Data | Get to Know Cloud Firestore #5
https://www.youtube.com/watch?v=haMOUb3KVSo
[UPDATED] December 26th
Others videos that might help to model and query your data is these videos:
How to Connect Firebase Users to their Data - 3 Methods
https://www.youtube.com/watch?v=jm66TSlVtcc
How to NOT get a 30K Firebase Bill
https://www.youtube.com/watch?v=Lb-Pnytoi-8
Model Relational Data in Firestore NoSQL
https://www.youtube.com/watch?v=jm66TSlVtcc
I'm building a chat app with Firebase (and AngularJS) and I have a data structure that is similar to the one on this Firebase documentation page. This structure is good for not having to retrieve huge amounts of unneeded data but I don't seem to understand a very simple thing.
With my data looking like below, when a user connects to my app:
How do I retrieve their 10 most recently updated groups and keep this list updated as new messages are posted in groups?
// An index to track Ada's memberships
{
"users": {
"alovelace": {
"name": "Ada Lovelace",
// Index Ada's groups in her profile
"groups": {
// the value here doesn't matter, just that the key exists
"techpioneers": true,
"womentechmakers": true
}
},
...
},
"groups": {
"techpioneers": {
"name": "Historical Tech Pioneers",
"members": {
"alovelace": true,
"ghopper": true,
"eclarke": true
},
"lastUpdateTime": [SOME TIMESTAMP HERE]
},
...
}
}
More information if you care to read
As you can see, I've added "lastUpdateTime": [SOME TIMESTAMP HERE] to the code above because it's how I do it for my app. I can't figure out what should be the "refresh process" for a user group list.
If my user has 100 groups, should I retrieve a list of the group IDs and get the actual groups one by one to be able to only keep the 10 most recent (I'm pretty sure this is not the way to go)?
Also, whenever someone posts a message in a group, it will update the lastUpdateTime in Firebase but how do I keep the user group list synchronized to this?
I've tried some very ugly combinations of child events, orderBys as well as entire chains of functions executing whenever something fires but it doesn't work and seems extremely complicated, for nothing. The whole idea of flattening the data is to keep the queries/fetching to a minimum and I feel that what I have done so far is way too heavy.
To show the list of the 10 most recently updated groups:
ref.child("groups").orderByChild("lastUpdateTime").limitToLast(10)
If you use this approach, please flatten your data further, since the query will now end up retrieving the members of each group, which is not needed for displaying a list of groups.
If you want to a list of the groups the user is subscribed to by order of the last update, you have a few options:
store the last update timestamp for each user's subscriptions
load the user's groups and re-order them client-side
store the last update timestamp for each user's subscriptions
Store the timestamp the group was last updated for each user subscribed to the group:
"usersgroups": {
"alovelace": {
// the value is the timestamp the group was last updated
"techpioneers": 14952982198532978,
"womentechmakers": 14852982198532979
},
You'll note that I split the group memberships from the user profiles here, since you shouldn't nest such loosely related data.
Then you can get the list of the user's group in the correct order with:
ref.child("usersgroups/alovelace").orderByValue()
The main problem with this approach is that you'll need to update the timestamp of a group for all members for ever post. So writes become a lot more expensive.
load the user's groups and re-order them client-side
This may sound like it'll be slower, but it actually won't be too bad. Since you're only loading the groups the user is a member of, the number won't be too high. And Firebase pipelines the requests, so performance is a lot better than you may expect. See Speed up fetching posts for my social network app by using query instead of observing a single event repeatedly
When is it appropriate to filter a collection vs. having several collections in Backbone?
For example, consider a music library app. It would have a view for displaying genres and another view for displaying the selected genre's music.
Would you rather make one huge collection with all the music and then filter it or several smaller ones?
Having just one collection would allow you add features for filtering by other attributes as well, but suppose you have tons of music: how do you prevent loading it all in when the application starts if the user if only going to need 1 genre?
I think the simplest approach is having a common unique Collection that, intelligently, fetch an already filtered by genre data from the server:
// code simplified and no tested
var SongsCollection = Backbone.Collection.extend({
model: Song,
url: function() {
return '/songs/' + this.genre;
},
initialize: function( opts ){
this.genre = opts.genre;
}
});
var mySongsCollection = new SongsCollection({ genre: "rock" });
mySongsCollection.fetch();
You have to make this Collection to re-fetch data from the server any time the User changes the selected genre:
mySongsCollection.genre = "punk";
mySongsCollection.fetch();
It's mostly a design choice, but my vote would be to choose a scheme that loosely reflects the database storing the collections.
If you're likely to be storing data in an SQL database, you will more likely than not have separate tables for songs and genres. You would probably connect them either via a genre_id column in the song table, or (if songs can have more than one genre) in terms of a separate song_genres join table. Consequently, you would probably want separate collections representing genres and the songs within them. In this case, backbone-relational might be very useful tool for helping keep them straight.
If you're storing information in any kind of relational/key-value/document store, it might make sense to simply store the genre with the song directly and filter accordingly. In this case, you might end up storing your document keys/queries in such a way that you could access songs either directly (e.g., via songs) or through the genre (e.g., genre:genre_id/songs). If this is the route you go, it may be more convenient to simply create a single huge collection of songs and plan to set up corresponding filters in both the application and database environment.