I am working on a portal where a large chunk of data is the same all the time - let's say it is like a very advanced dictionary with a base of questions to it. Those do not change. I thought that since it is a significant number of docs (total around 20mB) it is not worth downloading it every single time (and for many actions all of it has to be downloaded), but instead, to store it hashed on front-end and access as needed. This would significantly limit server's computation. However, I see that localStorage is limited to 5mB.
My questions is are there any other good solutions/practices to apply in this situation?
Database is mongodb, and it is a MERN stack app.
On frontend to use significant payload you can try to use IndexedDB, database located in browser itself https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API/Using_IndexedDB
Related
I'm building a web app where in certain cases users may want to deep link to a state consisting of a list of items to have some action carried out upon.
We are considering whether to serialize the state on the URL or implement a solution that will persist the state on the backend and provide an ID to look it up.
So the url-scheme would either be something like:
/batch/item-1&item-2&item-3
Or:
/batch/4jk5kjdl3k
Where 4jk5kjdl3k represents an ID which allows the backend to lookup the list of item IDs.
We do not care about SEO in this instance as it is an editing interface that will not generally be crawlable or meaningful to search engines. From my research IE has a limit of 2083 characters and SEO sitemaps 2048 characters. We should be able to keep URL lengths below either limit. In the typical use case scenario we would sit somewhere around 200-500 characters, but there could be extreme cases getting close to 2000.
If we serialize the state on the URL directly, we can save a fair amount of work on the backend, but I'm worried that I'm overlooking some disadvantages to these potentially very long URLs.
I found this question on the topic, but it appears to be more concerned with the matter of exceeding 2083 characters:
What is the maximum length of a URL in different browsers?
So far the only disadvantages I see are:
Perhaps the IDs we store could change design and become longer in the future and bump us over the limit (quite unlikely)
These very long URLs can seem a bit unwieldy for users to deal with when copy/pasting and they could potentially end up cutting them off by accident
For context, the app is built in React, but I don't think that should be a determining factor.
I am using ionic framework for my mobile app and I would like to make some functions like when user requests data (json) from database (using REST API) and it will keep on local storage on the device. whenever the user come back, application will use the data from local storage instead.
I read there are many options to do this ($localstorage, sqlite) but I'm not sure which on is better in terms of performance and easy coding.
The data is text only and I would be around 2,000 rows per one query.
For performance, I would suggest going with Sqlite and also, your data will be securely stored in your app.
You can use localStorage for temporary data which is not very important as the localStorage data can be deleted also due to activities of the device's internet browser.
With regards to performance, I suggested sqlite as sqlite does not block the DOM or your view while performing query on it but getting data out of the storage takes few milliseconds more than localStorage, whereas, localStorage completely blocks the DOM when being queried but is a little faster (very minor) than sqlite in fetching the data.
Localstorage: its data will not be stored permanently. data stored in localstorage is unreliable. It is possible for the data to be wiped due to various reasons. Besides, it has the storage limit of 5 to 10 MB! Though It is better in terms of performance when compared to the below option.
pouchDB: If you have installed the SQLite Cordova Plugin then pouchDB will automatically use SQLite. It's a full database which overcomes both the above limitations by storing data on a specific location in the device's storage. You get unlimited storage. Apart from that, it can handle complex queries easily whereas localstorage is a simple key-value system. Since you are installing the Cordova Plugin plugin, it makes sure that you have full cross-platform support.
Best option will be to store data into sqlite db because the data you want to store is quite large. The reason is quite simple -CRUD operations require easy coding and the performance is great. Moreover when you plan your architecture you think of all possible expansions whereas local storage can only store limited amount of data.
I'm part of a team developing an ecommerce site through the ordercloud.io API. I'm trying to figure out how much information I can store in a JSON object's xp key. The project I'm working on requires some pretty custom configuration and data storage, and I want to make sure the process I come up with will scale well.
Thanks!
The XP value in OrderCloud is limited by a maximum string size of 8000 characters. However, you really shouldn't try to reach this value because it may drain on the performance of your API requests (that's a lot of data to send over the wire, especially if you are getting a list of objects).
Make sure you review existing features to see if there is a way to accomplish what you are trying to store/behavior you are trying to create. Or consider storing in a separate location (your own db or an integration that helps fulfill the feature).
I have a new project about mobile app using react native tech.
I am thinking about using redux to manage the whole data from remote server api. our product have more business data need to display in mobile app.
So, My question is: redux state store our business data and it will take more memory on mobile device, like a ListView component. how can i solve this problem if i want to reduce the memory usage?
I am choosing, based on your background description of what you're trying to do, to address the underlying concern about the size of your redux store generally and the approach of storing everything on the client in my answer, and will not address specifically how to actually reduce the size of your data store here (the only answer to that is simply "don't store so much").
This is just a total swag and ignores things like compression, data duplication, the difference between storing something in AsyncStorage vs simply being in memory, etc.
That having been said, if you need some sort of gut check on whether memory/storage will be an issue, take a representative chunk of record data served by your API, serialize it as a JSON string, and figure out how big it is.
For example, this example twitter response is roughly 8.5 KB with whitespace removed. Let's say 10KB for each individual record for simplicity.
Now, how many records do you plan on bringing down? 10? 100? 1000? Let's say 1000 records of this type. That's 10,000KB or roughly 10MB.
With the constructs here, 10 MB is (Edit: depending on the specific constraint you're concerned about, may or may not be) a trivial amount of memory/storage to use in your application.
You need to do this similar process to your particular use case, and see if the amount of data you wish to store will be a problem for the devices you have to support.
A more relevant thing to consider is the performance impact of churning through large quantities of data on a single thread to do things like data manipulation, joining/merging, etc if that will be a need.
Redux is a tiny library that doesn't actually do that much for you by itself. This consideration is a general one, and is totally unique to your own application and cannot be concretely answered.
I am building a mobile app using AngularJS and PhoneGap. The app allows the user to access a large amount of data-items. These data-items come with the app in form of a number of .json files.
One use-case is that a user can favorite any of those data-items.
Currently, I store the (ids of) the items that have been favorited in localStorage. It works and it's great and very simple.
But now I would like create an online-backend for the app. By this I mean that the (ids of) the items that have been favorited should also be stored on a server somewhere in some form of database.
Now my question is:
How do I best do this?
How do I keep the localStorage data and online-backend data in synch?
In particular, the user might not have an internet connection at the time were he favorites a data-item. Additionally, if the user favorites x data-items in a row, I would need to make x update calls to the server db, which clearly isn't great.
So, how do people do it?
Does Angular have anything build-in for this?
Is there any plugin?
Any other framework?
This very much seems like a common problem that must have a well-known solution?
I think you've almost got the entire solution. All you need to do is periodically (on app start load the data from the service if available, otherwise use current local storage, then maybe with a timer and on app close update the data if connected) send the JSON out to a service (I generally prefer PHP, but Python, Java, Ruby, Perl, whatever floats your boat) that puts it in a database. If you're concerned with merging synchronization changes you'll need to use timestamps in the data in local storage and in the database to make the right call on what should be inserted vs what should be updated.
I don't think there's a one size fits all solution to the problem, though I imagine someone may have crafted a library that handles the different potential scenarios the configuration may be as complicated as just writing the logic yourself.