I am using ionic framework for my mobile app and I would like to make some functions like when user requests data (json) from database (using REST API) and it will keep on local storage on the device. whenever the user come back, application will use the data from local storage instead.
I read there are many options to do this ($localstorage, sqlite) but I'm not sure which on is better in terms of performance and easy coding.
The data is text only and I would be around 2,000 rows per one query.
For performance, I would suggest going with Sqlite and also, your data will be securely stored in your app.
You can use localStorage for temporary data which is not very important as the localStorage data can be deleted also due to activities of the device's internet browser.
With regards to performance, I suggested sqlite as sqlite does not block the DOM or your view while performing query on it but getting data out of the storage takes few milliseconds more than localStorage, whereas, localStorage completely blocks the DOM when being queried but is a little faster (very minor) than sqlite in fetching the data.
Localstorage: its data will not be stored permanently. data stored in localstorage is unreliable. It is possible for the data to be wiped due to various reasons. Besides, it has the storage limit of 5 to 10 MB! Though It is better in terms of performance when compared to the below option.
pouchDB: If you have installed the SQLite Cordova Plugin then pouchDB will automatically use SQLite. It's a full database which overcomes both the above limitations by storing data on a specific location in the device's storage. You get unlimited storage. Apart from that, it can handle complex queries easily whereas localstorage is a simple key-value system. Since you are installing the Cordova Plugin plugin, it makes sure that you have full cross-platform support.
Best option will be to store data into sqlite db because the data you want to store is quite large. The reason is quite simple -CRUD operations require easy coding and the performance is great. Moreover when you plan your architecture you think of all possible expansions whereas local storage can only store limited amount of data.
Related
I'm trying to build a very simple wiki-like system in Clojure and serving the http using Ring.
Instead of using a regular database i was thinking about using just an atom and serialise it to a file when it gets changed. Something like https://github.com/alandipert/enduro just with a delayed write.
Having the data in-mem in vectors and maps will surely make the service faster and the code simpler/more intuitive to write?
Will that work with a multithreaded Jetty/Ring server?
The content of the atom will surely fit in memory for now, but that might not hold true in the future. Any ideas to how i can structure the code to make it easier to switch to an alternative storage backend in the future?
This is the best guide for keeping data in memory and storing it to a single file: http://www.brandonbloom.name/blog/2013/06/26/slurp-and-spit/
Datomic would give you a few options.
You could use the in-memory db which would give you query power and thread safety. It would also be very easy to switch to a persistent datastore if/when the time comes. However, I'm not sure about serialization of the in-memory db.
Or you could use Datomic just for Datalog, which can be used for querying data structures. In that case, you could use an atom and then serialize as planned. Moving to a persistent datastore would be more work than the first case, but still not much. In either case, most of your code wouldn't need to change.
In my opinion, you'd be better of just starting with the free version of Datomic that uses the file system as a datastore. I don't think using an atom simplifies your code very much.
I second the recommendation for Datomic.
I've been using it on a "real" project for a few weeks now, and the more I use it, the more I realize that it would be a solid foundation for handling your data in any non-trivial project. Even if you never plan to use a "real" database in the future, just having a fact-based data model, powerful querying, and even full-text search built in is a huge win over just using an atom to store some huge map.
I checked and the free version does give you local storage as well as the in-memory database, so that would solve your storage needs perfectly (it uses an H2 database behind the scenes). And if you ever find yourself needing to scale to something bigger, you're already set.
I'm pretty new with angularjs and I'm wriring my first application.
I'd like to know if there is a specific best practice about how often I should pull data from the server when I have to deal with big dataset. Is it better to pull one big JSON dataset and make a single call to the server, or it's advisable to fetch small bunch of data with multiple requests?
I try to explain. My application is now fetching from the server all the JSON data required by the application when the main page loads. It's a lot of stuff (about 3MB). Then it never fetches any other data, I can apply filters to the data and sorting it, all it's done client-side with no interaction with the server.
Now, is it worth to fetch few data at the beginning and then, based on the applied filters, re-fetch the data from the server?
Thanks!
It all depends on the specific requirements and usage patterns. If you are worried on quick load times, there are patterns similar to the ones used with jQuery.dataTables, which allow for super-quick loading of data, relying on server-side filtering.
If you have good cacheability (data is the same for all users) and no worries for long load times, go for the eager load (and use a filesystem-based cache with nginx serving the cached data).
In general, having local copies of the whole database is only useful when you can't do the work server-side, as RDBMS are much better at data analysis than any javascript implementation.
I'm creating an app that uses a SQLite database to store it's data. The application is similar to a task app, where it mostly just grabs and displays data and updates it when needed.
My question is whether it is worth it to initialize separate objects (which I would assume would happen when the app is loading), or whether it's better to simply read/write values from/to the database directly.
I've seen both methods used, but my intuition says it would be much cheaper to interface with the database directly, as there would be less memory overhead, and the values will have to be read from/saved to the database eventually anyway, but perhaps running queries every time data is loaded or updated would be slower than interacting with objects.
As for constraints, I am using a SQLite database over an ORM as I would like the code and data store to be as cross-platform as possible, and I haven't found any ORMs that interface with Python, (Obj-)C, and java, which are the target languages I'm using. If anyone has any suggestions that work with each of the languages, please let me know.
I think caching will help, especially for objects that will be needed often (like users - permissions).
You can use a mode like this on the objects that you want stored in cache and later on easily set this cache on/off at any time (for java I recommend ehcache):
getObject (key){
if(object present in cache) return object from cache;
load object from database;
store it to cache;
return object
}
Mihai
Usually the cache will help, especially if you have a large number of concurrent accesses to your application.
Each time you have to go to the webserver you loose time (you need to go to another application, usually to a different server, transport the data back and forward). Accessing a local object is much faster.
The easiest solution is to try this with cache on/off for a class like Users and see if it makes a difference.
If the database is not too big, I'd suggest you to (create and) use the in-memory style of SQLite database; you can have a look at this just to begin with. Plus, after you are done with accessing/writing to the database in memory, you can always dump it to a file to be used later; loading and saving an SQLite database to memory and disk respectively is pretty straightforward using sqlite3_backup_init(), sqlite3_backup_step() and sqlite3_backup_finish() as is given here.
(I am not clear as to what functionality you require in an ORM.)
For my new project I'm looking forward to use JSON data as a text file rather then fetching data from database. My concept is to save a JSON file on the server whenever admin creates a new entry in the database.
As there is no issue of security, will this approach will make user access to data faster or shall I go with the usual database queries.
JSON is typically used as a way to format the data for the purpose of transporting it somewhere. Databases are typically used for storing data.
What you've described may be perfectly sensible, but you really need to say a little bit more about your project before the community can comment on your approach.
What's the pattern of access? Is it always read-only for the user, editable only by site administrator for example?
You shouldn't worry about performance early on. Worry more about ease of development, maintenance and reliability, you can always optimise afterwards.
You may want to look at http://www.mongodb.org/. MongoDB is a document-centric store that uses JSON as its storage format.
JSON in combination with Jquery is a great fast web page smooth updating option but ultimately it still will come down to the same database query.
Just make sure your query is efficient. Use a stored proc.
JSON is just the way the data is sent from the server (Web controller in MVC or code behind in standind c#) to the client (JQuery or JavaScript)
Ultimately the database will be queried the same way.
You should stick with the classic method (database), because you'll face many problems with concurrency and with having too many files to handle.
I think you should go with usual database query.
If you use JSON file you'll have to sync JSON files with the DB (That's mean an extra work is need) and face I/O problems (if your site super busy).
I have been hired to help write an application that manages certain information for the end user. It is intended to manage a few megabytes of information, but also manage scanned images in full resolution. Should this project use a database, and why or why not?
Any question "Should I use a certain tool?" comes down to asking exactly what you want to do. You should ask yourself - "Do I want to write my own storage for this data?"
Most web based applications are written against a database because most databases support many "free" features - you can have multiple webservers. You can use standard tools to edit, verify and backup your data. You can have a robust storage solution with transactions.
The database won't help you much in dealing with the image data itself, but anything that manages a bunch of images is going to have meta-data about the images that you'll be dealing with. Depending on the meta-data and what you want to do with it, a database can be quite helpful indeed with that.
And just because the database doesn't help you much with the image data, that doesn't mean you can't store the images in the database. You would store them in a BLOB column of a SQL database.
If the amount of data is small, or installed on many client machines, you might not want the overhead of a database.
Is it intended to be installed on many users machines? Adding the overhead of ensuring you can run whatever database engine you choose on a client installed app is not optimal. Since the amount of data is small, I think XML would be adequate here. You could Base64 encode the images and store them as CDATA.
Will the application be run on a server? If you have concurrent users, then databases have concepts for handling these scenarios (transactions), and that can be helpful. And the scanned image data would be appropriate for a BLOB.
You shouldn't store images in the database, as is the general consensus here.
The file system is just much better at storing images than your database is.
You should use a database to store meta information about those images, such as a title, description, etc, and just store a URL or path to the images.
When it comes to storing images in a database I try to avoid it. In your case from what I can gather of your question there is a possibilty for a subsantial number of fairly large images, so I would probably strong oppose it.
If this is a web application I would use a database for quick searching and indexing of images using keywords and other parameters. Then have a column pointing to the location of the image in a filesystem if possible with some kind of folder structure to help further decrease the image load time.
If you need greater security due to the directory being available (network share) and the application is local then you should probably bite the bullet and store the images in the database.
My gut reaction is "why not?" A database is going to provide a framework for storing information, with all of the input/output/optimization functions provided in a documented format. You can go with a server-side solution, or a local database such as SQLite or the local version of SQL Server. Either way you have a robust, documented data management framework.
This post should give you most of the opinions you need about storing images in the database. Do you also mean 'should I use a database for the other information?' or are you just asking about the images?
A database is meant to manage large volumes of data, and are supposed to give you fast access to read and write that data in spite of the size. Put simply, they manage scale for data - scale that you don't want to deal with. If you have only a few users (hundreds?), you could just as easily manage the data on disk (say XML?) and keep the data in memory. The images should clearly not go in to the database so the question is how much data, or for how many users are you maintaining this database instance?
If you want to have a structured way to store and retrieve information, a database is most definitely the way to go. It makes your application flexible and more powerful, and lets you focus on the actual application rather than incidentals like trying to write your own storage system.
For individual applications, SQLite is great. It fits right in an app as a file; no need for a whole DRBMS juggernaut.
There are a lot of factors to this. But, being a database weenie, I would err on the side of having a database. It just makes life easier when things changes. and things will change.
Depending on the images, you might store them on the file system or actually blob them and put them in the database (Not supported in all DBMS's). If the files are very small, then I would blob them. If they are big, then I would keep them on he file system and manage them yourself.
There are so many free or cheap DBMS's out there that there really is no excuse not to use one. I'm a SQL Server guy, but f your application is that simple, then the free version of mysql should do the job. In fact, it has some pretty cool stuff in there.
Our CMS stores all of the check images we process. It uses a database for metadata and lets the file system handle the scanned images.
A simple database like SQLite sounds appropriate - it will let you store file metadata in a consistent, transactional way. Then store the path to each image in the database and let the file system do what it does best - manage files.
SQL Server 2008 has a new data type built for in-database files, but before that BLOB was the way to store files inside the database. On a small scale that would work too.