I would like to know if a pure node.js web app can be developed, which means very simple deployment. From my understanding since node.js is good at i/o, a database in node.js should be good too. Does one exist? Especially one that lives in RAM and occasionally persists to disk.
First of I don't see the problem in installing redis or mongodb. It can be done without any effort at all.
That said there are a number of such databases like:
ministore: save at specified intervals.
alfred: Reads are fast because indexes into files are kept in memory.
nStore: Also a index of all documents and their exact location on the disk is stored in in memory for fast reads of any document.
jsonds: Jsonds is a 'data store' which is just a JSON object which is written to disk at a set frequency.
supermarket
chaos
node-dirty
node-tiny
nedb: Embedded pure JS database with MongoDB-compatible API.
Also most of these product are very young and should probably not be used in production yet.
You could also code something yourself I assume using node-sqlite3 to store data back to disc.
If you want a database in Node that exists only in ram you could simply use javascript objects and arrays to contain your data. If you need something more powerful with queries that ressemble SQL, then maybe pure javascript objects would not be the best idea. Also, with this idea you could make it persistant by flushing the data to disk using JSON.stringify at a set interval.
Try looking here: https://github.com/joyent/node/wiki/modules#database
Sorry for the short answer guys.
Related
I'm trying to build a very simple wiki-like system in Clojure and serving the http using Ring.
Instead of using a regular database i was thinking about using just an atom and serialise it to a file when it gets changed. Something like https://github.com/alandipert/enduro just with a delayed write.
Having the data in-mem in vectors and maps will surely make the service faster and the code simpler/more intuitive to write?
Will that work with a multithreaded Jetty/Ring server?
The content of the atom will surely fit in memory for now, but that might not hold true in the future. Any ideas to how i can structure the code to make it easier to switch to an alternative storage backend in the future?
This is the best guide for keeping data in memory and storing it to a single file: http://www.brandonbloom.name/blog/2013/06/26/slurp-and-spit/
Datomic would give you a few options.
You could use the in-memory db which would give you query power and thread safety. It would also be very easy to switch to a persistent datastore if/when the time comes. However, I'm not sure about serialization of the in-memory db.
Or you could use Datomic just for Datalog, which can be used for querying data structures. In that case, you could use an atom and then serialize as planned. Moving to a persistent datastore would be more work than the first case, but still not much. In either case, most of your code wouldn't need to change.
In my opinion, you'd be better of just starting with the free version of Datomic that uses the file system as a datastore. I don't think using an atom simplifies your code very much.
I second the recommendation for Datomic.
I've been using it on a "real" project for a few weeks now, and the more I use it, the more I realize that it would be a solid foundation for handling your data in any non-trivial project. Even if you never plan to use a "real" database in the future, just having a fact-based data model, powerful querying, and even full-text search built in is a huge win over just using an atom to store some huge map.
I checked and the free version does give you local storage as well as the in-memory database, so that would solve your storage needs perfectly (it uses an H2 database behind the scenes). And if you ever find yourself needing to scale to something bigger, you're already set.
I'm creating an app that uses a SQLite database to store it's data. The application is similar to a task app, where it mostly just grabs and displays data and updates it when needed.
My question is whether it is worth it to initialize separate objects (which I would assume would happen when the app is loading), or whether it's better to simply read/write values from/to the database directly.
I've seen both methods used, but my intuition says it would be much cheaper to interface with the database directly, as there would be less memory overhead, and the values will have to be read from/saved to the database eventually anyway, but perhaps running queries every time data is loaded or updated would be slower than interacting with objects.
As for constraints, I am using a SQLite database over an ORM as I would like the code and data store to be as cross-platform as possible, and I haven't found any ORMs that interface with Python, (Obj-)C, and java, which are the target languages I'm using. If anyone has any suggestions that work with each of the languages, please let me know.
I think caching will help, especially for objects that will be needed often (like users - permissions).
You can use a mode like this on the objects that you want stored in cache and later on easily set this cache on/off at any time (for java I recommend ehcache):
getObject (key){
if(object present in cache) return object from cache;
load object from database;
store it to cache;
return object
}
Mihai
Usually the cache will help, especially if you have a large number of concurrent accesses to your application.
Each time you have to go to the webserver you loose time (you need to go to another application, usually to a different server, transport the data back and forward). Accessing a local object is much faster.
The easiest solution is to try this with cache on/off for a class like Users and see if it makes a difference.
If the database is not too big, I'd suggest you to (create and) use the in-memory style of SQLite database; you can have a look at this just to begin with. Plus, after you are done with accessing/writing to the database in memory, you can always dump it to a file to be used later; loading and saving an SQLite database to memory and disk respectively is pretty straightforward using sqlite3_backup_init(), sqlite3_backup_step() and sqlite3_backup_finish() as is given here.
(I am not clear as to what functionality you require in an ORM.)
For my new project I'm looking forward to use JSON data as a text file rather then fetching data from database. My concept is to save a JSON file on the server whenever admin creates a new entry in the database.
As there is no issue of security, will this approach will make user access to data faster or shall I go with the usual database queries.
JSON is typically used as a way to format the data for the purpose of transporting it somewhere. Databases are typically used for storing data.
What you've described may be perfectly sensible, but you really need to say a little bit more about your project before the community can comment on your approach.
What's the pattern of access? Is it always read-only for the user, editable only by site administrator for example?
You shouldn't worry about performance early on. Worry more about ease of development, maintenance and reliability, you can always optimise afterwards.
You may want to look at http://www.mongodb.org/. MongoDB is a document-centric store that uses JSON as its storage format.
JSON in combination with Jquery is a great fast web page smooth updating option but ultimately it still will come down to the same database query.
Just make sure your query is efficient. Use a stored proc.
JSON is just the way the data is sent from the server (Web controller in MVC or code behind in standind c#) to the client (JQuery or JavaScript)
Ultimately the database will be queried the same way.
You should stick with the classic method (database), because you'll face many problems with concurrency and with having too many files to handle.
I think you should go with usual database query.
If you use JSON file you'll have to sync JSON files with the DB (That's mean an extra work is need) and face I/O problems (if your site super busy).
What's the best way to store large JSON files in a database? I know about CouchDB, but I'm pretty sure that won't support files of the size I'll be using.
I'm reluctant to just read them off of disk, because of the time required to read and then update them. The file is an array of ~30,000 elements, so I think storing each element separately in a traditional database would kill me when I try to select them all.
I have lots of documents in CouchDB that exceed 2megs and it handles them fine. Those limits are outdated.
The only caveat is that the default javascript view server has a pretty slow JSON parser so view generation can take a while with large documents. You can use my Python view server with a C based JSON library (jsonlib2, simplejson, yajl) or use the builtin erlang views which don't even hit JSON serialization and view generation will be plenty fast.
If you intend to access specific elements one (or several) at a time, there's no way around breaking the big JSON into traditional DB rows and columns.
If you'd like to access it in one shot, you can convert it to XML and store that in the DB (maybe even compressed - XMLs are highly compressible). Most DB engines support storing an XML object. You can then read it in one shot, and if needed, translate back to JSON, using forward-read approaches like SAX, or any other efficient XML-reading technology.
But as #therefromhere commented, you could always save it as one big string (I would again check if compressing it enhances anything).
You don't really have a variety of choices here, you can cache them in RAM using something like memcached or push them to disk reading and writing them with a databsae (RDBMS like PostgreSQL/MySQL or DOD like CouchDB). The only real alternative to these is a hybrid system of caching the most frequently accessed documents in memcached for reading which is how a lot of sites operate.
2+MB isn't a massive deal to a database and providing you have plenty of RAM they will do an intelligent enough job of caching and using your RAM effectively. Do you have a frequency pattern of when and how often these documents are accessed and how man users you have to serve?
A web app I'm working on requires frequent parsing of diverse web resources (HTML, XML, RSS, etc). Once downloaded, I need to cache these resources to minimize network load. The app requires a very straightforward cache policy: only re-download a cached resource when more than X minutes have passed since the access time.
Should I:
Store both the access time (e.g. 6/29/09 at 10:50 am) and the resource itself in the database.
Store the access time and a unique identifier in the database. The unique identifier is the filename of the resource, stored on the local disk.
Use another approach or third party software solution.
Essentially, this question can be re-written as, "Which is better for storing moderate amounts of data - a database or flat files?"
Thanks for your help! :)
NB: The app is running on a VPS, so size restrictions on the database/flat files do not apply.
To answer your question: "Which is better for storing moderate amounts of data - a database or flat files?"
The answer is (in my opinion) Flat Files. Flat files are easier to backup, and easier to remove.
However, you have extra information that isn't encapsulated in this question, mainly the fact that you will need to access this stored data to determine if a resource has gone stale.
Given this need, it makes more sense to store it in a database. Flat Files do not lend themselves well for random access, and search, compared to a relational DB.
Depends on the platform, IF you use .NET
The answer is 3, use Cache object, ideally suited for this in ASP.NET
You can set time and dependency expiration,
this doc explains the cache object
https://web.archive.org/web/1/http://articles.techrepublic%2ecom%2ecom/5100-10878_11-5034946.html
Neither.
Have a look at memcached to see if it works with your server/client platform. This is easier to set up and performs much better than filesystem/rdbms based caching, provided you can spare the RAM needed for the data being cached.
All of the proposed solutions are reasonable. However, for my particular needs, I went with flat files. Oddly enough, though, I did so for reasons not mentioned in some of the other answers. It doesn't really matter to me that flat files are easier to backup and remove, and both DB and flat-file solutions allow for easy checking of whether or not the cached data has gone stale. I went with flat files first and foremost because, on my mid-sized one-box VPS LAMP architecture, I think it will be faster than a third-party cache or DB-based solution.
Thanks to all for your thoughts! :)