What is a memory database? Is sqlite a memory database?
On this mode, does it support persistence data to a local file?
An in-memory database supports all operations and database access syntax, but doesn't actually persist; it's just data structures in memory. This makes it fast, and great for developer experimentation and (relatively small amounts of) temporary data, but isn't suited to anything where you want the data to persist (it's persisting the data that really costs, yet that's the #1 reason for using a database) or where the overall dataset is larger than you can comfortably fit in your available physical memory.
SQLite databases are created coupled to a particular file, or to the pseudo-file “:memory:” which is used when you're wanting an in-memory database. You can't change the location of a database while it is open, and an in-memory database is disposed when you close its connection; the only way to persist it is to use queries to pull data out of it and write it to somewhere else (e.g., an on-disk database, or some kind of dump file).
SQLite supports memory-only databases - it's one of the options. It is useful when persistence is not important, but being able to execute SQL queries quickly against relational data is.
The detailed description of in-memory databases:
https://www.sqlite.org/inmemorydb.html
Related
Is SQL Server 2014's In-Memory OLTP (Hekaton) the same or similar concept with Redis?
I use Redis for in-memory storage (storage in RAM) and caching, while having a separate SQL Server database (like StackExchange does). Can Hekaton do the same thing?
They're similar by both being primarily in-memory, but that's about it.
Redis is an in-memory key-value database. It can persist data to disk if configure it, but it keeps the entire dataset in memory so you need enough RAM for that. The key-value architecture allows various different data types so you can store a value as a simple string or lists, sets, hashes, etc. Basically all the data structures you can use inside of a programming language are available in Redis natively.
SQL Server Hekaton (In-Memory OLTP) is a new engine designed to run relational tables in memory. All the data for these tables is kept in RAM but also stored to disk so they are fully durable.
Hekaton can take individual tables in a SQL Server database and run them in a different process using MVCC (instead of pages and locks) and other optimizations so operations are thousands of times faster than the traditional disk-based engine. There is a lot of research that went into this and the primary use-case would be to take a table that is under heavy load and switch it to run in-memory to increase performance and scalability.
Hekaton was not meant to run an entire database in memory (although you can do that if you really want to) but rather as a new engine designed to handle specific cases while keeping the interface the same. Everything to the end-user is identical to the rest of SQL Server: you can use SQL, stored procedures, triggers, indexes, atomic operations with ACID properties and you can work seamlessly with data in both regular and in-memory tables.
Because of the performance potential of Hekaton, you can use it to replace Redis if you need the speed and want to model your data within traditional relational tables. If you need the other key-value and data structure features of Redis, you're better off staying with that.
With SQL 2016 SP1 and newer, all tiers of SQL Server now have access to the same features and the only difference is pricing for support and capacity.
Firstly you need the enterprise edition (very expensive) of SQL Server to use Hekaton (In-Memory OLTP). Note you have to pay for sql server per CPU, adding more workload to SQL server may require you to have more CPU and therefore a lot more licence costs.
But unlike Redis, you can have a trigger or stored proc update your “in memory cache” as part of the database transaction. You may also find that Hekaton is fast enough that you don’t need a separate set of caches from your main tables.
So yes, Hekaton can do the same as Redis, but it is unlikely to be sensible to use it in that way unless its usage does not cost you much.
Hekaton comes into its own when it allows you to process a lot more data without having to invest in the programming cost of re-designing your system to make use of caching with Redis or otherwise.
In my project (similar to mediafire and rapidshare), clients can upload files to the server. I am using DB2 database and IBM WAS web server and JSP as server side scripting. I am creating my own encryption algorithm, as it is the main aim of the project.
I need suggestion whether files themselves should be stored in the database or if only the location of the files should be stored. Which approach is best?
There are Pros and Cons for storing BLOBs in the database.
Advantages
DBMS support for BLOBs is very good nowadays
JDBC driver support for BLOBs is very good
access to the "documents" can happen inside a transaction. No need to worry about manual cleanup or "housekeeping". If the row is deleted, so is the BLOB data
Don't have to worry about filesystem limits. Filesystems are typically not very good at storing million of files in a single directory. You will have to distribute your files across several directories.
Everything is backed up together. If you take a database backup you have everything, no need to worry about an additional filesystem backup (but see below)
Easily accessible through SQL (no FTP or other tools necessary). That access is already there and under control.
Same access controls as for the rest of the data. No need to set up OS user groups to limit access to the BLOB files.
Disadvantages
Not accessible from the OS directly (problem if you need to manipulate the files using commandline tools)
Cannot be served by e.g. a webserver directly (that could be performance problem)
Database backup (and restore) is more complicated (because of size). Incremental backups are usually more efficient in the filesystem
DBMS cache considerations
Not suited for high-write scenarios
You need to judge for yourself which advantage and which disadvantage is more important for you.
I don't share the wide-spread assumption that storing BLOBs in a database is always a bad idea. It depends - as with many other decisions.
It's general knowledge that storing files in the database -especially big ones- it's generally a bad idea. There are brilliant explanations in these questions:
Storing a file in a database as opposed to the file system?
Storing Images in DB - Yea or Nay?
And I'd like to highlight some points myself:
Storing files in your DBMS will make your data very big, and big databases are a maintaining hell (specially backups)
Portability becomes an issue, as every DBMS vendor makes its own implementation of BLOB files
There's a performance lost related to SELECT sentences to BLOB fields, compared to disk access
Well my Opinion would be to store the relevant information like path, name, description, etc... in the database and keep the file evtl. encrypted on the filesystem, it would be cheaper to scale your system adding a webserver than adding a database one as webspace is cheap comparing with databases, all you will need then is to add an IP column to your database or server name so you can address teh new webserver.
Is in-memory database a viable backup option for performing read operations in case of database failures? One can insert data into an in-memory database once in a while and in case the database server/web server goes down (rare occurence), one can still access the data present in the in-memory database outside of web server.
If you're going to hold your entire database in memory, you might just as well perform all operations there and hold your backup on disk.
No, since a power outage means your database is gone. Or if the DB process dies, and the OS deallocates all the memory it was using.
I'd recommend a second hard drive, external or internal, and dump the data to that hard drive.
Obviously it probably depends on your database usage. For instance it would be hard for me to imagine StackOverflow doing this.
On the other hand not every application is SO. If your database usage is limited you could take a cue from Mobile Applications which accept the fact that a server may not always be available. And treat your web application as though it were a Mobile Client. See Architecting Disconnected Mobile Applications Using a Service Oriented Architecture
I'm writing a Comet application that has to keep track of each open connection to the server. I want to write an entry to the database for each connection, and I will have to search the database for the proper connections every time the application receives new data (often), which is why I don't want to start off on the wrong foot by choosing slow database software. Any suggestions for a database that favors rapid, small pieces of data (rather than occasional large pieces of data)?
I suggest rather using a server platform that allows the creation of persistent servers, that keep all such info in the memory. Thus all database access will be limited to writing (if you want to actually save any information permanently), which usually is signifficantly less in typical Comet-apps (such as chats/games).
Databases are not made to keep such data. Accessing a database directly always means composing query strings, often sending them to a db server (sometimes even over the network), db lookup, serialization of the results, sending back, deserialization and traversing the fetched results. There is no way this can be even nearly as fast as just retrieving a value from memory.
If you really want to stick with PHP, then I suggest you have a look at memcached and similar caching servers.
greetz
back2dos
SQL Server 2008 has a FileStream data type that can be used for rapid, small pieces of data. McLaren Electronic Systems uses it to capture and analyze telemetry/sensor data from Formula One race cars.
Hypersonic: http://hsqldb.org/
MySQL (for webapps)
I'm working with Datamapper which allows you to specify either a path to a database file on your system or simply the string "memory" and then an "in-memory" database is used. However, I have no idea what this means.
Is an "in-memory" database purely memory-based or does it get serialized to the filesystem at some point?
What's the benefit of an in-memory database?
Is it simply for efficiency - by removing the cost of filesystem access?
The reason why SQLite has in-memory databases isn't really for performance. In practice, having a file isn't really any slower than skipping that, except if you run out of memory. It's really mostly for temporary caching or testing. For instance you could use an in-memory db for your unit tests, so that you don't walk on production data. Or you could use it as an alternative to memcache if you wanted SQL functionality. Basically, it's there so you don't have to persist data, if you don't need to.
From the SQLite documentation:
If the filename is ":memory:", then a
private, temporary in-memory database
is created for the connection. This
in-memory database will vanish when
the database connection is closed.