This is just something I noticed and just very curious to understand why this was and if someone has a possible explanation for this behavior.
I created 2 sqlite3 files, both with the exact same data. One was version 3.7.5 and on CentOS. The other one was on version 3.7.13 and on OSX. The resulting file had sizes of 16K and 28K, and page sizes of 1024 and 4096.
Does this have anything to do with default block sizes on the OSes or something else file-system related? Or nothing at all and this is because of some additional information that SQLite now stores in its files?
Newer SQLite versions do not store anything additional in database files (as long as you do not use new features).
All tables and indexes use their own pages, so the database file size is affected by the page size.
Each page has a fixed amount of overhead, so increasing the page size typcially increases performance by a little bit.
Changing the page size allows you to trade off speed against space requirements.
The default page size is affected by the actual block size of the storage device, and by how the OS reports it.
Related
I'm wanting to use the rust implementation of LMDB but I can't seem to find whether it would always maintain a copy of whats in memory also in the disk and viceversa.
My reasoning would be that the DB will have some cache and any overflows would be written to disk but since I can't find it in the documentation I'm not sure.
The other case would be that LMDB maps its contents on disk to memory and that would work for small quantities but not for what I have in mind.
Does anyone have an idea on how LMDB works on this regard?
Does anyone have an idea on how LMDB works on this regard?
If you are worried about not being able to operate on a dataset that does not fit in memory, you are ok - LMDB does handle that. Your dataset is not limited to the size of ram.
LMDB is memory-mapped which is a technique that allows developers to read/write data on disk 'like' it is in memory - the OS does all the heavy lifting required
LMDB always stored BOTH in disk AND memory?
The entire dataset is on disk. Some parts of it are in memory. When the parts that are not in memory are needed - the OS fetches it from disk and gives it to the application by putting it in the process' memory
The other case would be that LMDB maps its contents on disk to memory and that would work for small quantities but not for what I have in mind.
Yes, that is it.
I read RocksDB support the usage you are looking while offering similar guarantees.
Small quantities depends on the available RAM. Major vendors including mongodb with wiredtiger backend, but not only, postgresql comes also to mind, that highly recommend to have as much memory as the working dataset.
In an opposite, you can find database systems such as rust full text search engine using an cold (offline?) object storage.
I have a project which uses BerkelyDB as a key value store for up to hundreds of millions of small records.
The way it's used is all the values are inserted into the database, and then they are iterated over using both sequential and random access, all from a single thread.
With BerkeleyDB, I can create in-memory databases that are "never intended to be preserved on disk". If the database is small enough to fit in the BerkeleyDB cache, it will never be written to disk. If it is bigger than the cache, then a temporary file will be created to hold the overflow. This option can speed things up significantly, as it prevents my application from writing gigabytes of dead data to disk when closing the database.
I have found that the BerkeleyDB write performance is too poor, even on an SSD, so I would like to switch to LMDB. However, based on the documentation, it doesn't seem like there is an option creating a non-persistent database.
What configuration/combination of options should I use to get the best performance out of LMDB if I don't care about persistence or concurrent access at all? i.e. to make it act like an "in-memory database" with temporary backing disk storage?
Just use MDB_NOSYNC and never call mdb_env_sync() yourself. You could also use MDB_WRITEMAP in addition. The OS will still eventually flush dirty pages to disk; you can play with /proc/sys/vm/dirty_ratio etc. to control that behavior.
From this post: https://lonesysadmin.net/2013/12/22/better-linux-disk-caching-performance-vm-dirty_ratio/
vm.dirty_ratio is the absolute maximum amount of system memory that can be filled with dirty pages before everything must get committed to disk. When the system gets to this point all new I/O blocks until dirty pages have been written to disk.
If the dirty ratio is too small, then you will see frequent synchronous disk writes.
I have 300000 documents stored in solr index. And used 4GB RAM for solr server. But It consumes more than 90% of physical memory. So I moved to my data to a new server which has 16 GB RAM. Again solr consumes more than 90% memory. I don't know how to resolve this issue. I used default MMapDirectory and solr version 4.2.0. Explain me if you have any solution or the reason for this.
MMapDirectory tries to use the OS memory (OS Cache) to the full as much as possible this is normal behaviour, it will try to load the entire index into memory if available. In fact, it is a good thing. Since these memory is available it will try to use it. If another application in the same machine demands more, OS will release it for it. This is one the reason why Solr/Lucene the queries are order of magnitude fast, as most of the call to server ends up memory (depending on the size memory) rather than disk.
JVM memory is a different thing, it can be controlled, only working query response objects and certain cache entries use JVM memory. So JVM size can be configured based on number request and cache entries.
what -Xmx value are you using when invoking the jvm? If you are not using an explicit value, the jvm will set one based on the machine features.
Once you give a max amount of heap to Solr, solr will potentially use all of it, if it needs to, that is how it works. If you to limit to say 2GB use -Xmx=2000m when you invoke the jvm. Not sure how large your docs are, but 300k docs would be considered a smallish index.
Per MSDN,
In SQL Server, the page size is 8 KB. This means SQL Server databases
have 128 pages per megabyte.
How do we change the page size, e.g. to 4 KB or 12 KB etc?
Also, is it due to an "innate hardware restriction" that page sizes are chosen to be 8 KB?
Or is 8 KB simply an arbitrary good-enough number chosen based on heuristics for optimization of majority use cases?
Short answer: you cannot change it.
For reason of efficiency of addressing (TLB) and efficiency of IO (see Reading From or Writing To Files Using a Scatter-Gather Scheme) the database page size must be some multiple of the OS page size, which is driven by the platform hardware architecture. x86 architecture has a 4k page size (see the sidebar on x86 Wiki article, or refer to the Intel Software Developer's Manuals), newer architectures offer large pages and SQL Server leverages them for memory allocations, see SQL Server and Large Pages Explained.
For reasons of database file portability the on-disk file format cannot be changed to accommodate modern CPU larger pages, it would break the capability to read the database on an older hardware. And experience shows that the 1MB page is quite difficult to allocate soon after OS start up, due to physical RAM fragmentation (a processor page must map to a contiguous hardware page). And such a large page would not play well with WAL (See ARIES).
SQL Server does not allow the page size to be changed - it is a fixed 8k in size with a fixed size header / data section.
1) The 8KB page is just a bit of legacy. I don't think there's any "good" reason for a fixed 8KB page size limitation but changing it would likely require a big investment from Microsoft and possibly a difficult upgrade path for customers.
2) Can't be done. The option to change it would be a very nice feature and is perhaps a long overdue enhancement but not something we're likely to see for a few years at least.
I am doing some analysis for a chunk of a desktop app we're working on.
One requirement is that it be able to do i/o of some legacy file formats, which in practice are running as large as 800Mb each. An import might reasonably be expected to be on the order of 5Gb in size.
Ideally, I'd just stick whatever files I want into a jar file, sign the thing, and send it off for re-import at a some later time.
But our app must support XP Pro (FAT32), which has a max file size limit of around 4Gb, from what I can tell.
Must I break my data up into multiple chunks? (And therefore take on the complexity of keeping track of what's going on?)
There's no other way of storing 5GB of data on FAT32 than splitting it in chunks.
Write a routine that will deal with archives of more than 4GB, i.e. split and merge. Encapsulate it in some util class or util file and call it from your save/load method.
The max file size on fat 32 is 4gb (actually 4Gb - 2bytes), so if you have to use it and your single file is over that you will have to split it.