Even if there is no redundant storage in a student coarse table, unless we have a student enrolled in a course, we can't include the coarse details right. Then, why is this a problem related to redundant storage?
Related
Let's say theoretically, I have database with an absurd number of tables (100,000+). Would that lead to any sort of performance issues? Provided most queries (99%+) will only run on 2-3 tables at a time.
Therefore, my question is this:
What operations are O(n) on the number of tables in PostgreSQL?
Please note, no answers about how this is bad design, or how I need to plan out more about what I am designing. Just assume that for my situation, having a huge number of tables is the best design.
pg_dump and pg_restore and pg_upgrade are actually worse than that, being O(N^2). That used to be a huge problem, although in recent versions, the constant on that N^2 has been reduced to so low that for 100,000 table it is probably not enough to be your biggest problem. However, there are worse cases, like dumping tables can be O(M^2) (maybe M^3, I don't recall the exact details anymore) for each table, where M is the number of columns in the table. This only applies when the columns have check constraints or defaults or other additional info beyond a name and type. All of these problems are particularly nasty when you have no operational problems to warn you, but then suddenly discover you can't upgrade within a reasonable time frame.
Some physical backup methods, like barman using rsync, are also O(N^2) in the number of files, which is at least as great as the number of tables.
During normal operations, the stats collector can be a big bottleneck. Everytime someone requests updated stats on some table, it has to write out a file covering all tables in that database. Writing this out is O(N) for the tables in that database. (It used to be worse, writing out one file for the while instance, not just the database). This can be made even worse on some filesystems, which when renaming one file over the top of an existing one, implicitly fsyncs the file, so putting it on a RAM disc can at least ameliorate that.
The autovacuum workers loop over every table (roughly once per autovacuum_naptime) to decide if they need to be vacuumed, so a huge number of tables can slow this down. This can also be worse than O(N), because for each table there is some possibility it will request updated stats on it. Worse, it could block all concurrent autovacuum workers while doing so (this last part fixed in a backpatch for all supported versions).
Another problem you might into is that each database backend maintains a cache of metadata on each table (or other object) it has accessed during its lifetime. There is no mechanism for expiring this cache, so if each connection touches a huge number of tables it will start consuming a lot of memory, and one copy for each backend as it is not shared. If you have a connection pooler which hold connections open indefinitely, this can really add up as each connection lives long enough to touch many tables.
pg_dump with some options, probably -s. Some other options make it depend more on size of data.
I am looking for an optimal data structure for these pseudo-code operations:
add/update(key, node-content);
node=walk/iterate();
delete(node);
I would do a hash-table, but it is not possible to walk/iterate through it in efficient way (all buckets need be examined). Currently, I consider rbtree, but my doubts revolve around the fact that I need re-balance it at each add/delete operation keeping a global tree mutex presumably... Could anyone share some expertise what are the best options may be?
UPDATE I would like to expand on usage of the sought data structure as it would clarify the questions being asked so far.
The structure will have a few hundred nodes at most - so not that large. The most frequent operation will be walk(), wherein each and every node is read by turn, the order does not matter. walk() can happen thousand times a second. So far linked-list or even array would do.
The second most frequent operation is update(node-key, node-content). This is where efficient search needed. The operation is likely to occur many hundreds times a second. Thus, hash table is appealing.
Sometimes, a node will be added - when the update() doesn't find an existing node - or deleted. Add()/delete() happens rarely, say, once a day - so the cost of these operations is irrelevant.
UPDATE2 I think here is a very good recap on structures in question. Currently I gravitate towards skiplist
I am new to Cassandra. As I understand the maximum number of tables that can be stored per keyspace is Integer.Max_Value. However, what are the implications from the performance perspective (speed, storage, etc) of such a big number of tables? Is there any recommendation regarding that?
While there are legitimate use cases for having lots of tables in Cassandra, they are rare. Your use case might be one of them, but make sure that it is. Without knowning more about the problem you're trying to solve, it's obviously hard to give guidance. Many tables will require more resources, obviously. How much? That depends on the settings, and the usage.
For example, if you have a thousand tables and write to all of them at the same time there will be contention for RAM since there will be memtables for each of them, and there is a certain overhead for each memtable (how much depends on which version of Cassandra, your settings, etc.).
However, if you have a thousand tables but don't write to all of them at the same time, there will be less contention. There's still a per table overhead, but there will be more RAM to keep the active table's memtables around.
The same goes for disk IO. If you read and write to a lot of different tables at the same time the disk is going to do much more random IO.
Just having lots of tables isn't a big problem, even though there is a limit to how many you can have – you can have as many as you want provided you have enough RAM to keep the structures that keep track of them. Having lots of tables and reading and writing to them all at the same time will be a problem, though. It will require more resources than doing the same number of reads and writes to fewer tables.
In my opinion if you can split the data into multiple tables, even thousands, is beneficial.
Pros:
Suppose you want to scale in future to 10+ nodes and with a RF of 2 will result in having the data evenly distributed across nodes, thus not salable.
Another point is random IO which will be big if you will read from many tables at the same time but I don't see why there is a difference when having just one table. Also you will seek for another partition key, so no difference in IO.
When the compactation takes place it will have to do less work if there is only one table. The values from SSTables must be loaded into memory, merged and saved back.
Cons:
Having multiple tables will result in having multiple memtables. I think the difference added by this to the RAM is insignificant.
Also, check out the links, they helped me A LOT http://manuel.kiessling.net/2016/07/11/how-cassandras-inner-workings-relate-to-performance/
https://www.infoq.com/presentations/Apache-Cassandra-Anti-Patterns
Please fell free to edit my post, I am kinda new to Big Data
Using SO as a prime example, let's say a question or answer to a question is deleted but garnered a few up-votes before it was deleted. I imagine these points are still awarded to the author (if they aren't, let's suppose they are), then how does SO keep an accurate reputation total?
Are the questions/answers actually not deleted from the DB itself, and perhaps have a status field that is processed and decides whether a question or answer is visible?
Or, they are, in fact, deleted and the reputation relies on the system being continuously accurate as each vote is counted and doesn't necessarily have a history of it (like a question that recorded the vote)
SO uses a combination of soft and hard deletes, to the best of my knowledge. I can say for sure that I've lost reputation that was gained on questions deleted by either the poster or the moderator. That is not the point of your question, however, so...
If you want to be able to deduce an accurate total, especially if you want to be able to account for that total (the way SO lets you do by looking at your points history) then you need to keep transactional information, not a running total.
If you want to have referential integrity for the transactional log of points then you will need to use a soft-delete mechanism to hide questions that are "deleted".
If you don't keep the transactional log and you don't have soft delete-able questions to back up your transactional points log, then you won't be able to either recalculate or justify point totals. You'll also have a much harder time displaying a graph of points awarded over time and accumulated reputation over time. You could do these graphs by keeping a daily point snapshot, but that would be much more onerous and costly in terms of storage than just tracking up and down votes.
I'm working with a sqlite3 database that could conceivably become quite large. Storage space is a concern, so I was considering setting the auto_vacuum pragma to on so that space occupied by deleted rows would be actually freed instead of just marked as available for re-use.
In my scenario, the database could grow by several hundred MB per month, while rows older than ~6 months would decay in a granular fashion. This is achieved by a job queue that randomly tacks on the task of removing the nn oldest records in addition to the current task, where nn is determined by how many high priority tasks there are in the queue.
I'm hoping that this avoids having to write maintenance jobs that cause protracted RW starvation (in the order of minutes, to delete rows and then run VACUUM) when the same could be achieved a few MS at a time. This might mean that 'old' rows remain in the DB a few days longer than they would otherwise, but that is an acceptable trade off.
My question is, in your experience (and perhaps opinion), would turning on auto_vacuum be an unacceptable compromise given my description? If so, for what reasons? I have not used sqlite3 extensively, much less the various pragmas it presents for tweaking so I am hoping to solicit the experience I'm lacking prior to making a judgement call that I might regret a few months from now :)
I'm using the C interface, if it makes any difference.
A liferea developer explains why:
The problem with it is that it also takes very long. With a 50MB DB file I experienced a runtime of over 1 minute. This is why this can be only a tool for experienced users that know how to do it manually knowing what to expect. For executing such a long term operation automatically on runtime would surely be unacceptable to the unsuspecting user. Also there is no good way how to decide when to do a VACUUM to save disk space and improve performance.