I am reading that RDMS stores table data on disk in some form of B-tree, and also that table indexes are stored in the B-tree form.
I read that primary key index is created automatically for a primary key defined, but that it could also be dropped anytime. So, it implies that primary-key index is an additional structure next to the B-tree used for just storing table data.
Isn't that wasting of resources - why wouldn't all table table be kept through primary-key index?
If it isn't like that, which order is then used for the B-tree used to store table data?
Thanks for clarifying
The primary key index is an optimization for finding the place on disk where the row is held. As a structure, it contains simply the PK data, not the whole row.
On a database, performance is often gated by how many pages are read from disk vs. cache. Since the PK index is smaller than the whole table, it is more likely to be in cache, it causes fewer blocks to be read from disk, and less blocks of other tables are removed from cache. It therefore is a major performance optimization.
Further, while modifying the table data, rows are locked. If the primary key were being scanned from the table data on disk, locked rows would slow access for all the other queries. By separating the index as a separate structure, the index can be used even while the row being pointed to is locked.
So overall, the separate PK structure is a classic space-for-time optimization.
EDIT What is the order of the rows in the table? The following answer is for Oracle, but is applicable to many databases.
Short answer: rows are not ordered on disk which is why the PK index (and other indexes) are so important.
Long answer:
While the primary-key b-tree structure is necessarily sorted (the b-tree) the rows of the table are scattered across the table-space. To understand this we need to drill down the various data structures.
First, the database is structured into logical entities called a tablespaces. A tablespace is the space in one or more files on one or more disks. The files start empty. When the tablespace become full (technically when the data in it reaches a threshold) the tablespace can be automatically grown. It can also be grown manually by enlarging the file (adding an 'extent', or adding new files). Tablespaces can be clustered across multiple machines as well as disks.
Second: A tablespace is divided segments, each segment for the use of a single table or index.
Third: The segment is divided into blocks, each block has space for one or more rows. These blocks are not the same as disk or OS blocks; Oracle blocks are one or more OS blocks. (This is for transportability, and for managing media with different block sizes).
On insert, a database will select a space in a block from anywhere in the tablespace. The row can be inserted sequentially (especially bulk inserting into an empty table), but normally the database will also reuse space where rows have been deleted or moved due to some types of update. While the placement is theoretically kind-a predictable, in practice you should never rely on or expect the row to be placed in any specific block.
One interesting thing in Oracle is the ROWID. This is the reference stored in the index that allows the DB to look up the row:
An extended rowid has a four-piece format, OOOOOOFFFBBBBBBRRR:
The first 6 characters OOOOOO represent data object number, using 32bits
The next 3 characters FFF represent tablespace-relative datafile number, using 10bits.
The next 6 characters BBBBB represent block number, using 22bits.
The last 3 character RRR represent row number, using 16bit
For much more detail, see http://docs.oracle.com/cd/E11882_01/server.112/e25789/logical.htm#autoId0
One other thought: There is a concept in the DB world called partitions, where a dataset is divided across different tablespaces (frequently different disks or nodes in a cluster) depending on some expression logic. For example, on a table of customers, a vertical partition could be defined by the country of the person. That way you can ensure that the US customers are physically on one disk while the Australians are on another.
Related
How can I combine three tables in a database, but not have a large increase in data storage needs at any point in the process?
I have inherited a database with three tables. Two have the same columns, and one drops one of the columns:
Table1:
name
info
longkey
shortkey
Table2:
name
info
longkey
shortkey
Table3:
name
info
longkey
I want to create a single table with name and info columns and have no need for the remnants of other tables or the key fields. There are likely high numbers of duplicate entries between the three tables - an entry in table 1 will likely appear in table 2 and/or table 3.
The big problem, and the reason that this solution is not suitable is that it leads to a large increase in size and I have limited space available.
What can I do, either in SQL or perhaps through Python Scripting or other methods, that will not lead to a large increase in data storage at any time during the process?
What is heap table (as DB storage format)?
According to http://en.wikipedia.org/wiki/Heap_%28data_structure%29 heap is special data structure with properties that I can't directly map to DB performance benefits...
I found a lot of heap vs clustered discussions and understand that is clustered (there are a lot of images and definitions) but heap tables are not defined at all...
Quoting from this MSDN link:
A heap is a table without a clustered index.
One or more nonclustered indexes can be created on tables stored as a heap. Data is stored in the heap without specifying an order. Usually data is initially stored in the order in which is the rows are inserted into the table, but the Database Engine can move data around in the heap to store the rows efficiently; so the data order cannot be predicted
The most crucial part is that the data storage order on disk becomes non
deterministic in heap tables as you have NOT defined any clustered
index on it.
Once you define a clustered index on a table having a column let's say Id (of type int) you can say with guarantee that a row with id 1 will come before the row with id 2 in database page files. More useful information related to heap tables in SQL Server can be found here and here.
In short, a heap is a table in SQL Server without a clustered index. But..
Table scans on heaps follow the allocation order, as written in IAM pages; clustered tables follow the doubly-linked ordered list of pages (mostly to avoid some consistency problems, so if an X lock on the table or the READ UNCOMMITTED is in effect you can get the allocation order scan with clusters as well).
Heaps can be faster in an NCI seek + lookup scenario than clustered tables. The leaf level of NCIs on heaps contains the physical pointer to data, as opposed to clusters where there is a value of clustered index instead, which in most cases requires several more reads from non-leaf levels on CI to navigate the b-tree structure and find the data.
If there are variable-length columns in a heap, an update increasing the length of the columns could cause the whole row to move to some other page, leaving a forwarding pointer in place (that's to avoid updating all the NCIs with new pointers to the data). This could be very bad for the performance and the only cure is to rebuild the table.
As a rule of thumb, in SQL Server use clustered tables, unless you have a very good reason not to.
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 8 years ago.
Improve this question
Please suggest me answer for the mentioned query.
Q What is the reason
"When we create an index then CPU cost is less as compair to normal select query."?
When we create an index then CPU cost is less as compared to normal select query Is not always true.
If indexes are not used to evaluate predicates (In the where clause), then there is no use of the index. In that case, consider creating new indexes or rewriting the statement or query so that indexes can be used. For example, indexes can only be used to evaluate WHERE clauses when single columns appear on one side of a comparison predicate (equalities and inequalities), or in a BETWEEN predicate. The presence of indexes does slow down write operations such as UPDATE, INSERT, DELETE and CREATE VIEW. If an application does few reads but many writes to a table, an index on that table may hurt overall performance rather than help it.
As per oracle documentation, Indexes are optional structures associated with tables and clusters that allow SQL statements to execute more quickly against a table. Just as the index in this manual helps you locate information faster than if there were no index, an Oracle Database index provides a faster access path to table data. You can use indexes without rewriting any queries. Your results are the same, but you see them more quickly.
In general, create an index on a column in any of the following situations:
The column is queried frequently.
A referential constraint exists on the column.
A UNIQUE key constraint exists on the column.
Indexes are logically and physically independent of the data in the associated table. Being independent structures, they require storage space. You can create or drop an index without affecting the base tables, database applications, or other indexes. The database automatically maintains indexes when you insert, update, and delete rows of the associated table. If you drop an index, all applications continue to work. However, access to previously indexed data might be slower.
This beautiful article explains more on indexes. Do have a read.
Answering your question,
The logical reads per segment are all on index objects, if properly used. CPU performance is good when you read a table by Index since How oracle reads the blocks (and which blocks) with and without index is different.
Lets see how index work internally. Read more on this document
Consider a OLTP DB, with a table having B-Tree index. B-tree indexes are ordered lists of values divided into ranges with a key associated with a row or range of rows, thereby providing excellent retrieval performance for queries such as exact match and range searches.
This type of index contains 3 types of blocks - the root block, one or more branch blocks and one or more leaf blocks. The root block holds keys and pointers to the branch blocks which in turn hold pointers to the leaf blocks which contain the key (data) values that have been indexed and the rowids of the rows in the Oracle database table associated with each key value.
Branch blocks hold the minimum key prefix needed to be able to choose the required branch and a pointer to the child block containing the key. The number of keys and pointers is limited by the block size.
Index leaf blocks are double linked - each block contains pointers to the next and previous leaf blocks.
B-tree indexes are always balanced because the height of the index (the number of levels in the index) is the same throughout the index. In other words, the number of blocks that have to be read to find the rowid associated with any particular index value is not dependent on that value. For example if you had an index on the last_name column of the employee table in the sample Oracle database the number or blocks that would need to be read to find the rowid associated with "Ernst" would be the same as for "King".
The height of a b-tree index is the number of blocks required to go from the root block to a leaf block. For an index with one level of branch blocks the height is 3 (1 root block + 1 branch block + 1 leaf block).
Other "common sense" things :
rebuild indexes on a schedule. everone knows indexes need to be rebuilt
you must separate tables from indexes by tablespace, it is only common sense
if your query is not using your indexes, the optimizer did the wrong thing
select * from emp where deptno = 200;
Look how many comparisons with deptno 200 the CPU would have to do when querying the table without an index:
Jones 100
Miller 200
Smith 100
Blake 300
Sawyer 300
...
Now the same with an index:
100 -> Jones, Smith, ...
200 -> Miller, ...
300 -> Sawyer, ...
I've been researching best practices for creating clustered indexes and I'm just trying to totally understand these two suggestions that's listed with pretty much every BLOG or article on the matter
Columns that contain a large number of distinct values.
Queries that return large result sets.
These seem to be slightly contrary or I'm guessing maybe it just depends on how you're accessing the table.. Or my interpretation of what "large result sets" mean is wrong....
Unless you're doing range queries over the clustered column it seems like you typically won't be getting large result sets that matter. So in cases where SQL Server defaults the clustered indexes on the PK you're rarely going to fulfill the large result set suggestion but of course it does the large number of distinct values..
To give the question a little more context. This quetion stems from a vertical auditing table we have that has a column for TABLE.... Every single query that's written against this table has a
WHERE TABLE = 'TABLENAME'
But the TableName is highly non distinct... Each result set of tablenames is rather large which seems to fulfill that second conditon but it's definitely not largerly unique.... Which means all that other stuff happens with having to add the 4 byte Uniquifer (sp?) which makes the table a lot larger etc...
This situation has come up a few times for me when I've come upon DBs that have say all the contact or some accounts normalized into a single table and they are only separated by a TYPE parameter. Which is on every query....
In the case of the audit table the queries are typically not that exciting either they are just sorted by date modified, sometimes filtered by column, user that made the change etc...
My other thought with this auditing scenario was to just make the auditing table a HEAP so that inserting is fast so there's not contention between tables being audited and then to generate indexed views over the data ...
Index design is just as much art as it is science.
There are many things to consider, including:
How the table will be accessed most often: mostly inserts? any updates? more SELECTs than DML statements? Any audit table will likely have mostly inserts, no updates, rarely deletes unless there is a time-limit on the data, and some SELECTs.
For Clustered indexes, keep in mind that the data in each column of the clustered index will be copied into each non-clustered index (though not for UNIQUE indexes, I believe). This is helpful as those values are available to queries using the non-clustered index for covering, etc. But it also means that the physical space taken up by the non-clustered indexes will be that much larger.
Clustered indexes generally should either be declared with the UNIQUE keyword or be the Primary Key (though there are exceptions, of course). A non-unique clustered index will have a hidden 4-byte field called a uniqueifier that is required to make each row with a non-unique key value addressable, and is just wasted space given that the order of your rows within the non-unique groupings is not apparently obvious so trying to narrow down to a single row is still a range.
As is mentioned everywhere, the clustered index is the physical ordering of the data so you want to cater to what needs the best I/O. This relates also to the point directly above where non-unique clustered indexes have an order but if the data is truly non-unique (as opposed to unique data but missing the UNIQUE keyword when the index was created) then you miss out on a lot of the benefit of having the data physically ordered.
Regardless of any information or theory, TEST TEST TEST. There are many more factors involved that pertain to your specific situation.
So, you mentioned having a Date field as well as the TableName. If the combination of the Date and TableName is unique then those should be used as a composite key on a PK or UNIQUE CLUSTERED index. If they are not then find another field that creates the uniqueness, such as UserIDModified.
While most recommendations are to have the most unique field as the first one (due to statistics being only on the first field), this doesn't hold true for all situations. Given that all of your queries are by TableName, I would opt for putting that field first to make use of the physical ordering of the data. This way SQL Server can read more relevant data per read without having to seek to other locations on disk. You would likely also being ordering on the Date so I would put that field second. Putting TableName first will cause higher fragmentation across INSERTs than putting the Date first, but upon an index rebuild the data access will be faster as the data is already both grouped ( TableName ) and ordered ( Date ) as the queries expect. If you put Date first then the data is still ordered properly but the rows needed to satisfy the query are likely spread out across the datafile(s) which would require more I/O to get. AND, more data pages to satisfy the same query means more pages in the Buffer Pool, potentially pushing out other pages and reducing Page Life Expectancy (PLE). Also, you would then really need to inculde the Date field in all queries as any queries using only TableName (and possibly other filters but NOT using the Date field) will have to scan the clustered index or force you to create a nonclustered index with TableName being first.
I would be weary of the Heap plus Indexed View model. Yes, it might be optimized for the inserts but the system still needs to maintain the data in the indexed view across all DML statements against the heap. Again you would need to test, but I don't see that being materially better than a good choice of fields for a clustered index on the audit table.
If I have a table column with data and create an index on this column, will the index take same amount of disc space as the column itself?
I'm interested because I'm trying to understand if b-trees actually keep copies of column data in leaf nodes or they somehow point to it?
Sorry if this a "Will Java replace XML?" kind question.
UPDATE:
created a table without index with a single GUID column, added 1M rows - 26MB
same table with a primary key (clustered index) - 25MB (even less!), index size - 176KB
same table with a unique key (nonclustered index) - 26MB, index size - 27MB
So only nonclustered indexes take as much space as the data itself.
All measurements were done in SQL Server 2005
The B-Tree points to the row in the table, but the B-Tree itself still takes some space on disk.
Some database, have special table which embed the main index and the data. In Oracle, it's called IOT -- index-organized table.
Each row in a regular table can be identified by an internal ID (but it's database specific) which is used by the B-Tree to identify the row. In Oracle, it's called rowid and looks like AAAAECAABAAAAgiAAA :)
If I have a table column with data and
create an index on this column, will
the index take same amount of disc
space as the column itself?
In a basic B-Tree, you have the same number of node as the number of item in the column.
Consider 1,2,3,4:
1
/
2
\ 3
\ 4
The exact space can still be a bit different (the index is probably a bit bigger as it need to store links between nodes, it may not be balanced perfectly, etc.), and I guess database can use optimization to compress part of the index. But the order of magnitude between the index and the column data should be the same.
I'm almost sure it's quite a DB dependent, but generally – yeah, they take additional space. This happens because of two reasons:
This way you can utilize the fact
the data in BTREE leafs is sorted;
You gain lookup speed advantage as
you don't have to seek back and
forth to fetch neccessary stuff.
PS just checked our mysql server: for a 20GB table indexes take 10GB of space :)
Judging by this article, it will, in fact, take at least the same amount of space as the data in the column (in PostgreSQL, anyway).
The article also goes to suggest a strategy to reduce disk and memory usage.
A way to check for yourself would be to use e.g. the derby DB, create a table with a million rows and a single column, check it's size, create an index on the column and check it's size again. If you take the 10-15 minutes to do so, let us know the results. :)