I have a very large table (700GB) in Aurora Database Cluster. When I run ALTER command to add new fields, it shows error 'Table is Full'. This is due to filling up the temporary space on the instance, which is sized according to the host instance size we select.
In my case free local storage is around 300 GB for (db r5 large). What is the alternate way of adding field to the table. I am thinking of Adding a new table with updated field and then copying the data from original table.
Related
We used to store logs in a postgres table and with time that table has grown in size more than 600GB. We tried to implement the purging strategy on the table to delete table's data. Purging query run fine and returns with number of rows deleted, however the table size is still not reducing.
We tried to delete the data using delete queries and more than 348 million rows were deleted. The size still didn't reduce. Auto-vacuum process was also monitored which ran fine but size still didn't reduce.
We are using RDS postgres and the current storage capacity of RDS is around 1200GB, free space is around 100GB. I want to know if I am missing anything.
I have a Oracle 11g database with block size = 8192. So, if I'm correct maximum datafile size will be 32GB.
I have a huge table containing around 10 million records. Data in this table will be purged often. For purging we chose CTAS as a better option as we are going to delete greater portion of the data.
As we'll be dropping the old table after CTAS, the old tables are not releasing the space for new tables. I understand that a tablespace has AUTOEXTEND option but no AUTOSHRINK. But the space occupied by old tables should be available for new tables, which is not happening in this case.
I'm getting an Exception saying
ORA-01652: unable to extend temp segment by 8192 in tablespace
FYI the only operation happening all the time is CTAS + Dropping the old table. Nothing else. First time this is working fine, but when the same operation is done the second time, exception arises.
I tried adding an additional datafile to the tablespace, but after few more purge operations on the table, this is also getting full to 32GB and the issue continues.
I had an uniqueidentifier field in SQL Server (SQL Azure to be precise) that I wanted to get rid of. Initially, when I ran the code as mentioned in SQL Azure table size to determine the size of the table it was about 0.19 MB.
As a first step I set all values in the uniqueidentifier field to null. There are no constraints/indexes that use the column. Now, when I ran the code to determine the table sizes the table had increased in size to about 0.23 MB. There are no records being added to a table (its in a staging environment).
I proceeded to delete the column and it still hovered at the same range.
Why does the database size show an increase when I delete a column. Any suggestions?
Setting an uniqueidentifier column to NULL value does not change the record size in any way, since is a fixed size type (16 bytes). Dropping a fixed size column column does not change the record size, unless is the last column in the physical layout and the space can be reused later. ALTER TABLE ... DROP COLUMN is only a logical operation, it simply marks the columns as dropped, see SQL Server Columns Under the Hood.
In order to reclaim the space you need to drop the column and then rebuild the clustered index of the table, see ALTER INDEX ... REBUILD.
For the record (since SHRINK is not allowed in SQL Azure anyway) on the standalone SQL Server SHRINK would had solved nothing, this is not about page reservation but about physical record size.
It's counting the number of reserved pages to calculate the size. Deleting a column may reduce the number of pages that are actually utilized to store data, but the newly-freed pages are probably still reserved for future inserts.
I think you'd need to shrink the database to see the size decrease, as per: http://social.msdn.microsoft.com/Forums/en-US/ssdsgetstarted/thread/ae698613-79d5-4f23-88b4-f42ee4b3ad46/
As an aside, I am fairly sure that setting the value of a non-variable-length column (like a GUID) to null will not save you any space at all- only deleting the column will do so. This per Space used by nulls in database
I am looking into created an MD5 checksum persisted computed value for SQL Server 2008. There is gigabytes of data in this particular table(not my design) and I would like to know if I create the new computed column will this lock the entire table until the computation has finished?
Will the new column update all entries in the table or update them only when they are selected/updated via a SQL command?
What is the recommended practice for created MD5 computed values in SQL Server?
After kicking up an exact copy of the database on a virtual machine I experimented with adding the persisted column.
There were about 1 million rows and it took 2 hours to compute the MD5 hash column. In this time the entire table was locked for select, update and insert. On a production server you would have to factor in other issues.
I'm trying to convert the permanent tables used in a stored procedure to global temp tables. I've looked at the stats on these permanent tables and some have tens of millions of rows of data and are on the order if gigabytes in size (up to 10 GB).
So,
CREATE TABLE my_table (
column1 NUMBER,
column2 NUMBER,
etc...
)
TABLESPACE BIGTABLESPACE
NOLOGGING
NOCOMPRESS
NOCACHE
NOPARALLEL
MONITORING;
should become
CREATE GLOBAL TEMPORARY TABLE my_table (
column1 NUMBER,
column2 NUMBER,
etc..
)
ON COMMIT PRESERVE ROWS;
I'm creating an equivalent global temporary table with rows that should be preserved until the end of the session for each existing permanent table. This global temp table will be used in the procedure instead of the permanent table.
(EXECUTE IMMEDIATE 'TRUNCATE ...' at the start, and INSERT /*+ APPEND */ INTO at some later point)
All of the permanent tables have been created in a big tablespace BIGTABLESPACE
The Oracle docs state that the global temporary table will be created in the user's temp tablespace (I assume this is TEMP). The problem with this is that the TEMP tablespace is small and the extents are not set to grow to the size I need them to grow during the procedure.
The TEMP tablespace was created during the database creation
create database "$oracle\_sid"
user sys identified by "$sys\_password"
user system identified by "$system\_password"
set default bigfile tablespace
controlfile reuse
maxdatafiles 256
maxinstances $maxinstances
maxlogfiles 16
maxlogmembers 3
maxloghistory 1600
noarchivelog
character set WE8MSWIN1252
national character set AL16UTF16
datafile
'$oracle\_home/oradata/$oracle\_sid/system01.dbf' size 512M
logfile
'$oracle\_home/oradata/$oracle\_sid/redo01.log' size 1G,
'$oracle\_home/oradata/$oracle\_sid/redo02.log' size 1G,
'$oracle\_home/oradata/$oracle\_sid/redo03.log' size 1G
sysaux datafile
'$oracle\_home/oradata/$oracle\_sid/sysaux01.dbf' size 256M
default temporary tablespace temp tempfile
'$oracle\_home/oradata/$oracle\_sid/temp01.dbf' size 5G
undo tablespace "UNDOTBS1" datafile
'$oracle\_home/oradata/$oracle\_sid/undotbs01.dbf' size 5G;
The permanent tables (that I'm planning to replace) were originally created in tablespace BIGTABLESPACE
-- 50G bigfile datafile size
create bigfile tablespace "BIGTABLESPACE"
datafile '$oracle\_home/oradata/$oracle\_sid/bts01.dbf' size 50G
extent management local
segment space management auto;
The permanent table indexes were originally created in tablespace BIGTABLESPACE
-- 20G bigfile datafile size
create bigfile tablespace "BIGINDXSPACE"
datafile '$oracle\_home/oradata/$oracle\_sid/btsindx01.dbf' size 20G
extent management local
segment space management auto;
Is replacing these permanent tables with global temporary tables feasable?
The TEMP tablespace will run into a problem extending the TEMP tablespace. Is there a way to create global temporary tables and their indexes in tablespaces BIGTABLESPACE and BIGINDXSPACE?
If not, how can I make the TEMP tablespace behave like a bigfile tablespace and achieve index/table separation?
Can I create two TEMP bigfile tablespaces and create indexes into one and tables into another?
I want to use global temporary tables, but the volume of data I am handling in the procedure would seem to be above and beyond the indended design of global temporary tables.
Any suggestions?
There is no benefit to separating data and indexes into separate tablespaces other than potentially making DBAs more comfortable that similar objects are grouped together. There is a long-standing myth that separating indexes and data was beneficial for performance reasons-- that is not correct.
Temporary objects should (and must) be stored in a temporary tablespace. You could increase the size of your TEMP tablespace or create a separate temporary tablespace just for the user(s) that will own these objects if you wanted to segregate these large temporary tables into a separate tablespace. You can't (and wouldn't want to) store them in your permanent tablespaces.
Architecturally, though, I would be very curious about why temporary tables were necessary in your system. If you have sessions that are writing 10's of GB into temporary tables, then presumably reading those 10's of GB out again in order to write the data somewhere else, I would tend to suspect that there were more efficient solutions. It is very rare in Oracle to even need temporary tables-- it is far more common in other databases where readers can block writers to need to copy data out of tables before working on it. Oracle has no such limitations.
I don't think that there's anything in your description that makes GTT's unattractive. You obviously need very large temporary tablespaces but you're not consuming more space overall unless you've been making heavy use of table compression (unavailable in GTT's at least up to 10gR2). Look into the use of tablespace groups: http://download.oracle.com/docs/cd/B19306_01/server.102/b14231/tspaces.htm#ADMIN01103
Using a tablespace group, rather than a single temporary tablespace, can alleviate problems caused where one tablespace is inadequate to hold the results of a sort, particularly on a table that has many partitions. A tablespace group enables parallel execution servers in a single parallel operation to use multiple temporary tablespaces.
Also, don't neglect the use of subquery factoring clauses. They can often replace the use of temporary tables. However they might still require just as much temporary storage space because a large result set from a SQFC can spill to disk to avoid the consumption of too much memory, so you still have to go ahead with the increase in TEMP space. They're very handy for not having to deploy a new database object every time you need a new temporary table.
I looked at large sized Global Temporary Tables for a migration exercise. It worked but for debugging and rejection hadling I eventually went with plain tables.
If the GTTs don't work out, consider either Row-Level Security / VPD (or even views).
You can have a column derived from sys_context('USERENV','SESSIONID') and use that to ensure that the user can only see their own data.
Still the thought of multiple sessions dealing with multi-gigabyte datasets concurrently is a bit scary.
PS. I believe that for GTTs used through a procedure use the temp tablespace of the session user rather than the temp tablespace of the procedure owner. If you can get the sessions as separate oracle users then you have a chance at spreading your file IO over different tablespaces.