I have a table in a SQL Azure DB (s1, 250Gb limit) with 47.000.000 records (total 3.5Gb). I tried to add a new calculated column, but after 1 hour of script execution, I get: The service has encountered an error processing your request. Please try again. Error code 9002 After several tries, I get the same result.
Script for simple table:
create table dbo.works (
work_id int not null identity(1,1) constraint PK_WORKS primary key,
client_id int null constraint FK_user_works_clients2 REFERENCES dbo.clients(client_id),
login_id int not null constraint FK_user_works_logins2 REFERENCES dbo.logins(login_id),
start_time datetime not null,
end_time datetime not null,
caption varchar(1000) null)
Script for alter:
alter table user_works add delta_secs as datediff(second, start_time, end_time) PERSISTED
Error message:
9002 sql server (local) - error growing transactions log file.
But in Azure I can not manage this param.
How can I change my structure in populated tables?
Azure SQL Database has a 2GB transaction size limit which you are running into. For schema changes like yours you can create a new table with the new schema and copy the data in batches into this new table.
That said the limit has been removed in the latest service version V12. You might want to consider upgrading to avoid having to implement a workaround.
Look at sys.database_files by connecting to the user database. If the log file current size reaches the max size then you hit this. At this point either you have to kill the active transactions or update to higher tiers (if this is not possible because of the amount of data you modifying in a single transaction).
You can also get the same by doing:
DBCC SQLPERF(LOGSPACE);
Couple ideas:
1) Try creating an empty column for delta_secs, then filling in the data separately. If this still results in txn log errors, try updating part of the data at a time with a WHERE clause.
2) Don't add a column. Instead, add a view with the delta_secs column as a calculated field instead. Since this is a derived field, this is probably a better approach anyway.
https://msdn.microsoft.com/en-us/library/ms187956.aspx
Related
I try to add a column to a table with GUI Tableplus, but no response for long time.
So I turn to the db server, but got these error:
Maybe some inconsistent data generated during the operation through the Tableplus.
I am new to postgresql , and don't know what to do next.
-----updated------
I did some operation as #Dri372 told, and got some progress.
The failed reason for table sys_role and s2 is that the tables are not empty, they have some records.
If I run sql like this create table s3 AS SELECT * FROM sys_role; alter table s3 add column project_code varchar(50);, I successed.
Now how could I still work on the table sys_role?
I have a user uploading a csv having 200k rows of IPs. I have to verify if the uploaded IPs already exists in database. What will be the optimized way to query the database?
I am using hibernate to connect to the Oracle DB and I am aware that IN query operator accepts limited values (1000). So to fetch 200k values it will execute 200 queries, which is time consuming and not an optimized way.
What is the best way to query the database?
If you expect many collisions (uploaded IPs already in the database), I'd go for the staging table approach krokodilko and pmdba mentioned. Usually, I even compute a delta between existing and uploaded data, and log the numbers (120.000 ips already known, 25.000 new ips) to have some statistics.
On the other hand, if you don't expect collisions, and just want to protect your data from duplications, I'd use the error logging approach:
CREATE TABLE t (ip VARCHAR2(15) NOT NULL CONSTRAINT pkt PRIMARY KEY);
EXEC DBMS_ERRLOG.CREATE_ERROR_LOG(dml_table_name => 't');
Now you have your table t and an error log table err$_t which you can use to catch the offending rows:
INSERT INTO t (ip) VALUES ('127.0.0.1') LOG ERRORS;
1 row inserted.
INSERT INTO t (ip) VALUES ('127.0.0.1') LOG ERRORS;
0 rows inserted.
ORA-00001: unique constraint (SO.PKT) violated
SELECT ip, ora_err_mesg$ FROM err$_t;
IP ORA_ERR_MESG$
127.0.0.1 ORA-00001: unique constraint (SO.PKT) violated
We have a SQL Server 2008 database with a table containing more than 1.4 billion records. Due to adjustments of the coordinate system, we have to expand the datatype of the coordinate column from decimal(18, 2) to decimal(18, 3).
We've tried multiple things but everything resulted in an exception (transactionlog is full) after about 14 hours of execution.
These are the things we tried:
Alter Table
ALTER TABLE Adress
ALTER COLUMN Coordinate decimal(18, 3) NULL
Designer
Uncheck Tools > Options > Designer > Prevent saving changes that require table re-creation
Open Designer
Change datatype of column to decimal(18, 3)
Right-click > Generate Change Script...
What this script does, is creating a new table with the new datatype, copying the old data to the new table, drop the old table and rename the new table.
Unfortunately both attempts result in a transaction log full exception after 14 hours of execution.
I thought, that changing the datatype via ALTER TABLE... ALTER COLUMN... is only changing the metadata and should be finished in the matter of (milli)seconds?
Do you know of any other method I could try?
Why are my attempts (especially #1) needing that much time?
Thanks in advance
Well the main issue seems large amount of data saved into the table. Your both attempts also seem fine. They both will definitely take time I must say as the data is large.
Each time you alter a column data type the SQL SERVER tries to convert existing data into targeted data type. Processing the conversion on large amount of data may cause delay in execution.
Moreover I wonder if you have any trigger on the table.?
Well! Finally I would suggest you following steps. Give it a try at least
Remove any primary keys/indexes/constraints pointing to the old column, and disable any trigger (if there is any).
Introduce a new nullable column with the new data type (even if
it is meant to be NOT NULL) to the table.
Now make an update query on the table which will set the new column value to the old column value. You can do updating in chunks while updating 1000/100000 batches of the records. And also you can apply conditions to the query for better results.
Once you update all the table by setting new column values to old column then remove the NULL character to NOT NULL from designer (if it is meant to be NOT NULL).
Drop/Delete the old column. Perform Select Query and Verify Your Changes.
Last Point I should add is your database transaction log is also full which can be shrunk but with some precautions. Here is very good example how to reset your transaction log. Should take a look at this too.
Hope This Helps. :)
The solution is to do the updates in batches, easing the pressure on the log file.
Method 1:
a) Create a new table with the new definition.
b) Copy the data to the new table in batches.
c) Drop the old table.
d) Rename the new table.
Method 2:
a) Create a new column with the correct definition.
b) Update the new column with data from the old column in batches.
c) Drop the old column.
d) Rename the new column.
Method 3:
a) BCP the data into a file.
b) Truncate the table.
c) Alter the column.
d) Set the recovery model to bulk logged or simple.
e) BCP the data from the file into the table.
f) Set the recovery model back to full.
Add new column as the last column
If you try to insert before the last column it could take a long time
NewCoordinate decimal(18, 3) NULL
select 1
while(##rowcount > 0)
BEGIN
UPDATE TOP(10000) Adress
SET NewCoordinate = Coordinate
WHERE NewCoordinate <> Coordinate
END
That is my suggestion:
ADD a field to your table and name it like below:
NewCoordinate DECIMAL(18, 3) NULL
WHILE(1 = 1)
BEGIN
UPDATE TOP(1000) Adress SET NewCoordinate = Coordinate
WHERE NewCoordinate IS NULL
IF (##ROWCOUNT < 1000)
BREAK
END
Try to keep your transaction like small.
And Finaly drop your Coordinate field.
[I am asking here instead of GIS Stackexchange because this maybe more of a SQL Server issue?]
I have SQL Server ArcSDE connection in which data is batch inserted via some scripts. Currently, anytime there is a new row of data then an 'OBJECTID' column, set to INT and Identity Column increases by number 1. So far so good. Except I need to enable "versioning" on the table.
And so I follow this: http://resources.arcgis.com/en/help/main/10.1/index.html#//003n000000v3000000
but get errors because ArcGIS is complaining about the Identity column, per: http://support.esri.com/cn/knowledgebase/techarticles/detail/40329 ; and when I remove the Identity attribute to the column then the column value becomes NULL--not good.
So, in my scenario, how I can I increase the value of OBJECTID by 1 number as auto-increment? I supposed, I can just insert some GUID into the 'OBJECTID' field through the script? Also, if I follow the GUID route then I am not sure if I will be able to add rows manually via ArcGIS Desktop on occasional basis?
Thanks!
Update 1 Okay, so changed the OBJECTID field to a 'uniqueidentifier' one with a default GUID value and now I am able to enable "versioning" using ArcGIS Desktop. However, ArcGIS is expecting GUID to be an INT data type--and so no go?
In light of my "update 1" in the Question above I managed to take care of this by inserting an INT value for OBJECTID during the batch insertions per the following: How to insert an auto_increment key into SQL Server table
so per the above link, I ended doing:
INSERT INTO bo.TABLE (primary_key, field1, fiels2) VALUES ((SELECT ISNULL(MAX(id) + 1, 0) FROM bo.Table), value1, value2)
EXCEPT in my case the IDENTITY remains not 'ON' at all either in the database or, unlike the above link, I didn't have to set Identity On/Off during batch insertions; Works for some reasons anyway!
I'm changing some bad design on a table. The field I'm trying to change holds IIS7 session id's which are long numbers. I'm trying to change the field from nvarchar(20) to int. There are 349,000 records in the table.
SQL Server Management Studio times out after 35 seconds. However, if I check the query timeout setting for the connection it is set at 600 seconds (and I can't change it either).
Is there another timeout setting that could be causing this problem?
Here's the error message I'm getting:
- Unable to modify table.
Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
I've been able to change several other tables with no problem. Of course, they had fewer rows.
This database is on a shared hosting package at Arvixe.com. Do you think this could be part of the problem?
Can you try to run a T-SQL script instead of doing this using the visual designer?
ALTER TABLE dbo.YourTable
ALTER COLUMN YourColumn INT
Now, this will only work if all rows are truly valid INT values! Otherwise, it'll bomb out at some point....
To check if all your rows are truly valid INT, you could run this query:
SELECT * FROM dbo.YourTable
WHERE ISNUMERIC(YourColumn) = 0
This will select all rows that are not valid numerics ... if you get rows here, you have a problem...