I have a query that return a huge number of rows and I am using SELECT INTO (instead of INSERT INTO) to avoid having problems with transaction log.
The problem is: while this query is running, I can read objects but not showing them in object explorer. When I try to expand the tables item, for example, I receive the message bellow:
Is there a way to avoid this problem?
As M.Ali explained, SELECT INTO has a table lock on your new table, which is also locking the schema objects that SSMS is trying to query in order to build the tree browser.
I would suggest tuning the query so that the statement can run faster. Since this is inserting into a Heap with no indexes and has the tablock, it will be minimally logged as you stated. So it is likely the SELECT part of the statement that is causing things to be slow. See if that query can be optimized or broken into smaller pieces so that the statement does not run so long.
Alternatively, perform the insert in smaller batches using INSERT INTO (and not specifying the tablock hint)
Now here is a Test for you which will give answer to your question...
Open a Query window in SSMS. Write any query which will return any number or rows, could be only one row or maybe 10. and do as follows
Query window 1
BEGIN TRANSACTION;
SELECT *
INTO NEW_Test_TABLE
FROM TABLE_NAME
Query Window 2
Now Open another window and write a SELECT statement against this NEW_Test_TABLE.
SELECT * FROM NEW_Test_TABLE
Your Query will never finish executing,,, no results will be returned (At this time NEW_Test_TABLE only exists in buffer chache). Unless you go back to your 1st Query Window and commit the transaction, And if you goto query window 1 and ROLLBACK TRANSACTION NEW_Test_TABLE would have existed once in buffer chache and no longer exist anywhere.
Similarly when your Select into statement in being executed nothing is committed to disk, therefore SSMS cannot see it neither can show you any information about it via Object explorer.
So the answer is while the query is being executed be patient and let SQL Server Commit the SELECT INTO transaction to disk and you will be able to access it VIA querying it or via Object explorer.
Related
I want to copy one big database table to another. This is my current approach:
OPEN CURSOR WITH HOLD lv_db_cursor FOR
SELECT * FROM zcustomers.
DO.
REFRESH gt_custom.
FETCH NEXT CURSOR lv_db_cursor
INTO TABLE gt_custom
PACKAGE SIZE lv_package_size.
IF sy-subrc NE 0.
CLOSE CURSOR lv_db_cursor.
EXIT.
ENDIF.
INSERT zcustomers1 FROM TABLE gt_custom.
* Write code to modify u r custom table from gt_custom .
ENDDO.
But the problem is that I get a error "Enterprise]ASE has run out of LOCKS".
I tried to use COMMIT statement after insert some piece of records, but it closes the cursor.
I don't want to increase max locks by database setting or make a copy on database level.
I want to understand how I can copy with best performance and low usage memory in ABAP...
Thank you.
You can also "copy on database level" from within ABAP SQL using a combined INSERT and SELECT:
INSERT zcustomers1 FROM ( SELECT * FROM zcustomers ).
Unlike the other solution, this runs in one single transaction (no inconsistency on the database) and avoids moving the data between the database and the ABAP server, so should be by magnitudes faster. However, like the code in question this might still run into database limits due to opening many locks during the insert (though might avoid other problems). This should be solved on database side and is not a limitation of ABAP.
By using COMMIT CONNECTION instead of COMMIT WORK it is possible to commit only the transaction writing to zcustomers1, while keeping the transaction reading from zcustomers open.
Note that having multiple transactions (one reading, multiple writing) can create inconsistencies in the database if zcustomers or zcustomers1 are written while this code runs. Also reading from zcustomers1 shows only a part of the entries from zcustomers.
DATA:
gt_custom TYPE TABLE OF ZCUSTOMERS,
lv_package_size TYPE i,
lv_db_cursor TYPE cursor.
lv_package_size = 10000.
OPEN CURSOR WITH HOLD lv_db_cursor FOR
SELECT * FROM zcustomers.
DO.
REFRESH gt_custom.
FETCH NEXT CURSOR lv_db_cursor
INTO TABLE gt_custom
PACKAGE SIZE lv_package_size.
IF sy-subrc NE 0.
CLOSE CURSOR lv_db_cursor.
EXIT.
ENDIF.
MODIFY zcustomers2 FROM TABLE gt_custom.
" regularily commiting here releases a partial state to the database
" through that, locks are released and running into ASE error SQL1204 is avoided
COMMIT CONNECTION default.
ENDDO.
I am using Sql Profiler all the time to capture the SQL statements and rerun problematic ones. Very useful.
However, some code uses the SqlBulkCopy API and I have no idea how to capture those. I see creation of temp tables, but nothing that populates them. Seems like SqlBulkCopy bypasses Sql Profiler or I do not capture the right events.
Capturing event info for bulk insert operations ( BCP.EXE, SqlBulkCopy, and I assume BULK INSERT, and OPENROWSET(BULK... ) is possible, but you won't be able to see the individual rows and columns.
Bulk Insert operations show up as a single (well, one per batch, and the default is to do all rows in a single batch) DML statement of:
INSERT BULK <destination_table_name> (
<column1_name> <column1_datatype> [ COLLATE <column1_collation> ], ...
) [ WITH (<1 or more hints>) ]
<hints> := KEEP_NULLS, TABLOCK, ORDER(...), ROWS_PER_BATCH=, etc
You can find the full list of "hints" on the MSDN page for the BCP Utility. Please note that SqlBulkCopy only supports a subset of those hints (e.g. KEEP_NULLS, TABLOCK, and a few others) but does not support ORDER(...) or ROWS_PER_BATCH= ** (which is quite unfortunate, actually, as the ORDER() hint is needed in order to avoid a sort that happens in tempdb in order to allow the operation to be minimally logged (assuming the other conditions for such an operation have also been satisfied).
In order to see this statement, you need to capture any of the following events in SQL Server Profiler:
SQL:BatchStarting
SQL:BatchCompleted
SQL:StmtStarting
SQL:StmtCompleted
You will also want to select, at least, the following columns (in SQL Server Profiler):
TextData
CPU
Reads
Writes
Duration
SPID
StartTime
EndTime
RowCounts
And, since a user cannot submit an INSERT BULK statement directly, you can probably filter on that in Column Filters if you merely want to see these events and nothing else.
If you want to see the official notification that a BULK INSERT operation is beginning and/or ending, then you need to capture the following event:
SQLTransaction
and then add the following Profiler columns:
EventSubClass
ObjectName
For ObjectName you will always get events showing "BULK INSERT" and whether that is beginning or ending is determined by the value in EventSubClass, which is either "0 - Begin" or "1 - Commit" (and I suppose if it fails you should see "2 - Rollback").
If the ORDER() hint was not specified (and again, it cannot be specified when using SqlBulkCopy), then you will also get a "SQLTransaction" event showing "sort_init" in the ObjectName column. This event also has "0 - Begin" and "1 - Commit" events (as shown in the EventSubClass column).
Finally, even though you cannot see the specific rows, you can still see operations against the Transaction Log (e.g. insert row, modify IAM row, modify PFS row, etc) if you capture the following event:
TransactionLog
and add the following Profiler column:
ObjectID
The main info of interest will be in the EventSubClass column, but unfortunately it is just ID values and I could not find any translation of those values in MSDN documentation. However, I did find the following blog post by Jonathan Kehayias: Using Extended Events in SQL Server Denali CTP1 to Map out the TransactionLog SQL Trace Event EventSubClass Values.
#RBarryYoung pointed out that EventSubClass values and names can be found in the sys.trace_subclass_values catalog view, but querying that view shows that it has no rows for the TransactionLog event:
SELECT * FROM sys.trace_categories -- 12 = Transactions
SELECT * FROM sys.trace_events WHERE category_id = 12 -- 54 = TransactionLog
SELECT * FROM sys.trace_subclass_values WHERE trace_event_id = 54 -- nothing :(
** Please note that the SqlBulkCopy.BatchSize property is equivalent to setting the -b option for BCP.EXE, which is an operational setting that controls how each command will break up the rows into sets. This is not the same as the ROWS_PER_BATCH= hint which does not physically control how the rows are broken up into sets, but instead allows SQL Server to better plan how it will allocate pages, and hence reduces the number of entries in the Transaction Log (sometimes by quite a bit). Still my testing showed that:
specifying -b for BCP.EXE did set the ROWS_PER_BATCH= hint to that same value.
specifying the SqlBulkCopy.BatchSize property did not set the ROWS_PER_BATCH= hint, BUT, the benefit of reduced Transaction Log activity was somehow definitely there (magic?). The fact that the net effect is to still gain the benefit is why I did not mention it towards the top when I said that it was unfortunate that the ORDER() hint was not supported by SqlBulkCopy.
You cann't capture SqlBulkCopy in SQL Profiler because SqlBulkCopy doesn't generate SQL at all when inserts data in SQL Server table. SqlBulkCopy works similar to bcp utility and loads data directly into SQL Server file system. It's even can ignore FKs and triggers when inserts the rows!
I used an SQLite database and run an EXPLAIN statement before executing the actual query to verify if there was any attempt to write on the database.
Now, we have migrated to SQL Server and I need to know if a query tries to write on the database or is just a simple SELECT statement. I basically try to avoid any malicious statement.
You can see the estimated query plan of any query in SSMS by clicking the estimated query plan button.
See MSDN.
However, if the user shouldn't be writing to the database, is shouldn't have the permissions to do so. Ensure it belongs to a role that has restricted permissions.
If you do decide to go this route, you could do the following:
set showplan_xml on
go
set noexec on
go
select * from sysobjects
go
set noexec off
go
set showplan_xml off
go
This will return 3 result sets containing a single column of XML. The 2nd result set is the query plan for the actual query (in this case, select * from sysobjects)
But as noted in my comment, you'd be better off preventing the user having permissions to make any changes.
It's also possible to craft statements that are "only" selects but that are also pretty malicious. I could easily write a select that exclusively locks every table in the database and takes an hour to run.
i have a table which was always updatable before, but then suddenly i can no longer update the any of the columns in the table. i can still query the whole table and the results come back very fast, but the moment i try to update a column in the table, the update query simply stalls and does nothing.
i tried using
select req_transactionUOW
from master..syslockinfo
where req_spid = -2
to see if some orphaned transaction was locking the table, but it returns no results.
i can't seems to find signs of my table being locked, but simply cannot update it. any clues as to how to fix the table or whatever state it is in?
Could you please issue this query:
SELECT COUNT(*)
FROM mytable WITH (UPDLOCK, READPAST)
which will skip the locked records and make sure it returns the same number of records as
SELECT COUNT(*)
FROM mytable
You may need to repeat it with every index on the table forced, to make sure that no index resources is locked as well.
When you say "times out", does it hit the client timeout? For example, the default .net command timeout is 30 seconds. I would suggest increasing this to a very large value or running the update in SQL tools (by default no timeout set).
Other than that, an update will finish at some point or error and rollback: are you leaving enough time?
There is also the blocking, last index rebuild, last statistics update, triggers, accidental cross join, MDF or LDF file growth, poor IO, OS paging... etc. And have you restarted the SQL instance or server to remove environmental issues and kill all other connections?
There simply isn't enough information to make a judgement right now sorry.
I'm guessing this isn't a permissions issue as you're not getting an error.
So the closest I have had to this before is when the indexes on the table have become corrupt. Have you tried dropping the indexes and recreating them? Try one by one at first.
If you suspect locking, one of the first things I would do would be to run sp_lock. It will give you a list of all of the current locks held. You can use the DB_NAME and OBJECT_NAME functions to get the names that correspond to the dbid and ObjId columns.
Have you got any triggers on the table?
If so it could be that the trigger is failing so preventing the update.
Can you update other tables? If not (or anyways, if you like) you could check if the transaction log is full (if you use the full recovery model)/the partition your transaction log resides on is full. I think if SQL Server is unable to write to the transaction log you would/could experience this behaviour.
DBCC would be your friend: DBCC SQLPERF(LOGSPACE) shows you, how much (in percent) of your log is used. If it is (close to) 100% this might be your issue.
Just my two pennies worth.
I am moving a system from a VB/Access app to SQL server. One common thing in the access database is the use of tables to hold data that is being calculated and then using that data for a report.
eg.
delete from treporttable
insert into treporttable (.... this thing and that thing)
Update treportable set x = x * price where (...etc)
and then report runs from treporttable
I have heard that SQL server does not like it when all records from a table are deleted as it creates huge logs etc. I tried temp sql tables but they don't persists long enough for the report which is in a different process to run and report off of.
There are a number of places where this is done to different report tables in the application. The reports can be run many times a day and have a large number of records created in the report tables.
Can anyone tell me if there is a best practise for this or if my information about the logs is incorrect and this code will be fine in SQL server.
If you do not need to log the deletion activity you can use the truncate table command.
From books online:
TRUNCATE TABLE is functionally
identical to DELETE statement with no
WHERE clause: both remove all rows in
the table. But TRUNCATE TABLE is
faster and uses fewer system and
transaction log resources than DELETE.
http://msdn.microsoft.com/en-us/library/aa260621(SQL.80).aspx
delete from sometable
Is going to allow you to rollback the change. So if your table is very large, then this can cause a lot of memory useage and time.
However, if you have no fear of failure then:
truncate sometable
Will perform nearly instantly, and with minimal memory requirements. There is no rollback though.
To Nathan Feger:
You can rollback from TRUNCATE. See for yourself:
CREATE TABLE dbo.Test(i INT);
GO
INSERT dbo.Test(i) SELECT 1;
GO
BEGIN TRAN
TRUNCATE TABLE dbo.Test;
SELECT i FROM dbo.Test;
ROLLBACK
GO
SELECT i FROM dbo.Test;
GO
i
(0 row(s) affected)
i
1
(1 row(s) affected)
You could also DROP the table, and recreate it...if there are no relationships.
The [DROP table] statement is transactionally safe whereas [TRUNCATE] is not.
So it depends on your schema which direction you want to go!!
Also, use SQL Profiler to analyze your execution times. Test it out and see which is best!!
The answer depends on the recovery model of your database. If you are in full recovery mode, then you have transaction logs that could become very large when you delete a lot of data. However, if you're backing up transaction logs on a regular basis to free the space, this might not be a concern for you.
Generally speaking, if the transaction logging doesn't matter to you at all, you should TRUNCATE the table instead. Be mindful, though, of any key seeds, because TRUNCATE will reseed the table.
EDIT: Note that even if the recovery model is set to Simple, your transaction logs will grow during a mass delete. The transaction logs will just be cleared afterward (without releasing the space). The idea is that DELETE will create a transaction even temporarily.
Consider using temporary tables. Their names start with # and they are deleted when nobody refers to them. Example:
create table #myreport (
id identity,
col1,
...
)
Temporary tables are made to be thrown away, and that happens very efficiently.
Another option is using TRUNCATE TABLE instead of DELETE. The truncate will not grow the log file.
I think your example has a possible concurrency issue. What if multiple processes are using the table at the same time? If you add a JOB_ID column or something like that will allow you to clear the relevant entries in this table without clobbering the data being used by another process.
Actually tables such as treporttable do not need to be recovered to a point of time. As such, they can live in a separate database with simple recovery mode. That eases the burden of logging.
There are a number of ways to handle this. First you can move the creation of the data to running of the report itself. This I feel is the best way to handle, then you can use temp tables to temporarily stage your data and no one will have concurency issues if multiple people try to run the report at the same time. Depending on how many reports we are talking about, it could take some time to do this, so you may need another short term solutio n as well.
Second you could move all your reporting tables to a difffernt db that is set to simple mode and truncate them before running your queries to populate. This is closest to your current process, but if multiple users are trying to run the same report could be an issue.
Third you could set up a job to populate the tables (still in separate db set to simple recovery) once a day (truncating at that time). Then anyone running a report that day will see the same data and there will be no concurrency issues. However the data will not be up-to-the minute. You also could set up a reporting data awarehouse, but that is probably overkill in your case.