We are currently running a database on SQL Server 2000. The database constantly imports data from flat files for later querying. This process is done by a series of SQL Server stored procedures and calling BCP from those procedures through xp_cmdshell. The scripts use BCP to read in the flat files into persistent tables in a secondary database on the same server. Then the scripts will pull the data from the import database tables and put it into the real database which is normalized and used for querying.
Normally this import process takes 5-10 minutes depending on the size of the files. However for the past week it has been taking 50-60 minutes. We have tried stepping through the procedures. We've noticed that as soon as we create a temporary table, that we are unable to query tempdb from another query window. But more importantly when can step through just fine up until we truncate the first import table. We allow the truncate to execute and then when we check for locks in the database using sp_lock, we see that the locks taken by truncate are not released. We then allow the xp_cmdshell call to BCP to execute and it will sit there. We look at CPU and I/O and see no significant activity when the procedure is stuck calling BCP. Also no other known queries are running when we try this. Note that the file itself is small; at most 20 lines.
If we run the bcp command with xp_cmdshell from a separate query it will work fine, but lock up if we have executed the truncate line but not yet the BCP line in the stored procedure.
So the questions we have are why is the server getting into this deadlock state and less importantly why would the locks made by truncate not release?
Locks in SQL Server are only held if there is an active transaction. You have to commit the transaction before you call the external BCP. Otherwise you will run into this Deadlock.
Incidentally this Deadlock cannot be detected by SQL Server, so if it happens, you will be stuck in it forever.
Related
I have an SSIS package which when runs, updates a table. It is using a staging table and subsequently, uses slowly changing dimension table to load data into the warehouse. We have set it up as a SQL Agent job and it runs every two hours.
The isolation level of the package is serializable. The database isolation level is read committed.
The issue is that when this job runs, this job blocks that table and therefore, clients cannot run any reports. It blanks it out.
So what would be the best option for me to avoid it? clients need to see that data, meanwhile, we need to update the table every two hours.
Using Microsoft SQL Server 2012 (SP3-GDR) (KB4019092) - 11.0.6251.0 (X64)
Thanks.
You're getting "lock escalation". It's a feature, not a bug. 8-)
SQL Server combines large numbers of smaller locks into a table lock to improve performance.
If INSERT performance isn't an issue, you can do your data load in smaller chunks inside of transactions and commit after each chunk.
https://support.microsoft.com/en-us/help/323630/how-to-resolve-blocking-problems-that-are-caused-by-lock-escalation-in
Another option is to give your clients/reports access to a clone of your warehouse table.
Do your ETL into a table that no one else can read from, and when it is finished, switch the table with the clone.
I have VBS called stored procedures starting but not completing and returning query timeout messages to the application.
We are on SQL Server 2008 SP2 release 1, Windows Server 2008 R2 Standard.
This is not an issue in production, but in dev we have VBS scripts associated with the DrillThrough database producing:
VBS<105,4> Microsoft OLE DB Provider for SQL Server: Query timeout expired.
These scripts are automated on the application server side. The stored procedure queries run fine in SSMS but do take over 40 seconds.
ALL the scripts are all set not to timeout:
conn.ConnectionTimeout = 0
conn.CommandTimeout = 0
The OLEDB login is the dbo. The only scripts failing are on the DrillThrough database. The failing scripts use more than one database and functions in the DrillThrough database to gather data in temp tables to update the actual drill through table.
There is one nightly process that is completing on the DrillThrough database where the stored procedure runs in about 2 seconds. It does not use any functions but does compare data from two records pulled from two other databases before writing the difference to a table.
I have compared the scripts and they are set up the same except for the stored procedures they call.
I updated statistics on the tables just in case. The query still timed out on the application side, and actually took a little longer to run in SSMS.
The SQL Server remote connections is set to 0 for no timeout.
It definitely gets into the procedure because it truncates the tables, bulk loads the holding table and deletes from the actual drill through table based on the holding table data and creates the first temp table but it is not completing and is dropping out somehow.
Again, the stored procedure runs fine in SSMS all the way through.
Everyone refers back to the SQL Server setting and the ConnectionTimeout and CommandTimeout. No one seems to be experiencing a situation where the stored procedure starts but doesn’t complete.
Other things I’ve tried with the same results:
Adding ‘With Recompile’ to the stored procedure
Setting an actual timeout time in the VB Script
I have tuned the query as much as I can
I can't get a faster development server.
Accidentally the same stored procedure was run twice at one time on our MS SQL Server 2008 R2. They were run from the same SQL Server Management Studio client and I tried to cancel them both. After 45 minutes neither process has cancelled. I read somewhere that it would help to close the query windows in SSMS so I did. However the tables the stored procedure should be writing to is still locked even though almost 20 hours has passed. I guess there is a deadlock. I hit cancel seconds after the stored procedure was initiated.
The table the stored procedure is reading from is not locked. The stored procedure is in CLR. It reads from a table, manipulate data and then use SqlBulkCopy to insert into three other tables.
The data in the tables the SP writes to can very easily be recreated. However I can neither drop nor truncate them due to the lock. I also tried KILL SPID with no result.
I have been thinking about restarting the server, but I guess it would not help because of SQL Servers data integrity.
I would really like some input on how release the lock. Several websites with lots of users depend on the database server so solutions that do not involve restarting would be much appreciated.
I am using SQL server 2005, SQL server studio client. I am having a long stored procedure (it does a bunch of table joins, some delete, some insert and some updates) periodically running (approximately every other 2 minutes).
After I have this sp, I noticed that my database is sometimes not responding (it happens a few times when the SP is not running, and many times during the SP is running). When the DB is not responding, i can't open new connection from the SQL server studio client, if i run a query/sp the status will become running and stays that forever, until i manually reset the SQL service from control panel admin-tools.
Have you seen similar problems?
Is it perhaps because the SP I newly created is doing too much things and cause the DB to crash?
A few more quick tips:
You can use the sp_who2 command to view current sessions on the DB and determine if the SP is blocked, or blocking other processes.
Check the estimated execution plan for the SP, and look for sources of slowness, like table scans.
Your SP might be making too many changes in a single transaction and filling up the transaction log or causing it to grow. Inspect your DB's recovery mode, and whether backups are occurring regularly so that transaction log space can be reused. Consider batching large modifications into smaller chunks.
It sounds like your SP might be causing locks. You could use SQL Profiler to try and dig deeper into what might be happening. Here's a link providing further info -
http://www.simple-talk.com/sql/learn-sql-server/how-to-track-down-deadlocks-using-sql-server-2005-profiler/
Stored procedures are compiled on first use.
There are options to clear cache:
DBCC FREEPROCCACHE
DBCC DROPCLEANBUFFERS
--To Verify whether the cache is emptied
--DBCC PROCCACHE
or to recompile or to reduce recompilations.
But is it possible to force frequently used stored procedures' execution plans be pre-cached and stay in memory?
I know how to do it in ADO.NET, i.e. from outside of SQL Server, but this question is how to do inside SQL Server - to be launched with the start of SQL Server itself.
(*) For example, I see in SSMS Activity Monitor a running process (Task State: RUNNING, Command: SELECT) that is continuously executing T-SQL (according to Profiler) in context of tempdb database though SQL Server Agent is disabled and SQL Server is not loaded by anything, see "Details of session 54" in "Where are all those SQL Server sessions from?".
How would I do the similar resident process (or, rather, auto-starting by SQL Server start service or session) periodically recycling stored procedure?
Related question:
Stored procedure executes slowly on first run
Update:
Might be I should have forked this question in 2 but my main curiosity is how to have periodic/ looping activity with SQL Server Agent disabled?
How was it made with mentioned above RUNNING SELECT session (*)?
Update2:
Frequently I observe considerable delays while executing stored procedures querying very small amount of data which cannot be explained only through necessity to read huge amounts of data.
Can we consider this - considerable delays on insignificantly small data - as context of this question?
Just execute it from a script. You could do this after any sql server restart. If they are frequently used, it shouldn't be much of a problem after that.
Seems like this question eventually got answered in:
Can I get SQL Server to call a stored proc every n seconds?
Update: These tips will do the trick:
Keeping data available in the SQL Server data cache with PINTABLE
Automatically Running Stored Procedures at SQL Server Startup