I came in this morning and one of my staging tables is completely unresponsive. Any query involving the table runs indefintely or until timeout. I can't even do SELECT OBJECT_ID('staging.mytable') as it will run forever. I've tested this across several different computers so the problem isn't in the client (ssms). I see no hanging processes in Profiler. Any queries involving the table in question just start and then wait. I cannot drop or truncate the table.
Any ideas on where to go from here? Its just a staging table so there's no real loss in dropping it, but it is gumming up the works right now across all of my procs/scripts.
There are a few different ways to troubleshoot this when it happens.
1) Use sp_who2 (ms_shipped proc) or use sp_whoisactive (way better, written by MVP Adam Machanic, have to download it and run the script in SSMS to create the proc) that will show running processes and who is running them. Find the ones that are blocking and go from there.
2) launch the activity monitor with ctl+alt+A and filter the running sessions for blocked or head blocker to see who is running a process that is blocking others.
Related
I created a stored procedure which sends an email and accidentally called the stored procedure within itself creating an endless loop. Within a few seconds of executing the stored procedure I realized what I had done and fixed the loop, but it had already created 517 processes. I killed all the SPID's but they are stuck in a KILLED/ROLLBACK state.
This code shows me the processes:
select session_id,handle.percent_complete, *
from sys.dm_exec_requests handle
outer apply sys.fn_get_sql(handle.sql_handle) spname
where cast(handle.start_time as date) = '2022-01-10'
spname.text is showing 'xp_sysmail_format_query' for all the SPID's. It's been two days, and all 517 processes have been stuck in this rollback state with 0% progress. We are still able to use all our business applications and execute queries, with the exception of EXEC msdb.dbo.sp_send_dbmail which, when starting even a test email, gets stuck executing and has to be cancelled. This is not good because any auto generated email warnings will not be sent, and all other sql email functions are blocked. I'm not sure what other jobs are being blocked at this time.
This is a huge problem and I cannot find a solution. I've read every post I can find about this. I've tried everything I can think of except restarting the SQL server. Some posts state that restarting the SQL server can fix this and some state not to restart it or that the tasks will just resume in the killed/rollback state when restarted. I tried killing the spids again with statusonly but that just informs me that they are in a rollback state with 0% complete.
Should I restart the server and will this fix anything? Is there another solution other than restoring the DB to a backup that is more than 2 days old and losing all the work the entire business has done in the last couple days?
Any assistance will be greatly appreciated.
As robust as it is, sometimes (fortunately rarely) SQL Server leaves us no choice when killing a process to adopt the IT mantra of turning it off an on again when the rollback does not complete in a timely fashion.
This can be more prevalent when a transaction enlists external methods or functions, email is notorious for this inparticular.
As unwelcome as it is, it's often the least-expensive in terms of time and should be considered an option soon in the diagnosis process when the low-hanging fruit options have been exhausted.
I have a SQL Server agent job running every 5 minutes with SSIS package from SSIS Catalog, that package does:
DELETE all existing data ON OLTP_DB
Extract data from Production DB
DELETE all existing data on OLAP_DB and then
Extract data transformed from OLTP_DB into OLAP_DB ...
PROBLEM
That job I mentioned above is hanging randomly for some reason that I don't know,
I just realize using the activity monitor, every time it hangs it shows something like:
and if I try to run any query against that database it does not response just say executing.... and nothing happen until I stop the job.
The average running time for that job is 5 or 6 minutes, but when it hangs it can stay running for days if I don´t stop it. :(
WHAT I HAVE DONE
Set delayValidation : True
Improved my queries
No transactions running
No locking or blocking (I guess)
Rebuild and organize index
Ran DBCC FREEPROCCACHE
Ran DBCC FREESESSIONCACHE
ETC.....
My settings:
Recovery Mode Simple
SSIS OLE DB Destination
1-Keep Identity (checked)
2-Keep Nulls (checked)
3-Table lock (checked)
4-Check constraints (unchecked)
rows per batch (blank)
Maximum insert commit size(2147483647)
Note:
I have another job running a ssis package as well (small) in the same instance but different databases and when the main ETL mentioned above hangs then this small one sometimes, that is why I think the problem is with the instance (I guess).
I'm open to provide more information as need it.
Any assistance or help would be really appreciated!
As Joeren Mostert said, it's showing CXPACKET which means that it's executing some work in parallel. (cxpacket)
It's also showing ASYNC_NETWORK_IO (async_network_io) which means it's also transfering data to the network.
There could be many reasons. Just a few more hints:
- Have you checked if network connection is slow? - What is the size of the data being transfered vs the speed of the network? - Is there an antivirus running that could slow the data transfer?
My guess is that there is lots of data to transfer and that it's taking a long time. I would think either I/O or network but since you have an asyn_network_io that takes most of the cumulative wait time, I would go for network.
As #Jeroen Mostert and #Danielle Paquette-Harvey Said, By doing right click using the activity monitor I could figure out that I had an object that was executing in parallel (for some reason in the past), to fix the problem I remove the parallel structure and put everything to run in one batch.
Now it is working like a charm!!
Before:
After:
I am running several DB instances on the dedicated Azure SQL Server. All instances are S0. They all run the same processes yet one of them got into trouble (never happened before). The automated SP which purges old records from tables periodically did not seem to be able to finish. It has been running over 12 hours now (should finish in minutes) and I can't figure out how to stop it (obviously restart is not available).
I tried to KILL the process (the SP in question) but that did not seem to work (see below). Any help appreciated. No need to worry about the data but trying not to go nuclear on this and delete the whole database.
Try renaming the database and rename it back or alter the database to different SLO and revert it. The other option is to turn ON / OFF RCSI with rollback immediate
I'm using SQL Server 2012 and I'm stuck with a strange problem.
I tried to restore a database snapshot to a database. Usually it doesn't take much time to restore, but now it took 5 minutes and is still restoring, so I stopped query execution. It was trying to stop the query execution for more that 5 minutes, so i closed SSMS using task manager.
Then I tried to kill the restore process using KILL.
Now I am able to can connect to that server, but the list of databases is not opening. I mean: whoever is connected to this server, they are not able to get the databases. When I checked the sysprocesses table, it is showing lastwaittype as LCK_M_S.
None of my users can see databases. Looks like I kind of messed up. I cannot restart SQL Server as others are connected to the server.
How do I solve this? Please help.
EDIT:
i tried this approach and when i checked with
select db_name(dbid), * from sysprocesses where blocked <> 0
i got two records,
do you think because of these two rest of the process are getting locked up.
I'm guessing there still is some hidden lock on the sysdatabases table in the master database. This could very well be caused by the KILL of the restore command.
The article here might help you:
http://ellisweb.net/2012/02/clearing-out-a-mysterious-table-lock-lck_m_s-in-sql-server-2008/
It basically advises you to identify where the hidden lock is coming from, and then issuing a KILL on that process ID.
Try restarting the instance. Can't hurt if your users can't see any of the databases anyway.
Our application runs alongside another application on a customers machine.
We have put some efforts regarding avoiding long-running locks in tempdb since this obviously affects concurrency badly.
The other application, however does things like:
begin transaction
create #Table(...);
insert into #Table(....) values(...);
operation_for_totally_six_seconds().
commit;
Since the operations take time, our application get stuck waiting for the locks aquired by the other application.
Now, I expect there to be a way to isolate my application from the other application by for example telling sql server to assign me another tempdb, but I have not found a way. Is this somehow possible or is the solution to install our database on another mssql-instance?
Regards,
Jens Nordenbro
long-running locks in tempdb since
this obviously affects concurrency
badly
That is actually not obvious at all. Long held locks are of importance only if you and the other application go after the same locks. The code sample you posted is perfectly legitimate. First of all #temp is a connection specific table that no other connection can even see it. But even if it would be global resource, it would belong to the other application and hence you would have no business acquiring locks on it.
As an exercise, open an SSMS query window and run this:
begin transaction;
create table #temp (a int);
Then opne a second query window and run the same. QED they don't block each other, despite creating the very same #temp table.
If tempdb is indeed a bottle neck you need to do some more investigation and find the actual resources that contention occurs on.
One option is to run your app in a different instance of sql server on the same machine. This way you would have your own tempdb.