We had the error that the Microsoft Distributed Transaction Coordinator (MS DTC) has cancelled a transaction of BizTalk which causes some stuck messages in the Receive Port, it has never happened before for years but it just occurs recently in 1 minute which causes few stuck messages. Is there anyway to make the Receive Location retries on error? What caused the DTC issue? I notice all the instances of stuck messages having in active running service instances, is it an issue? and how can I fix it?
Error from BizTalk:
There was a failure executing the receive pipeline:
"CargoWise.eHub.Gateway.Pipelines.Rcv_ResolveInboundGatewayMessage,
CargoWise.eHub.Gateway.Pipelines, Version=3.0.0.0, Culture=neutral,
PublicKeyToken=4f570df270576350" Source: "Unknown " Receive Port:
"Gateway_SelectInboxMessageByStatus_CACustoms" URI:
"mssql://ehubtransactions.db.wisegrid.net//eHubTransactions?InboundId=SelectInboxMessagesByStatus&Category=CACustoms"
Reason: 0x8004d00e
Error from SQL Server:
Microsoft Distributed Transaction Coordinator (MS DTC) has stopped this transaction
Error: 8522, Severity: 18, State: 1
It is a WCF-SQL receive port which pulls a message from database and its pipeline will dissemble and add properties to the message. The pipeline components don't access to database. However, there is no error in the query as it will update the db status to 1:
UPDATE top (1) Inbox WITH (ROWLOCK)
SET Status = 1 WHERE PK = #PK
AND Status = 0
SELECT PK,MessageTrackingID,Content
FROM InboxMessage WITH (ROWLOCK)
WHERE PK = #PK and InsertUTC = #InsertUTC
Related
I have a database running on an azure vm with sql server. The db is in full recovery mode. The backup is configured through the web interface. Database and log backups have been working flawlessly for years. But recently the log backup was interrupted halfway through and the log backup process somehow got stuck. The following event has been logged every 5 minutes since then (reading log with managed_backup.sp_get_backup_diagnostics):
[SSMBackup2WAAdminXevent] Database Name = DB, Database ID = 777, Stage =
VerifyJobOutcome, Error Code = 0, Error Message = Warning, Additional Info = A
progress update hasn't been received from SQL Server in more than 30 minutes
for log backup. SSMBackup2WA will continue to wait.
SSMBackup2WA seem to be stuck waiting for a progress update never being received. This has resulted in no log backups being taken. The database backup have continued running without problem.
I have trouble finding the job/task used by SSMBackup2WA. I understand its not in the usual batch of SQL Server Agent jobs but somehow hidden.
My idea is to somehow cancel the existing job that is stuck in waiting loop but I have not figured out how.
I have tried to "reset" the backup process by turning off the backup and then turning it on again but that did not help.
I have no possibility to restart the sql server (and I don't know if that would help).
So since no one seemed to have an answer to this one I resorted to restarting the SQL-server. And after the restart the transaction log backup started working again!
What is interesting is the following log that appeared in the application event log during the restart. It does seem like there was a thread hanging indefinitely, waiting for an status update that never arrived. The restart seems to have taken care of it by killing this status thread and not restarting it again in the erroneous state it had ended up in.
Log Name: Application
Source: Microsoft SQL Server Automated Backup
Date: 1/15/2022 11:16:20 AM
Event ID: 57007
Task Category: None
Level: Warning
Keywords: Classic
User: N/A
Computer: wn-sqlserver1
Description:
[Warning] AutomatedBackupStatusMonitorError:
System.Exception:
Error in auto-backup status monitor thread --->
Microsoft.SqlServer.Management.IaaSAgentSqlQuery.Contract.IaaSAgentSqlQueryException:
A network-related or instance-specific error occurred while
establishing a connection to SQL Server. The server was not
found or was not accessible. Verify that the instance name
is correct and that SQL Server is configured to allow remote
connections. (provider: Named Pipes Provider, error: 40 - Could
not open a connection to SQL Server) --->
I am losing my patience with this problem. I'm running SQL Server 2008 R2 with all updates.
It is very intermittent. Every now and again this SQL server instance will report the error: Login failed for user 'XX'. Reason: Failed to open the database configured in the login object while revalidating the login on the connection.
Error: 18456, Severity: 14, State: 46.
I know this is a good login, it works all of the other times, and no one changes the credentials.
The above error is immediately followed by this error:
The client was unable to reuse a session with SPID 150, which had been
reset for connection pooling. The failure ID is 46. This error may
have been caused by an earlier operation failing. Check the error logs
for failed operations immediately before this error message.
Error: 18456, Severity: 14, State: 46.
When this occurs a few minutes later this is reported also:
Message
All schedulers on Node 0 appear deadlocked due to a large number of worker threads waiting on ASYNC_NETWORK_IO. Process Utilization 0%.
The above message just keeps repeating until I restart the server.
From researching this issue I've increased the maximum worker threads (currently at 1100). The server has 72G memory and 4 vCPUs, running Windows Server 2008 R2.
i don't encounter this issue at all on my other database servers that run the same environment. Just this one. Any help would be greatly appreciated, I'm running out of options.
I am troubleshooting an error with database mail, and when I went to management->database mail to send a test e mail, I get the following erorr:
An exception occured while executing a Transact-SQL statement or batch.
(Microsoft.SQLServer.ConnectionInfo)
Additional information:
Lock request time out period exceeded
The statement has been terminated (Microsoft SQL Server, Error: 1222)
When I investigate this further by looking at all blocking transactions on msdb, I find one transaction, that has the name "implicit transaction". Is this the one which is blocking? What can I do?
Ok I found out what happened by looking at all transactions (right click -> reports -> all transactions). From there I found that a server user had a locking transaction open. After making him close it, everything went fine.
I want to know the SQL Exception error code for the above exception. It occurs when there is an unexpected sql cluster service shutdown, and an application tries to access the server.
These are two separate messages: "SHUTDOWN is in progress" (code 6005) and "Login failed for user '%.*ls'. Only administrators may connect at this time.%.*ls" (code 18451). You can determine this by querying sys.messages:
select * from sys.messages
where [text] like '%only administrators%' or [text] like 'SHUTDOWN is in progress%'
and language_id = 1033
Because "SHUTDOWN is in progress" has severity 10, it's merely informational and will not result in an exception. "Login failed" will, however (and the "SHUTDOWN" message will be incorporated into it).
Note that if your server has already shut down, or is in the process of shutting down and its network name is no longer accessible, you can get other errors during the connection attempt that aren't generated by the server. Looking for this error is not a reliable way of detecting a shutdown condition, and you probably shouldn't have special-case logic for it.
ERROR 6005 - Shutdown is in progress
List of errors in this group: http://www.sql-server-helper.com/error-messages/msg-6001-7000.aspx
Likely related to SQL Server cluster
After a breakdown on my virtual webhost, they finally restored the container, but now I randomly get the error when I try to execute queries. I can do a
Select top 20000 from Orders
But
Select * from Orders (about 100k rows total)
Receives the error in my management studio:
A transport-level error has occurred when receiving results from the server. (provider: Shared Memory Provider, error: 0 - No process is on the other end of the pipe.)
And this error in my error log:
A read operation on a large object failed while sending data to the client. A common cause for this is if the application is running in READ UNCOMMITTED isolation level. This connection will be terminated.
I have googled the error and can see that it could be because "Lightweight pooling" is enabled, but this is not the case here.
I run mssql 2008 express on a windows 2003 server.
We get the same random issue on MSSQL where I work and (for us any way) its down to network issues (too much traffic and such).
This blog post by Michael Aspengren explains the error message "A transport-level error has occurred when sending the request to the server."