Consider this scenario.
Coordinator sends prepare messages to 2 participants, and crashes
Participants lock resources successfully, and then wait for the coordinator to recover
Coordinator recovers, but did not receive messages from participants about prepare_success messages
Is manual intervention required to unlock the locked resources? Or do the participants poll the coordinator to find the status of the transaction?
At the outset, this sounds similar to the case when a participant does not receive a commit message, but the main difference is that the coordinator redrives the messages in that scenario. In the scenario listed above, the coordinator does not even know that it has to redrive a global transaction because no record of it was made in it's log.
I can put here some details on how Narayana works. The strategy on XA recovery could be various depending on the transaction manager implementation.
The scenario which you talks about is managed in Narayana by a procedure named orphan detection.
As you pointed down the Narayana transaction manager crashes before the prepare phase was ended thus there is no information about transaction existence in the Narayana log. Here the requirement is that Narayana configuration has to know all possible participants. In case of WildFly it's ensured by definition of datasources or resource managers in standalone.xml. The recovery process asks all the available resources with XAResource.recover call ( https://docs.oracle.com/javase/7/docs/api/javax/transaction/xa/XAResource.html#recover(int) ). Resource returns Xid of all in-doubt transactions it is aware of.
The Xid was constructed by Narayana (passed to resource during prepare and saved in resource txn log, during recovery returned back to Narayana) and contains transaction manager id (https://wildscribe.github.io/WildFly/11.0.CR1/subsystem/transactions/index.html -> node-identifier). Narayana checks if the Xid belongs to the current Narayana instance (node identifiers matches). If so and there is no notion about the Xid in the Narayana transaction log, based on 2PC presumed abort optimization (https://access.redhat.com/documentation/en-us/red_hat_jboss_enterprise_application_platform/7.0/html/development_guide/java_transaction_api_jta#about_the_presumed_abort_optimization), it finally asks the resource to rollback. That effectively remove locks.
Related
Is any possible way to find which process caused a deadlock last week in SQL Server? I checked Extended Events that only keeps data from last night. If anyone can share a script to find out would be helpful.
The job has completed with an error - Database server name: ################- Transaction (Process ID 279) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
System.Data.SqlClient.SqlException (0x80131904): Transaction (Process ID 279) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
It is possible to retrieve deadlocks details from Extended Events -> Sessions -> System_health -> right click on package0.event_file and choose view target data, next you need to sort by name to see the deadlock report in XML form and also as graphs. But for this you need to enable trace 1204 and 1222 prior to the deadlocks occurrence.
These are the trace that you need to enable.
DBCC TRACEON (1204,-1)
DBCC TRACEON (1222,-1)
Click here for more details...
I am currently using tMomInput to read from Active MQ.
Is there any way in Talend that we can read from a Queue in Active MQ without deleting? Is it possible to delete the contents only when they are successfully copied over to a temporary table? If there is a failure such as server shut down and the job fails before copying them to the DB table then there will be no way of recovering the data.
The general way to do this in JMS is to use transactions. This way the message is only deleted from the queue when the transaction is committed in the success case. In the error case the transaction is rolled back and the message is given back to the jms server for retransmit.
We have a process that was running for 4 hours. Because it was running so long, it was causing other issues in the database, so it was decided to kill the process.
Now, the process is in a suspended state. It also states that it's being blocked by itself after querying sp_who2.
In activity monitor, here's the waitresource information:
objectlock lockPartition=0 objid=xxx subresource=FULL dbid=2 id=lockyyyy mode=X associatedObjectid=xxx
You'll notice that the objid and associatedObjectId are the same value.
Querying the sys.objects table shows NO results for that object id.
Is Sql Server waiting for a lock on an object that doesn't exist anymore? How can I get rid of this process without restarting Sql Server? (our DBA's are not responding to help requests).
Keep in mind, this is a test environment, but it is stopping all development/testing because we are unable to deploy any changes to our database, because one of those changes is affecting one of the objects that the process was accessing.
Edit: more info from activity monitor:
Command = 'KILLED/ROLLBACK'
TASK STATE = 'SUSPENDED'
I have experienced this may times. When you kill a large INSERT/UDPATE/DELETE statement, it can take hours to recover (if it ever does recover) from this state.
run kill <spid> with statusonly.
It will give you a percentage and estimated wait time of the ROLLBACK process.
Sometimes it says 0% or 100% and 0 estimated time. If you are patient, it may recover eventually. If you restart the server, the rollback process will be completed offline, and the database will show IN RECOVERY state and usually will be faster than waiting the server to recover itself.
Be aware that users won't be able to use the database until the recovery process ends, but if the SPID in KILLED/ROLLBACK state is locking other process, it might be an option to restart.
Well, this seems to be lock due to parallel processing inside the tempdb.
You can try kill [processid] if you have the rights to?
Another way is to get more detailed process information with this:
SELECT * FROM sys.sysprocesses WHERE spid = YOURSPID
As the Process runs in DB:2 try this:
SELECT * FROM tempdb.sys.all_objects WHERE object_id = OBJECTID
As I've seen, you have edited your question. If the Spid is in KILLED/ROLLBACK you have to wait until your transaction is rolled back. After that the process will be killed and removed. You can't do anything else, as the transaction security must be given.
Can jdbc connections which are closed due to database un-availability be recovered.
To give back ground I get following errors in sequence. It doesn't look to be manual re-start. The reason for my question is that I am told that the app behaved correctly without
the re-start. So if the connection was lost, can it be recovered, after a DB re-start.
java.sql.SQLException: ORA-12537: TNS:connection closed
java.sql.SQLRecoverableException: ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
IBM AIX RISC System/6000 Error: 2: No such file or directory
java.sql.SQLRecoverableException: ORA-01033: ORACLE initialization or shutdown in progress
No. The connection is "dead". Create a new connection.
A good approach is to use a connection pool, which will test if the connection is still OK before giving it to you, and automatically create a new connection if needed.
There are several open source connection pools to use. I've used Apache's JDCP, and it worked for me.
Edited:
Given that you want to wait until the database comes back up if it's down (interesting idea), you could implement a custom version of getConnection() that "waits a while and tries again" if the database doesn't respond.
p.s. I like this idea!
The connection cannot be recovered. What can be done is to failover the connection to another database instance. RAC and data guard installations support this configuration.
This is no problem for read-only transactions. However for transactions that execute DML this can be a problem, especially if the last call to the DB was a commit. In case of a commit the client cannot tell if the commit call completed or not. When did the DB fail; before executing the commit, or after executing the commit (but not sending back the acknowledgment to the client). Only the application has this logic and can do the right thing. If the application after failing over does not verify the state of the last transaction, duplicate transactions are possible. This is a known problem and most of us experienced it buying tickets or similar web transactions.
I'd like to use SQL Server 2008 Service Broker to log the progress of a long-running (up to about 30 minutes) transaction that is dynamically created by a stored procedure. I have two goals:
1) To get real-time logging of the dynamically-created statements that make up the transaction so that the progress of the transaction can be monitored remotely,
2) To be able to review the steps that made up the transaction up to a point where a failure may have occurred requiring a rollback.
I cannot simply PRINT (or RAISERROR(msg,0,0)) to the console because I want to log the progress messages to a table (and have that log remain even if the stored procedure rollsback).
But my understanding is that messages cannot be received from the queue until the sending thread commits (the outer transaction). Is this true? If so, what options do I have?
It is true that you cannot read messages from the service queue until the transaction is committed.
You could try some other methods:
use a sql clr procedure to send a .net remoting message to a .net app that receives the messages and them log them.
use a sql clr procedure to write a text or other log file to disk.
Some other method...
Regards
AJ