I have console c# program that is accessing a database.
Part of the code is doing some inserts and updates that I want to control with a transaction. This is the part of the code that is handling the business logic.
Another part of the code is doing some inserts and updates that is more system support logic that I want to commit immediately on insert. Specifically this code is inserting a row when the program starts and updating the row with the end. It also logs certain events in the code. I don't want these logged events to go away just because something failed in the business logic.
I tried to do the business logic with SqlCommand like this:
SqlCommand command = new SqlCommand(sql, connection, transaction);
and system logic like this:
SqlCommand command = new SqlCommand(sql, connection);
But I get this error:
ExecuteNonQuery requires the command to have a transaction when
connection assigned to the command is in a pending local transaction.
My goal is to have the business logic commit only upon transaction.Commit() and the system logic to commit immediately.
Can I accomplish that with two separate transactions?
Will I need to open two different connections?
In the end I created two different Connections: one for transaction based IO and another for non-transactions based IO.
I then used code that looked like this to set the transaction on each command:
transaction = new connection.BeginTransaction();
someCommand1.Transaction = transaction;
someCommand2.Transaction = transaction;
Where someCommand1/2 are SqlCommand()s previously created in the code.
This avoids the problem of calling SqlCommand multiple times.
Turned out to be very clean in the code.
Related
I have a database where data is processed in some kind of batches, where each batch may contain even a million records. I am processing data in a console application, and when I'm done with a batch, I mark it as Done (to avoid reading it again in case it does not get deleted), delete it and move on to a next batch.
I have the following simple stored procedure which deletes processed "batches" of data
CREATE PROCEDURE [dbo].[DeleteBatch]
(
#BatchId bigint
)
AS
SET XACT_ABORT ON
BEGIN TRANSACTION
DELETE FROM table1 WHERE BatchId = #BatchId
DELETE FROM table2 WHERE BatchId = #BatchId
DELETE FROM table3 WHERE BatchId = #BatchId
COMMIT
RETURN ##Error
I am using NHibernate with command timeout value 10 minutes, and the DeleteBatch procedure call times out occasionally.
Actually I don't want to wait for DeleteBatch to complete. I already have marked the batch as Done, so I want to go processing a next batch or maybe even exit my console application, if there are no more pending batches.
I am using Microsoft SQL Express 2012.
Is there any simple solution to tell the SQL server - "launch DeleteBatch and run it asynchronously even if I disconnect, and I don't even need the result of the procedure"?
It would also be great if I could set a lower processing priority for DeleteBatch because other queries are more important than DeleteBatch.
I dont know much about NHibernate. But if you were or can use ADO.NET in this scenario then you can implement asynchronous database operations easliy using the SqlCommand.BeginExecuteNonQuery Method in C#. This method starts the process of asynchronously executing a Transact-SQL statement or stored procedure that does not return rows, so that other tasks can run concurrently while the statement is executing.
EDIT: If you really want to exit from your console app before the db operation ends then you will have to manually create threads in your code and perform the db operation in those threads. Now when you close your console app these threads would still be alive because Threads created using System.Thread.Thread are foreground threads by default. But having said that it is also important to consider how many threads you will create. In your case you would have to assign 1 thread for each batch. If number of batches is very large then large number of threads would need to be created which would inturn eat a large amount of your CPU resources and would even freeze your OS for a long time.
Another simple solution I could suggest is to insert the BatchIds into some database table. Create an INSERT TRIGGER on that table. This trigger would then call a stored proc with BatchId as its parameter and would perform the required tasks.
Hope it helps.
What if your console application were, instead of trying to delete the batch, just write the batch id into a "BatchIdsToDelete" table. Then, you could use an agent job running every x minutes/seconds or whatever, to delete the top x percent records for a given batch id, and maybe sleeping a little before tackling the next x percent.
Maybe worth having a look at that?
Look at this article which explains how to do reliable asynchronous procedure execution, code included. IS based on Service Broker.
the problem with trying to use .NEt async features (like BeginExecute, or task etc) is that the call is unreliable: if the process exits before the procedure completes the execution is canceled in the server as the session is disconnected.
But you need to also look at the task itself, why is the deletion taking +10 minutes? is it blocked by contention? are you missing indexes on BatchId? Use the Performance Troubleshooting Flowchart.
Late to the party, but if someone else has this problem use SQLCMD. With express you are limited in the number of users (I think 2, but it may have changed since I the last time I did much with express). You can have sqlcmd, run queries, stored procedures ...
And you can kick off the sqlcmd with Windows Scheduler. A script, an outlook rule ...
I used it to manage like 3 or 4 thousand SQL Server Express instances, with their nightly maintenance scheduled with the Windows Scheduler.
You could also create and run a PowerShell script, it's more versatile and probably a more widely used than sqlcmd.
I needed a same thing..
After searching for long time I found the solution
Its d easiest way
SqlConnection connection = new SqlConnection();
connection.ConnectionString = "your connection string";
SqlConnectionStringBuilder builder = new SqlConnectionStringBuilder(connection.ConnectionString);
builder.AsynchronousProcessing = true;
SqlConnection newSqlConn = new SqlConnection(builder.ConnectionString);
newSqlConn.Open();
SqlCommand cmd = new SqlCommand(storeProcedureName, newSqlConn);
cmd.CommandType = CommandType.StoredProcedure;
cmd.BeginExecuteNonQuery(null, null);
Ideally SQLConnection object should take an optional parameter / property, URL of a web service, be that WCF or WebApi, or something yet to be named, and if the user wishes to, notify user of execution advance and / or completion status by calling this URL with well known message.
Theoretically DBConnection is extensible object one is free to implement. However, it will take some review of what really can be and needs to be done, before this approach can be said feasible.
Suppose you set up a TransactionScope object as illustrated per the Microsoft example here. Now suppose that you need to update a lot of database tables, and you want them all in the scope of the TransactionScope object. Continually nesting SqlConnection and SqlCommand objects 10 deep will create a source code mess. If instead you call other functions which create connections (in your data access layer, for example), will they be within scope of the TransactionScope object?
Example:
' Assume variable "x" is a business object declared and populated with data.
Using scope As New TransactionScope()
Dal.Foo.SaveProducts(x.Products)
Dal.Foo.SaveCustomer(x.Customer)
Dal.Foo.SaveDetails(x.Details)
' more DAL calls ...
Dal.Foo.SaveSomethingElse(x.SomethingElse)
scope.Complete()
End Using
Assume that each DAL function contains its own using statements for connections. Example:
Public Shared Sub SaveProducts(x As Object)
Using conn As New SqlConnection("connection string")
Using cmd As New SqlCommand("stored procedure name", conn)
With cmd
' etc.
End With
End Using
End Using
End Sub
Yes, they will be inside the TransactionScope. What the TransactionScope basically does is to create a Transaction object and set Transaction.Current to that.
In other words, this:
Using scope As New TransactionScope()
... blah blah blah ...
End Using
is basically the same as this:
try
{
// Transaction.Current is a thread-static field
Transaction.Current = new CommittableTransaction();
... blah blah blah ...
}
finally
{
Transaction.Current.Commit(); // or Rollback(), depending on whether the scope was completed
Transaction.Current = null;
}
When a SqlConnection is opened, it checks if Transaction.Current (on this thread) is null or not, and if it is not null then it enlists (unless enlist=false in the connection string). So this means that SqlConnection.Open() doesn't know or care if the TransactionScope was opened in this method or a method that called this one.
(Note that if you wanted the SqlConnection in the child methods to NOT be in a transaction, you can make an inner TransactionScope with TransactionScopeOption.Suppress)
When you create a TransactionScope, all connections you open while the TransactionScope exists join the transaction automatically (they're 'auto enlisted'). So you don't need to pass connection strings around.
You may still want to, when SQL Server sees different transactions (even if they are all contained by one DTC transaction), it doesn't share locks between them. If you open too many connections and do a lot of reading and writing, you're headed for a deadlock.
Why not put the active connection in some global place and use it?
Some more info after some research. Read this: TransactionScope automatically escalating to MSDTC on some machines? .
If you're using SQL Server 2008 (and probably 2012, but not any other database), some magic is done behind the scenes, and if you open two SQL Connections one after the other, they are going to be united into a single SQL transaction, and you're not going to have any locking problem.
However, if you're using a different database, or you may open two connections concurrently, you will get a DTC transaction, which means SQL Server will not manage the locks properly, and you may encounter very unpleasant and unexpected deadlocks.
While it's easy to make sure you're only running on SQL Server 2008, making sure you don't open two connections at the same time is a bit harder. It's very easy to forget it and do something like this:
class MyPersistentObject
{
public void Persist()
{
using(SQLConnection conn=...)
{
conn.Open()
WriteOurStuff()
foreach(var child in this.PersistedChildren)
child.Persist()
WriteLogMessage()
}
}
}
If the child's Persist method opens another connection, your transaction is escalated into a DTC transaction and you're facing potential locking issues.
So I still suggest maintaining the connection in one place and using it through your DAL. It doesn't have to be a simple global static variable, you can create a simple ConnectionManager class with a ConnectionManager.Current property which will hold the current connection. Make ConnectionManager.Current as [ThreadStatic] and you solved most of your potential problems. That's exactly how the TransactionScope works behind the scenes.
e.g. When deadlock occurs, following SQL commands are successfully executed, even if they have assigned SQL transaction that is after rollback. It seems, it is caused by a new implicit transaction that is created on SQL Server.
Someone could expect that ADO.NET would throw an exception that the commands are being executed on a zombie transaction. However, such exception is not thrown. (I think this is a bug in ASP.NET.) Moreover, because of zombie transaction the final Dispose() silently ignores the rollback.
Any ideas, how can I ensure that nobody can execute commands on implicit transaction?
Or, how to check that transaction is zombie? I found that Commit() and Rollback() check for zombie transaction, however I can call them for a test:)
I also found that also reading IsolationLevel will do the check, but I am not sure whether simple calling transaction.IsolationLevel.ToString(); will not be removed by a future optimizer. Or do you know any other safe way invoke a getter (without using reflection or IL emitting)?
EDIT: Remus Rusanu pointed out that this situation would not normally occur. Yes, that's true. It usually occurs when there is a bug in a code. In our case, there was some logging routine in a finally statement that was trying to store the failure into the database. Now, I am trying to find a solution how to detect such bugs in a future. Since these bugs are difficult to test. If ADO.NET would check that the provided transaction is zombie, this bug will be found much more easily. I found two possibilities:
Turn off creation of implicit transactions - I am not sure whether this is possible.
Ensure that before executing any commands, check for zombie transaction will run.
What you describe doesn't exists. A transaction rollback will throw a very visible exception in your application. for some reason, I much rather believe your code captures the exception and silently ignores it, continuing to execute statements after the transaction had rolled back.
Probably not directly related to your problem, as it was caused by a bug, but still might be of interest. Not all errors cause a transaction to rollback, so sometimes a transaction might be "partially successful" - some statements errored while others compeleted fine.
There is an option SET XACT_ABORT ON that makes the server abort transaction on any error.
Considering your question, you cannot turn off implicit transactions (if you execute an SQL statement, an implicit transaction will be created unless another transaction is already active). So you just have to handle the errors correcty to make sure that a transaction is there when you need it.
Have a look at TransactionScope class, you can use it to avoid managing these transactions in your code.
//Based upon your description I'm guessing that your code effectively does this
SqlConnection conn = new SqlConnection("ConnectionString");
SqlCommand cmd = new SqlCommand("insert into ....");
cmd.Connection = conn;
conn.Open();
SqlTransaction tran = conn.BeginTransaction();
cmd.Transaction = tran;
tran.Rollback(); //or tran.Dispose();
cmd.ExecuteNonQuery();
This causes the cmd to be executed outside the scope of a transaction.
Removing the line cmd.Connection = conn; will achieve the behavior that I think you're looking for (e.g. the command will fail because the transaction is no longer valid.)
SqlConnection conn = new SqlConnection("ConnectionString");
SqlCommand cmd = new SqlCommand("insert into ....");
//cmd.Connection = conn;
conn.Open();
SqlTransaction tran = conn.BeginTransaction();
cmd.Connection = tran.Connection;
cmd.Transaction = tran;
tran.Rollback(); //or tran.Dispose();
cmd.ExecuteNonQuery();
Let's assume we have a bunch of LINQ2SQL InsertOnSubmit statements against a given DataContext. If the SubmitChanges call is successful, is there any way to automatically generate a list of SQL commands (or even LINQ2SQL statements) that could undo everything that was submitted at a later time? It's like executing a rollback even though everything worked as expected.
Note: The destination database will either be Oracle or SQL Server, so if there is specific functionality for both databases that will achieve this, I'm happy to use that as well.
Clarification:
I do not want the "rollback" to happen automatically as soon as the inserts have succesfully completed. I want to have the ability to "undo" the INSERT statements via DELETE (or some other means) up to 24 hours (for example) after the original program finished inserting data. We can ignore any possible referential integrity issues that may come up.
Assume a Table A with two columns: Id (autogenerated unique id) and Value (string)
If the LINQ2SQL code performs two inserts
INSERT INTO Table A VALUES('a') // Creates new row with Id = 1
INSERT INTO Table A VALUES('z') // Creates new row with Id = 2
<< time passes>>
At some point later I would want to be able "undo" this by executing
DELETE FROM A Where Id = 1
DELETE FROM A Where Id = 2
or something similar. I want to be able to generate the DELETE statements to match the INSERT ones. Or use some functionality that would let me capture a transaction and perform a rollback later.
We cannot just 'reset the database' to a certain point in time either as other changes not initiated by our program could have taken place since.
It is actually quite easy to do this, because you can pass in a SqlConnection into the LINQ to SQL DataContext on construction. Just run this connection in a transaction and roll that transaction back as soon as you're done.
Here's an example:
string output;
using (var connection = new SqlConnection("your conn.string"))
{
connection.Open();
using (var transaction = connection.StartTransaction())
{
using (var context = new YourDataContext(connection))
{
// This next line is needed in .NET 3.5.
context.Transaction = transaction;
var writer = new StringWriter();
context.Log = writer;
// *** Do your stuff here ***
context.SubmitChanges();
output = writer.ToString();
}
transaction.Rollback();
}
}
I am always required to provide a RollBack script to our QA team for testing before any change script can be executed in PROD.
Example: Files are sent externally with a bunch of mappings between us, the recipient and other third parties. One of these third parties wants to change, on an agreed date, the mappings between the three of us.
Exec script would maybe update some exisiting, delete some now redundant and insert some new records - scope_identity used in subsequent relational setup etc etc.
If, for some reason, after we have all executed our changes and the file transport is fired up, just like in UAT, we see some errors not encountered in UAT, we might multilaterally make the decision to roll back the changes. Hence the roll back script.
SQL has this info when you BEGIN TRAN until you COMMIT TRAN or ROLLBACK TRAN. I guess your question is the same as mine - can you output that info as a script.
Why do you need this?
Maybe you should explore the flashback possibilities of Oracle. It makes it possible to travel back in time.
It makes it possible to reset the content of a table or a database to how it once was at a specific moment in time (or at a specific system change number).
See: http://www.oracle.com/technology/deploy/availability/htdocs/Flashback_Overview.htm
We have an application that was originally written as a desktop app, lo these many years ago. It starts a transaction whenever you open an edit screen, and commits if you click OK, or rolls back if you click Cancel. This worked okay for a desktop app, but now we're trying to move to ADO.NET and SQL Server, and the long-running transactions are problematic.
I found that we'll have a problem when multiple users are all trying to edit (different subsets of) the same table at the same time. In our old database, each user's transaction would acquire record-level locks to every record they modified during their transaction; since different users were editing different records, everyone gets their own locks and everything works. But in SQL Server, as soon as one user edits a record inside a transaction, SQL Server appears to get a lock on the entire table. When a second user tries to edit a different record in the same table, the second user's app simply locks up, because the SqlConnection blocks until the first user either commits or rolls back.
I'm aware that long-running transactions are bad, and I know that the best solution would be to change these screens so that they no longer keep transactions open for a long time. But since that would mean some invasive and risky changes, I also want to research whether there's a way to get this code up and running as-is, just so I know what my options are.
How can I get two different users' transactions in SQL Server to lock individual records instead of the entire table?
Here's a quick-and-dirty console app that illustrates the issue. I've created a database called "test1", with one table called "Values" that just has ID (int) and Value (nvarchar) columns. If you run the app, it asks for an ID to modify, starts a transaction, modifies that record, and then leaves the transaction open until you press ENTER. I want to be able to
start the program and tell it to update ID 1;
let it get its transaction and modify the record;
start a second copy of the program and tell it to update ID 2;
have it able to update (and commit) while the first app's transaction is still open.
Currently it freezes at step 4, until I go back to the first copy of the app and close it or press ENTER so it commits. The call to command.ExecuteNonQuery blocks until the first connection is closed.
public static void Main()
{
Console.Write("ID to update: ");
var id = int.Parse(Console.ReadLine());
Console.WriteLine("Starting transaction");
using (var scope = new TransactionScope())
using (var connection = new SqlConnection(#"Data Source=localhost\sqlexpress;Initial Catalog=test1;Integrated Security=True"))
{
connection.Open();
var command = connection.CreateCommand();
command.CommandText = "UPDATE [Values] SET Value = 'Value' WHERE ID = " + id;
Console.WriteLine("Updating record");
command.ExecuteNonQuery();
Console.Write("Press ENTER to end transaction: ");
Console.ReadLine();
scope.Complete();
}
}
Here are some things I've already tried, with no change in behavior:
Changing the transaction isolation level to "read uncommitted"
Specifying a "WITH (ROWLOCK)" on the UPDATE statement
Just checking, but do you have a primary key or unique index on the ID column?
Look into optimistic versus pessimistic locking.
Edit:
Previous article linked to classic ado...sorry.
http://msdn.microsoft.com/en-us/library/cs6hb8k4(VS.71).aspx
Probably the index was created with row locks set to "off".
"WITH (ROWLOCK)" in a query would have no effect in that case.
You can turn them back on with ALTER INDEX, e.g.:
ALTER INDEX [PK_Values] ON [Values] SET (ALLOW_ROW_LOCKS = ON)