I am using DbUp (Documentation) package to maintain and execute scripts on the database.
Currently, I am using
var builder = DeployChanges.To
.SqlDatabase(connectionString)
.WithExecutionTimeout(TimeSpan.FromSeconds(300))
.WithTransactionPerScript()
.WithScriptsFromFileSystem(rootPath, new FileSystemScriptOptions { IncludeSubDirectories = true })
I want to rollback all the transactions if any scripts fails while executing a bunch of scripts.
Using,
WithTransactionAlwaysRollback()
Transaction rolls back even if the all the scripts execute successfully. I only want to rollback on failure.
Is there another way to solve this problem?
Use
.WithTransaction()
instead of
.WithTransactionPerScript()
And here's an implementation of these extension methods:
public static UpgradeEngineBuilder WithTransaction(this UpgradeEngineBuilder builder)
{
builder.Configure(delegate (UpgradeConfiguration c)
{
c.ConnectionManager.TransactionMode = TransactionMode.SingleTransaction;
});
return builder;
}
public static UpgradeEngineBuilder WithTransactionPerScript(this UpgradeEngineBuilder builder)
{
builder.Configure(delegate (UpgradeConfiguration c)
{
c.ConnectionManager.TransactionMode = TransactionMode.TransactionPerScript;
});
return builder;
}
Related
I am developing small POC application to test .NET7 support for distributed transactions since this is pretty important aspect in our workflow.
So far I've been unable to make it work and I'm not sure why. It seems to me either some kind of bug in .NET7 or im missing something.
In short POC is pretty simple, it runs WorkerService which does two things:
Saves into "bussiness database"
Publishes a message on NServiceBus queue which uses MSSQL Transport.
Without Transaction Scope this works fine however, when adding transaction scope I'm asked to turn on support for distributed transactions using:
TransactionManager.ImplicitDistributedTransactions = true;
Executable code in Worker service is as follows:
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
int number = 0;
try
{
while (!stoppingToken.IsCancellationRequested)
{
number = number + 1;
using var transactionScope = TransactionUtils.CreateTransactionScope();
await SaveDummyDataIntoTable2Dapper($"saved {number}").ConfigureAwait(false);
await messageSession.Publish(new MyMessage { Number = number }, stoppingToken)
.ConfigureAwait(false);
_logger.LogInformation("Publishing message {number}", number);
_logger.LogInformation("Worker running at: {time}", DateTimeOffset.Now);
transactionScope.Complete();
_logger.LogInformation("Transaction complete");
await Task.Delay(1000, stoppingToken);
}
}
catch (Exception e)
{
_logger.LogError("Exception: {ex}", e);
throw;
}
}
Transaction scope is created with the following parameters:
public class TransactionUtils
{
public static TransactionScope CreateTransactionScope()
{
var transactionOptions = new TransactionOptions();
transactionOptions.IsolationLevel = IsolationLevel.ReadCommitted;
transactionOptions.Timeout = TransactionManager.MaximumTimeout;
return new TransactionScope(TransactionScopeOption.Required, transactionOptions,TransactionScopeAsyncFlowOption.Enabled);
}
}
Code for saving into database uses simple dapper GenericRepository library:
private async Task SaveDummyDataIntoTable2Dapper(string data)
{
using var scope = ServiceProvider.CreateScope();
var mainTableRepository =
scope.ServiceProvider
.GetRequiredService<MainTableRepository>();
await mainTableRepository.InsertAsync(new MainTable()
{
Data = data,
UpdatedDate = DateTime.Now
});
}
I had to use scope here since repository is scoped and worker is singleton so It cannot be injected directly.
I've tried persistence with EF Core as well same results:
Transaction.Complete() line passes and then when trying to dispose of transaction scope it hangs(sometimes it manages to insert couple of rows then hangs).
Without transaction scope everything works fine
I'm not sure what(if anything) I'm missing here or simply this still does not work in .NET7?
Note that I have MSDTC enable on my machine and im executing this on Windows 10
We've been able to solve this by using the following code.
With this modification DTC is actually invoked correctly and works from within .NET7.
using var transactionScope = TransactionUtils.CreateTransactionScope().EnsureDistributed();
Extension method EnsureDistributed implementation is as follows:
public static TransactionScope EnsureDistributed(this TransactionScope ts)
{
Transaction.Current?.EnlistDurable(DummyEnlistmentNotification.Id, new DummyEnlistmentNotification(),
EnlistmentOptions.None);
return ts;
}
internal class DummyEnlistmentNotification : IEnlistmentNotification
{
internal static readonly Guid Id = new("8d952615-7f67-4579-94fa-5c36f0c61478");
public void Prepare(PreparingEnlistment preparingEnlistment)
{
preparingEnlistment.Prepared();
}
public void Commit(Enlistment enlistment)
{
enlistment.Done();
}
public void Rollback(Enlistment enlistment)
{
enlistment.Done();
}
public void InDoubt(Enlistment enlistment)
{
enlistment.Done();
}
This is 10year old code snippet yet it works(im guessing because .NET Core merely copied and refactored the code from .NET for DistributedTransactions, which also copied bugs).
What it does it creates Distributed transaction right away rather than creating LTM transaction then promoting it to DTC if required.
More details explanation can be found here:
https://www.davidboike.dev/2010/04/forcibly-creating-a-distributed-net-transaction/
https://github.com/davybrion/companysite-dotnet/blob/master/content/blog/2010-03-msdtc-woes-with-nservicebus-and-nhibernate.md
Ensure you're using Microsoft.Data.SqlClient +v5.1
Replace all "usings" System.Data.SqlClient > Microsoft.Data.SqlClient
Ensure ImplicitDistributedTransactions is set True:
TransactionManager.ImplicitDistributedTransactions = true;
using (var ts = new TransactionScope(your options))
{
TransactionInterop.GetTransmitterPropagationToken(Transaction.Current);
... your code ..
ts.Complete();
}
I've set up a Unit Test project in Visual studio for my SQL Server project.
The test itself work, and the setup includes deploying that database. My problem is that I want to have a "clean slate" test, and every time I run a test, the data accumulates.
I tried manually calling a 'DROP DATABASE' from the SqlDatabaseSetup.cs, but it seems that I don't have a DataConnection at this point.
[TestClass()]
public class SqlDatabaseSetup
{
[AssemblyInitialize()]
public static void InitializeAssembly(TestContext ctx)
{
var q = ctx.DataConnection.CreateCommand();
q.CommandText = "DROP DATABASE MyDb;";
q.ExecuteNonQuery();
// Setup the test database based on setting in the
// configuration file
SqlDatabaseTestClass.TestService.DeployDatabaseProject();
SqlDatabaseTestClass.TestService.GenerateData();
}
}
Is there anyway to indicate that the DB should be flushed first (without manually calling DELETE FROM XX for every table), or is there a way I can pass through a command to do that for me?
The solution was as follows :
[AssemblyInitialize()]
public static void InitializeAssembly(TestContext ctx)
{
// Setup the test database based on setting in the
// configuration file
try
{
var conn = SqlDatabaseTestClass.TestService.OpenExecutionContext();
var cmd = conn.Connection.CreateCommand();
cmd.CommandText = "use master; ALTER DATABASE [MyTestDb] SET SINGLE_USER WITH ROLLBACK IMMEDIATE;DROP DATABASE MyTestDb;";
cmd.ExecuteNonQuery();
}
catch
{
}
SqlDatabaseTestClass.TestService.DeployDatabaseProject();
SqlDatabaseTestClass.TestService.GenerateData();
}
I can send logs to log collectors from C# applications using log4net+GELF appender.
But how to send logs to GrayLog from T-SQL procedures?
There are code:
WinForms app works fine, I run it on the same machine where SQL Server installed. I see all logs received in GrayLog:
private void button1_Click(object sender, EventArgs e)
{
string facility = "DoBeDo";
string host = "my-host-name";
int port = 12201;
try
{
using (var logger = new GrayLogUdpClient(facility, host, port))
{
logger.Send("Hello", "Jonny Holiday", new { Username = "John", Email = "jonny#example.com" });
}
}
catch(Exception xx)
{
Console.WriteLine("***Exception:{0}", xx.Message);
}
}
There is SQLCLR code, it works, I see messages in SSMS but no any records in GrayLog and no any exceptions:
public partial class StoredProcedures
{
[Microsoft.SqlServer.Server.SqlProcedure]
public static void SqlSPHelper(SqlString msg)
{
try
{
SqlContext.Pipe.Send(#"SqlSPHelper:: Start");
string facility = "DoBeDo";
string host = "my-host-name";
int port = 12201;
try
{
using (var logger = new GrayLogUdpClient(facility, host, port))
{
logger.Send("Hey", "Donny Hooligan", new { Username = "Donald", Email = "Donny#example.com" });
}
}
catch (Exception xx)
{
Console.WriteLine("***Exception:{0}", xx.Message);
}
}
catch(Exception xx)
{
SqlContext.Pipe.Send("1:"+xx.Message);
}
SqlContext.Pipe.Send(#"SqlSPHelper:: Completed");
}
What is wrong? How to send logs to GrayLog?
I think you can do this in couple of ways:
Define a CLR stored procedure, which does the logging part
In the CATCH Block of the stored procedure, RAISERROR WITH LOG to write to Windows event viewer & SQL Server error log. Later you can filter and read these events from event viewer in your GrayLog.
The first method is better, as it is cleaner. Second method makes the SQL Server error log to have many log entries, which would cause false alarms for DBA team.
I'm having trouble with transactions in Spring Boot using #Transactional annotation. The latest Spring is connected to a MS SQL Database.
I have following service, which periodically executes transactional method according to some criteria:
#Service
public class SomeService {
SomeRepository repository;
public SomeService(SomeRepository someRepository) {
this.repository = someRepository;
}
#Scheduled(fixedDelayString="${property}") //10 seconds
protected scheduledIteration() {
if(something) {
insertDataInNewTransaction(getSomeData());
}
}
#Transactional(propagation = Propagation.REQUIRED, rollbackFor = Exception.class)
protected void insertDataInNewTransaction(List<Data> data) {
//insert data to db
repository.saveAll(data);
//call verify proc
repository.verifyData();
}
}
The algorithm supposed to process data, insert them into table and perform check (db procedure). If the procedure throws an exception, the transaction should be rollbacked. I'm sure, that the procedure does not perform commit of the transaction.
The problem I'm facing is, that calling the method does not begin new transaction (or does but it's auto-commited), because I've tried following:
#Transactional(propagation = Propagation.REQUIRED, rollbackFor = Exception.class)
protected void insertDataInNewTransaction(List<Data> data) {
int counter = 0;
for(Data d : data) {
repository.save(d);
counter++;
//test
if(counter == 10) {
throw new Exception("test");
}
}
}
After the test method is executed, the first 10 rows remain in the table, where they were supposed to be rollbacked. During debugging I've noticed, that calling repository.save() in the loop inserts to the table outside transaction, because I can see the row from DB IDE while debugger sitting on next row. This gave me an idea, that the problem is caused by auto-commit, as it's MS SQL default. So I have tried to add following properties, but without any difference:
spring.datasource.hikari.auto-commit=false
spring.datasource.auto-commit=false
Is there anything I'm doing wrong?
If you use Spring Proxy AOP, then you need to turn the method insertDataInNewTransaction as public.
Remember that if the method is public, but it is invoked from the same bean, it will not create a new transaction (because spring proxies won't be call).
Short answer:
#Transactional(propagation = Propagation.REQUIRED, rollbackFor = Exception.class)
public void insertDataInNewTransaction(List<Data> data) {
//insert data to db
repository.saveAll(data);
//call verify proc
repository.verifyData();
}
But if you really need a new separate transaction use Propagation.REQUIRES_NEW instead of Propagation.REQUIRED.
I'm having trouble with a rather complex save operation inside a ServiceStack service.
To simplify the explanation the service starts an Ormlite transaction and within it calls another service through ResolveService:
public ApplicationModel Post(ApplicationModel request)
{
using (IDbTransaction tr = Db.OpenTransaction())
{
using (var cases = ResolveService<CaseService>())
{
request.Case = cases.Post(request.Case);
}
}
Db.Save<Application>(request.Application, true);
}
The other service (CaseService) uses also a transaction to perform its logic:
public CaseModel Post(CaseModel request)
{
using (IDbTransaction tr = Db.OpenTransaction())
{
Db.Insert<Case>(request);
Db.SaveAllReferences<CaseModel>(request);
}
}
In a similar situation with higher hierarchy of services calling other services a "Timeout expired" error is thrown, and so far I've not been able to resolve, although I closely monitored the SQL Server for deadlocks.
My question is whether this is the right way of using/sharing Ormlite transactions across services or there is another mechanism?
Thanks in advance.
You shouldn't have nested transactions, rather than calling across services to perform DB operations you should extract shared logic out either using a separate shared Repository or re-usable extension methods:
public static class DbExtensions
{
public static void SaveCaseModel(this IDbConnection db,
CaseModel case)
{
db.Insert<Case>(case);
db.SaveAllReferences<CaseModel>(case);
}
}
Then your Services can maintain their own transactions whilst being able to share logic, e.g:
public ApplicationModel Post(ApplicationModel request)
{
using (var trans = Db.OpenTransaction())
{
Db.SaveCaseModel(request.Case);
Db.Save<Application>(request.Application, true);
trans.Commit();
}
}
public CaseModel Post(CaseModel request)
{
using (var trans = Db.OpenTransaction())
{
Db.SaveCaseModel(request);
trans.Commit();
}
}