Can database Snapshots be passed to other methods? - database

I have tried to write this code in three different ways but none of them work correctly.
I did not want to write any business logic code in the class that opens and closes the database, so I have a database class. I am trying to take a Snapshot of the database to send it to the business logic class.
However what is the correct way to send a Snapshot or Iterator across to another class?
Here is my current code in the database connection class trying to return an Iterator to the business logic class:
public synchronized DBIterator getSnapShot() throws IOException {
ReadOptions ro = new ReadOptions();
Options options = new Options();
DBIterator iterator = null;
try {
DB database = (factory.open(new File("myFilePath"), options));
ro.snapshot(database.getSnapshot());
iterator = database.iterator(ro);
return iterator;
} catch (Exception e) {
e.printStackTrace();
} finally {
getLinkPacDatabase().close();
}
return iterator;
}
Then I try to iterate in the business logic class like this:
for (iterator.seekToFirst(); iterator.hasNext(); iterator.next()) {
I have also tried to grab a Snapshot only from the database like this, but it also does not work because the database connection has to close obviously:
public synchronized Snapshot generateSnapshot() {
Snapshot snapshot = null;
DB database = null;
try {
database = factory.open(new File("myFilePath"), options);
snapshot = database.getSnapshot();
} catch (Exception e) {
e.printStackTrace();
} finally {
try {
database.close();
} catch (IOException e) {
e.printStackTrace();
}
}
return snapshot;
}
There is no documentation for the levelDb Java port library I am using, therefore I am hoping someone can explain if I am using Snapshots incorrectly.

Related

.NET 7 Distributed Transactions issues

I am developing small POC application to test .NET7 support for distributed transactions since this is pretty important aspect in our workflow.
So far I've been unable to make it work and I'm not sure why. It seems to me either some kind of bug in .NET7 or im missing something.
In short POC is pretty simple, it runs WorkerService which does two things:
Saves into "bussiness database"
Publishes a message on NServiceBus queue which uses MSSQL Transport.
Without Transaction Scope this works fine however, when adding transaction scope I'm asked to turn on support for distributed transactions using:
TransactionManager.ImplicitDistributedTransactions = true;
Executable code in Worker service is as follows:
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
int number = 0;
try
{
while (!stoppingToken.IsCancellationRequested)
{
number = number + 1;
using var transactionScope = TransactionUtils.CreateTransactionScope();
await SaveDummyDataIntoTable2Dapper($"saved {number}").ConfigureAwait(false);
await messageSession.Publish(new MyMessage { Number = number }, stoppingToken)
.ConfigureAwait(false);
_logger.LogInformation("Publishing message {number}", number);
_logger.LogInformation("Worker running at: {time}", DateTimeOffset.Now);
transactionScope.Complete();
_logger.LogInformation("Transaction complete");
await Task.Delay(1000, stoppingToken);
}
}
catch (Exception e)
{
_logger.LogError("Exception: {ex}", e);
throw;
}
}
Transaction scope is created with the following parameters:
public class TransactionUtils
{
public static TransactionScope CreateTransactionScope()
{
var transactionOptions = new TransactionOptions();
transactionOptions.IsolationLevel = IsolationLevel.ReadCommitted;
transactionOptions.Timeout = TransactionManager.MaximumTimeout;
return new TransactionScope(TransactionScopeOption.Required, transactionOptions,TransactionScopeAsyncFlowOption.Enabled);
}
}
Code for saving into database uses simple dapper GenericRepository library:
private async Task SaveDummyDataIntoTable2Dapper(string data)
{
using var scope = ServiceProvider.CreateScope();
var mainTableRepository =
scope.ServiceProvider
.GetRequiredService<MainTableRepository>();
await mainTableRepository.InsertAsync(new MainTable()
{
Data = data,
UpdatedDate = DateTime.Now
});
}
I had to use scope here since repository is scoped and worker is singleton so It cannot be injected directly.
I've tried persistence with EF Core as well same results:
Transaction.Complete() line passes and then when trying to dispose of transaction scope it hangs(sometimes it manages to insert couple of rows then hangs).
Without transaction scope everything works fine
I'm not sure what(if anything) I'm missing here or simply this still does not work in .NET7?
Note that I have MSDTC enable on my machine and im executing this on Windows 10
We've been able to solve this by using the following code.
With this modification DTC is actually invoked correctly and works from within .NET7.
using var transactionScope = TransactionUtils.CreateTransactionScope().EnsureDistributed();
Extension method EnsureDistributed implementation is as follows:
public static TransactionScope EnsureDistributed(this TransactionScope ts)
{
Transaction.Current?.EnlistDurable(DummyEnlistmentNotification.Id, new DummyEnlistmentNotification(),
EnlistmentOptions.None);
return ts;
}
internal class DummyEnlistmentNotification : IEnlistmentNotification
{
internal static readonly Guid Id = new("8d952615-7f67-4579-94fa-5c36f0c61478");
public void Prepare(PreparingEnlistment preparingEnlistment)
{
preparingEnlistment.Prepared();
}
public void Commit(Enlistment enlistment)
{
enlistment.Done();
}
public void Rollback(Enlistment enlistment)
{
enlistment.Done();
}
public void InDoubt(Enlistment enlistment)
{
enlistment.Done();
}
This is 10year old code snippet yet it works(im guessing because .NET Core merely copied and refactored the code from .NET for DistributedTransactions, which also copied bugs).
What it does it creates Distributed transaction right away rather than creating LTM transaction then promoting it to DTC if required.
More details explanation can be found here:
https://www.davidboike.dev/2010/04/forcibly-creating-a-distributed-net-transaction/
https://github.com/davybrion/companysite-dotnet/blob/master/content/blog/2010-03-msdtc-woes-with-nservicebus-and-nhibernate.md
Ensure you're using Microsoft.Data.SqlClient +v5.1
Replace all "usings" System.Data.SqlClient > Microsoft.Data.SqlClient
Ensure ImplicitDistributedTransactions is set True:
TransactionManager.ImplicitDistributedTransactions = true;
using (var ts = new TransactionScope(your options))
{
TransactionInterop.GetTransmitterPropagationToken(Transaction.Current);
... your code ..
ts.Complete();
}

How to send logs to GrayLog from T-SQL procedures?

I can send logs to log collectors from C# applications using log4net+GELF appender.
But how to send logs to GrayLog from T-SQL procedures?
There are code:
WinForms app works fine, I run it on the same machine where SQL Server installed. I see all logs received in GrayLog:
private void button1_Click(object sender, EventArgs e)
{
string facility = "DoBeDo";
string host = "my-host-name";
int port = 12201;
try
{
using (var logger = new GrayLogUdpClient(facility, host, port))
{
logger.Send("Hello", "Jonny Holiday", new { Username = "John", Email = "jonny#example.com" });
}
}
catch(Exception xx)
{
Console.WriteLine("***Exception:{0}", xx.Message);
}
}
There is SQLCLR code, it works, I see messages in SSMS but no any records in GrayLog and no any exceptions:
public partial class StoredProcedures
{
[Microsoft.SqlServer.Server.SqlProcedure]
public static void SqlSPHelper(SqlString msg)
{
try
{
SqlContext.Pipe.Send(#"SqlSPHelper:: Start");
string facility = "DoBeDo";
string host = "my-host-name";
int port = 12201;
try
{
using (var logger = new GrayLogUdpClient(facility, host, port))
{
logger.Send("Hey", "Donny Hooligan", new { Username = "Donald", Email = "Donny#example.com" });
}
}
catch (Exception xx)
{
Console.WriteLine("***Exception:{0}", xx.Message);
}
}
catch(Exception xx)
{
SqlContext.Pipe.Send("1:"+xx.Message);
}
SqlContext.Pipe.Send(#"SqlSPHelper:: Completed");
}
What is wrong? How to send logs to GrayLog?
I think you can do this in couple of ways:
Define a CLR stored procedure, which does the logging part
In the CATCH Block of the stored procedure, RAISERROR WITH LOG to write to Windows event viewer & SQL Server error log. Later you can filter and read these events from event viewer in your GrayLog.
The first method is better, as it is cleaner. Second method makes the SQL Server error log to have many log entries, which would cause false alarms for DBA team.

Flink Jdbc sink

I have created an application where I read data from Kinesis streams and sink the data into mysql table.
I tried to load test the app. For 100k entries it takes more than 3 hours. Any suggestion why it's happening so slow. One more thing is the primary key of my table consist of 7 columns.
I am using hibernate to store POJOs directly into database.
code :
public class JDBCSink extends RichSinkFunction<CompetitorConfig> {
private static SessionFactory sessionFactory;
private static StandardServiceRegistryBuilder serviceRegistry;
private Session session;
private String username;
private String password;
private static final Logger LOG = LoggerFactory.getLogger(CompetitorConfigJDBCSink.class);
public JDBCSink(String user, String pass) {
username = user;
password = pass;
}
public static void configureHibernateUtil(String username, String password) {
try {
Properties prop= new Properties();
prop.setProperty("hibernate.dialect", "org.hibernate.dialect.MySQLDialect");
prop.setProperty("hibernate.connection.driver_class", "com.mysql.cj.jdbc.Driver");
prop.setProperty("hibernate.connection.url", "url");
prop.setProperty("hibernate.connection.username", username);
prop.setProperty("hibernate.connection.password", password);
org.hibernate.cfg.Configuration configuration = new org.hibernate.cfg.Configuration().addProperties(prop);
configuration.addAnnotatedClass(CompetitorConfig.class);
serviceRegistry = new StandardServiceRegistryBuilder().applySettings(configuration.getProperties());
sessionFactory = configuration.buildSessionFactory(serviceRegistry.build());
} catch (Throwable ex) {
throw new ExceptionInInitializerError(ex);
}
}
#Override
public void open(Configuration parameters) throws Exception {
configureHibernateUtil(username,password);
this.session = sessionFactory.openSession();
}
#Override
public void invoke(CompetitorConfig value) throws Exception {
Transaction transaction = null;
try {
transaction = session.beginTransaction();
session.merge(value);
session.flush();
} catch (Exception e) {
throw e;
} finally {
transaction.commit();
}
}
#Override
public void close() throws Exception {
this.session.close();
sessionFactory.close();
}
}
This is slow because you writing each record individually, wrapped in its own transaction. A high performance database sink will do buffered, bulk writes, and commit transactions as part of checkpointing.
If you need exactly once guarantees and can be satisfied with upsert semantics, you can use FLINK's existing JDBC sink. If you require two-phase commit, that's already been merged to master, and will be included in Flink 1.13. See FLINK-15578.
Update:
There's no standard SQL syntax for an upsert; you'll need to figure out if and how your database supports this. For example:
MySQL:
INSERT ... ON DUPLICATE KEY UPDATE ...
PostgreSQL:
INSERT ... ON CONFLICT ... DO UPDATE SET ...
For what it's worth, applications like this are generally easier to implement using Flink SQL. In that case you would use the Kinesis table connector together with the JDBC table connector.

ServiceStack Ormlite transaction between services

I'm having trouble with a rather complex save operation inside a ServiceStack service.
To simplify the explanation the service starts an Ormlite transaction and within it calls another service through ResolveService:
public ApplicationModel Post(ApplicationModel request)
{
using (IDbTransaction tr = Db.OpenTransaction())
{
using (var cases = ResolveService<CaseService>())
{
request.Case = cases.Post(request.Case);
}
}
Db.Save<Application>(request.Application, true);
}
The other service (CaseService) uses also a transaction to perform its logic:
public CaseModel Post(CaseModel request)
{
using (IDbTransaction tr = Db.OpenTransaction())
{
Db.Insert<Case>(request);
Db.SaveAllReferences<CaseModel>(request);
}
}
In a similar situation with higher hierarchy of services calling other services a "Timeout expired" error is thrown, and so far I've not been able to resolve, although I closely monitored the SQL Server for deadlocks.
My question is whether this is the right way of using/sharing Ormlite transactions across services or there is another mechanism?
Thanks in advance.
You shouldn't have nested transactions, rather than calling across services to perform DB operations you should extract shared logic out either using a separate shared Repository or re-usable extension methods:
public static class DbExtensions
{
public static void SaveCaseModel(this IDbConnection db,
CaseModel case)
{
db.Insert<Case>(case);
db.SaveAllReferences<CaseModel>(case);
}
}
Then your Services can maintain their own transactions whilst being able to share logic, e.g:
public ApplicationModel Post(ApplicationModel request)
{
using (var trans = Db.OpenTransaction())
{
Db.SaveCaseModel(request.Case);
Db.Save<Application>(request.Application, true);
trans.Commit();
}
}
public CaseModel Post(CaseModel request)
{
using (var trans = Db.OpenTransaction())
{
Db.SaveCaseModel(request);
trans.Commit();
}
}

database connection on local machine

I was originally coding on a server with Apache installed and used this function to connect to the database
public Connection getDBConnection()
{
java.sql.Connection conn=null;
//synchrnized(this)
try
{
DriverManager.registerDriver(new OracleDriver());
conn = DriverManager.getConnection("jdbc:oracle:thin:#dukeorrac01:1521:ORDB1","nrsc","nrsc");
}
catch (Exception e)
{
e.printStackTrace();
}
return conn;
}
Probably not the best way to do it, but it worked. However, I've moved to coding on my local machine and I was given jboss as an IDE to work with. I've gotten everything working (sorta) except, it seems to be connecting to a different databse. I get exhausted resultsets (which isn't right). I "thought" the "getConnection("jdbc:oracle....") was what established the connection to the actual database.
Why is this not working on my local machine when it worked perfectly fine on the remote server?
P.S. I'm new to database/server configuration stuff. So, don't assume I know some step in setting up a database or server. Also, I did not create this orginally. It was given to me to use.
Thanks
The error seems to indicate that you are not closing your result sets properly. In the following example see how after a statement is executed you have to close those resources in the finally block.
W.r.t why you did not see this error before is probably because your server probably had lots of resources whereas your local machine has limited resources and your default settings are not modified to reflect the needs of your application.
In general, its better to do basic house keeping like closing all the open result sets and statements whether or not you have enough resources allocated on your DB.
public void getDBConnection() {
Connection conn = null;
Statement statement = null;
ResultSet rs = null;
try {
connection = DriverManager.getConnection("jdbc:oracle:thin:#dukeorrac01:1521:ORDB1","nrsc","nrsc");
statment = connection.createStatement();
statement.setFetchSize(Integer.MIN_VALUE);
// Do more stuff, iterate to ResultSet etc...
} catch (SQLException ex) {
// Exception handling stuff
...
} finally {
if (rs != null) {
try {
rs.close();
} catch (SQLException e) { /* ignored */}
}
if (statment != null) {
try {
statment.close();
} catch (SQLException e) { /* ignored */}
}
if (conn != null) {
try {
conn.close();
} catch (SQLException e) { /* ignored */}
}
}
}

Resources