I'm inserting data into multiple tables, and I use the mybatis component to do that. I also need to create a temporary table before I can insert the data. High-level overview is:
Get data to insert
Create temp table
Insert data to temp table
Insert into table1 select x from temp table
Insert into table2 select y from temp table
Steps 2 to 5 should be their own single transaction, in case something fails. I've got this currently:
from(initialEndpoint)
.routeId("database-appender")
.aggregate().expression(constant(true)).completionSize(100).aggregationStrategy(new LinkListAggregator())
.transacted()
.bean(CreateTmpLinksTable.class)
.to("mybatis:prepareLinks?executorType=reuse&statementType=InsertList")
.to("mybatis:insertLinks?executorType=reuse&statementType=InsertList")
.to("mybatis:insertLinkSources?executorType=reuse&statementType=InsertList")
.end()
.log("Wrote at most ${body.size} links to the database")
The CreateTmpLinksTable needs to have access to the current connection, such that the creation of the temporary table does not happen in a different transaction (targeting PostgreSQL, if it matters).
I have this currently:
public class CreateTmpLinksTable {
public void createImportTable(Exchange exchange) throws SQLException {
final Connection conn = exchange.getIn().getHeader("TransactionConnection", Connection.class);
try (final Statement stat = conn.createStatement()) {
stat.execute("CREATE TEMPORARY TABLE tmp_links(" +
"url text, hostname text, service media, service_id bigint, user_id bigint, screen_name text, harvested_at timestamp with time zone, body text" +
") ON COMMIT DROP");
}
}
}
I also haven't setup my transaction manager. My suspicion is I have to get hold of the transaction manager, in order to correctly participate in the transaction.
Questions:
How do I get the transaction manager from a regular bean? Is it just a matter of getting the context, then from the context getting the manager through the registry?
Is there a better way to do what I need? I can see at least one: move all responsibilities into a single bean and do the work there. Any other ways?
NOTE: I'm learning Camel, and I like to do things using only code. Once I know how everything is wired up, then I can transfer that knowledge to Spring.
Q1,
If you can pass the instance of bean to the camel route, you can setup the transaction manager yourself, otherwise you have to use the registry to look up the transaction manager instance.
Q2,
You can wrap the DB update work in a single bean and use the transacted DSL in camel if you have other resources need to be managed.
Related
I have an ASP.NET core 2.0 using Entity Framework core on a SQL Server db.
I have to trace and audit all the stuff made by the users on the data. My goal is to have an automatic mechanism writing all what is happening.
For example, if I have the table Animals, I want a parallele table "Audit_animals" where you can find all the info about the data, the operation type (add, delete, edit) and the user who made this.
I already made this time ago in Django + MySQL, but now the environment is different. I found this and it seems interesting, but I'd like to know if there are better ways and which is the best approach to do this in EF Core.
UPDATE
I'm trying this and something happens, but I have some problems.
I added this:
services.AddMvc().AddJsonOptions(options => {
options.SerializerSettings.ReferenceLoopHandling = ReferenceLoopHandling.Ignore;
});
public Mydb_Context(DbContextOptions<isMultiPayOnLine_Context> options) : base(options)
{
Audit.EntityFramework.Configuration.Setup()
.ForContext<Mydb_Context>(config => config
.IncludeEntityObjects()
.AuditEventType("Mydb_Context:Mydb"))
.UseOptOut()
}
public MyRepository(Mydb_Context context)
{
_context = context;
_context.AddAuditCustomField("UserName", "pippo");
}
I also created a table to insert the audits (only one to test this tool), but the only thing I got is what you see in the image. A list of json files with the data I created.... why??
Read the documentation:
Event Output
To configure the output persistence mechanism please see Configuration and Data Providers sections.
Then, in the documentation on Configuration:
If you don't specify a Data Provider, a default FileDataProvider will be used to write the events as .json files into the current working directory. (emphasis mine)
Long and short, follow the documentation to configure the data provider you'd like to use.
If you are going to map the audit table (Audit_Animals) to the same EF context as the audited Animals table, you can use the EntityFramework Data Provider included on the same Audit.EntityFramework library.
Check the documentation here:
Entity Framework Data Provider
If you plan to store the audit logs in
the same database as the audited entities, you can use the
EntityFrameworkDataProvider. Use this if you plan to store the audit
trails for each entity type in a table with similar structure.
There is another library that can audit EF contexts in a similar way, take a look: zzzprojects/EntityFramework-Plus.
Cannot recommend one over the other since they provide different features (and I'm the owner of the audit.net library).
Update:
.NET 6 and Entity Framework Core 6.0 supports SQL Server temporal tables out of the box.
See this answer for examples:
https://stackoverflow.com/a/70017768/3850405
Original:
You could have a look at Temporal tables (system-versioned temporal tables) if you are using SQL Server 2016< or Azure SQL.
https://learn.microsoft.com/en-us/sql/relational-databases/tables/temporal-tables?view=sql-server-ver15
From documentation:
Database feature that brings built-in support for providing
information about data stored in the table at any point in time rather
than only the data that is correct at the current moment in time.
Temporal is a database feature that was introduced in ANSI SQL 2011.
There is currently an open issue to support this out of the box:
https://github.com/dotnet/efcore/issues/4693
There are third party options available today but since they are not from Microsoft it is of course a risk that they won't be supported in future versions.
https://github.com/Adam-Langley/efcore-temporal-query
https://github.com/findulov/EntityFrameworkCore.TemporalTables
I solved it like this:
If you use the included Visual Studio 2019 LocalDB (Microsoft SQL Server 2016 (13.1.4001.0 LocalDB) you will need to upgrade if you use cascading DELETE or UPDATE. This is because Temporal tables with cascading actions is not supported in that version.
Complete guide for upgrading here:
https://stackoverflow.com/a/64210519/3850405
Start by adding a new empty migration. I prefer to use Package Manager Console (PMC):
Add-Migration "Temporal tables"
Should look like this:
public partial class Temporaltables : Migration
{
protected override void Up(MigrationBuilder migrationBuilder)
{
}
protected override void Down(MigrationBuilder migrationBuilder)
{
}
}
Then edit the migration like this:
public partial class Temporaltables : Migration
{
List<string> tablesToUpdate = new List<string>
{
"Images",
"Languages",
"Questions",
"Texts",
"Medias",
};
protected override void Up(MigrationBuilder migrationBuilder)
{
migrationBuilder.Sql($"CREATE SCHEMA History");
foreach (var table in tablesToUpdate)
{
string alterStatement = $#"ALTER TABLE [{table}] ADD SysStartTime datetime2(0) GENERATED ALWAYS AS ROW START HIDDEN
CONSTRAINT DF_{table}_SysStart DEFAULT GETDATE(), SysEndTime datetime2(0) GENERATED ALWAYS AS ROW END HIDDEN
CONSTRAINT DF_{table}_SysEnd DEFAULT CONVERT(datetime2 (0), '9999-12-31 23:59:59'),
PERIOD FOR SYSTEM_TIME (SysStartTime, SysEndTime)";
migrationBuilder.Sql(alterStatement);
alterStatement = $#"ALTER TABLE [{table}] SET (SYSTEM_VERSIONING = ON (HISTORY_TABLE = History.[{table}]));";
migrationBuilder.Sql(alterStatement);
}
}
protected override void Down(MigrationBuilder migrationBuilder)
{
foreach (var table in tablesToUpdate)
{
string alterStatement = $#"ALTER TABLE [{table}] SET (SYSTEM_VERSIONING = OFF);";
migrationBuilder.Sql(alterStatement);
alterStatement = $#"ALTER TABLE [{table}] DROP PERIOD FOR SYSTEM_TIME";
migrationBuilder.Sql(alterStatement);
alterStatement = $#"ALTER TABLE [{table}] DROP DF_{table}_SysStart, DF_{table}_SysEnd";
migrationBuilder.Sql(alterStatement);
alterStatement = $#"ALTER TABLE [{table}] DROP COLUMN SysStartTime, COLUMN SysEndTime";
migrationBuilder.Sql(alterStatement);
alterStatement = $#"DROP TABLE History.[{table}]";
migrationBuilder.Sql(alterStatement);
}
migrationBuilder.Sql($"DROP SCHEMA History");
}
}
tablesToUpdate should contain every table you need history for.
Then run Update-Database command.
Original source, a bit modified with escaping tables with square brackets etc:
https://intellitect.com/updating-sql-database-use-temporal-tables-entity-framework-migration/
Testing Create, Update and Delete will then show a complete history.
[HttpGet]
public async Task<ActionResult<string>> Test()
{
var identifier1 = "OATestar123";
var identifier2 = "OATestar12345";
var newQuestion = new Question()
{
Identifier = identifier1
};
_dbContext.Questions.Add(newQuestion);
await _dbContext.SaveChangesAsync();
var question = await _dbContext.Questions.FirstOrDefaultAsync(x => x.Identifier == identifier1);
question.Identifier = identifier2;
await _dbContext.SaveChangesAsync();
question = await _dbContext.Questions.FirstOrDefaultAsync(x => x.Identifier == identifier2);
_dbContext.Entry(question).State = EntityState.Deleted;
await _dbContext.SaveChangesAsync();
return Ok();
}
Tested a few times but the log will look like this:
This solution has a huge advantage IMAO that it is not Object Relational Mapper (ORM) specific and you will even get history if you write plain SQL.
The History tables are also read only by default so less chance of a corrupt audit trail. Error received: Cannot update rows in a temporal history table ''
If you need access to the data you can use your preferred ORM to fetch it or audit via SQL.
I am using a camel jdbc component to insert a record into an Oracle table. the insert uses a sequence to populate a primary key ID column.
INSERT INTO my_table (id, data) VALUES (my_seq.nextval, 'some data')
The relevant part of the route looks like below:
from("some end point here")
.process(preInsertProcessor)
.to("jdbc:myDataSource")
.process(new Processor() {
public void process(Exchange exchange) throws Exception {
LOGGER.info("Extracting database generated id");
// This list is null
List<Integer> ids = exchange.getIn().getHeader(
JdbcConstants.JDBC_GENERATED_KEYS_DATA, List.class);
});
Inside the preInsertProcessir I set the message body to be my insert statement and also set some two header values to instruct camel I want the generated ID back:
message.setBody("INSERT INTO my_table (id, data) VALUES (my_seq.nextval, ?:data)");
message.setHeader("data", "some data");
message.setHeader(JDBC_RETRIEVE_GENERATED_KEYS, true);
message.setHeader(JDBC_GENERATED_COLUMNS, new String[]{"ID"});
End if I look at the logs I can see:
[DEBUG] org.apache.camel.component.bean.MethodInfo - Setting bean invocation result on the OUT message: [Message: INSERT INTO my_table(id, data)VALUES (my_seq.nextval, :?data]
[DEBUG] org.apache.camel.spring.spi.TransactionErrorHandler - Transaction begin (0x1de4bee0) redelivered(false) for (MessageId: ID-MELW1TYGC2S-62650-1438583607644-0-8 on ExchangeId: ID-MELW1TYGC2S-62650-1438583607644-0-9))
[INFO ] au.com.nab.cls.router.non.repudiation.GeneratedIdExtractor - Extracting database generated id
[DEBUG] org.apache.camel.processor.MulticastProcessor - Done sequential processing 1 exchanges
[DEBUG] org.apache.camel.spring.spi.TransactionErrorHandler - Transaction commit (0x1de4bee0) redelivered(false) for (MessageId: ID:414d5120445041594855423120202020027844552045b302 on ExchangeId: ID-MELW1TYGC2S-62650-1438583607644-0-7))
If I get it well from the look of the logs the insert would be executed and my final processor should be able to get the generated ID. In reality what happens is that no record gets inserted and no ID is present in the header of the message. Without the final processor everything works fine.
Obviously I am doing something wrong here but I cannot figure out what. I am aware I could use a message en-richer to get the ID before the insert but I would prefer to avoid an extra database trip.
Thank you in advance for your inputs.
UPDATE
I put a break point in org.apache.camel.component.jdbc.JdbcProducer and found out the reason for not having the INSERT executed and consequently not getting the ID back.
// JdbcProducer code; creating a prepared statement part
if (shouldRetrieveGeneratedKeys) {
...
} else if (expectedGeneratedColumns instanceof String[]) {
// Execution gets herestatement
ps = conn.prepareStatement(preparedQuery, (String[]) expectedGeneratedColumns);
...
}
// Expected count returned here is 2
int expectedCount = ps.getParameterMetaData().getParameterCount();
if (expectedCount > 0) {
...
// And here I get the crash:
// java.sql.SQLException: Number of parameters mismatch. Expected: 2, was:1
getEndpoint().getPrepareStatementStrategy().populateStatement(ps, it, expectedCount);
}
This is where my research stopped as digging too much in the various three parties code is not really easy. I suspect one of the following two options are the cause:
I am still not doing it the right way
A camel bug which does not work as expected when header contains both named parameters and retrieve generated keys settings
Please advise about any fix or work around.
Thanks again
I also ran into this. My workaround:
Use camel-sql instead of camel-jdbc. Add parametersCount option to the endpoint URL (in adddition to setHeader(SqlConstants.SQL_RETRIEVE_GENERATED_KEYS, constant(true)) and setHeader(SqlConstants.SQL_GENERATED_COLUMNS, constant(new String[] {"ID_COLUMN_NAME"})).
Update: Works with 11.2.0.4 jdbc driver (does not work with 12.2.0.1 jdbc driver).
We're having a strange problem in Oracle. I'll sketch some (simplified) context first:
Consider this mapping to an Entity:
public EntityMap()
{
Table("EntityTable");
Id(x => x.Id)
.Column("entityID")
.GeneratedBy.Native("ENTITYID").UnsavedValue(0);
Map(x => x.SomeBoolean).Column("SomeBoolean");
}
and this code:
var entity = new Entity();
using (var transaction = new TransactionScope(TransactionScopeOption.Required))
{
Session.Save(entity);
transaction.Complete();
}
//A lot of code
if(someCondition)
{
using (var transaction = new TransactionScope(TransactionScopeOption.Required))
{
enitity.SomeBoolean = true;
Session.Update(entity);
transaction.Complete();
}
}
This code is called a few times. The first time it generates the following queries:
select ENTITYID.nextval from dual
INSERT INTO Entity
(SomeBoolean, EntityID)
VALUES (0, 1216)
UPDATE Entity
SET SomeBoolean = 1
WHERE EntityID = 1216
The second time it is called these queries are generated (someCondition is false)
select ENTITYID.nextval from dual
INSERT INTO Entity
(SomeBoolean, EntityID)
VALUES (0, 1217)
And now the trouble begins. From now on, each insert will use the correct autoincremented value, but the update will always use 1217
select ENTITYID.nextval from dual
INSERT INTO Entity
(SomeBoolean, EntityID)
VALUES (0, 1218)
UPDATE Entity
SET SomeBoolean = 1
WHERE EntityID = 1217
And of course, this is not what we want to happen. If I inspect the value of the Id while debugging, it contains the correct autoincremented value. Somehow, deep in the bowels of NHibernate, the incorrect is is assigned to the WHERE clause...
The strange part is that this only happens on Oracle. If I switch NHibernate to MsSql, everything works like a charm.
So I found out what happened. NHibernate changed it's default connection release mode between versions 1.x and 2.x. Instead of closing the connection when the session is Disposed, the connections is now closed after each transaction. However, we were manually coordinating our transactions which apparently caused troubles in Oracle.
This question has some extra information and this entry in the NHibernate documentation also clarifies how the connections are handeled:
As of NHibernate, if your application manages transactions through .NET APIs such as System.Transactions library, ConnectionReleaseMode.AfterTransaction may cause NHibernate to open and close several connections during one transaction, leading to unnecessary overhead and transaction promotion from local to distributed. Specifying ConnectionReleaseMode.OnClose will revert to the legacy behavior and prevent this problem from occuring.
This blog post is what got me looking in the right direction.
How do I achieve the Transaction involving multiple DB operations to >1 tables using iBatis & Spring?
Let me explain in detail:
I have 2 tables A & B with Master-details relationship. [Both tables in single database].
/* Table A: */
a_id [Primary Key]
[plus other columns]
/* Table B: */
b_id [Primary Key]
a_id [Foreign Key = PK of table A]
[plus other columns]
In my Dao I have following methods (I am using iBatis sqlMap toperform DB operations):
insertA();
insertB();
updateA();
updateB();
deleteA();
deleteB();
Each of the above operations are Atomic (& can be called by client & commited in database -via Spring/iBatis).
Up to this point everything WORKS OK! [i.e. I am able to perform INDIVIDUAL insert/update/delete on each table.]
-- NEXT, I need to perform a combination of two of above DB operations as an ATOMIC operation;
Here is what I want to achieve from SVC layer:
start Tranaction
operation on Table-A (via method of Dao class) - op #1
operation on Table-B (via method of Dao class) - op #2
end Transaction
Example1:
start Tranaction
insertA();
insertB();
end Transaction
Example2:
start Tranaction
updateA();
updateB();
end Transaction
Here, if op#2 Fails, I want op#1 also to be Rolled back. i.e. Complete Rollback.
So, I wrote additional method within the Service layer, which calls above DAO methods.
Before running the (Svc) code, I manually [via cmd-line] change some data On database, so that 2nd operation FAILS due to DB Constraints.
Now, op #2 [Table-B] FAILS, but op #1 is commited in DB. i.e. there is NO complete rollback, ONLY PARTIAL rollback.
If op #2 Fails, shouldn't op#1 also Roll back?
Here is what I am using in ApplicationContext.xml:
"DataSourceTransactionManager" [Spring] for Transaction.
iBatis 2.3.x [SqlMapClient]
Spring 3.0
DefaultAutoCommit is set to FALSE.
In "tx:method": [service method from where ATOMIC operation is to be performed)
propagation="REQUIRED" [Tried with other values also, but no use]
rollback-for=Exception-Name-for-which-to-rollback
Is there anything else that needs to be done?
Am I doing something wrong?
Is this correct way or is there a better option?
<
In my opinion, you should consider the data integrity, if op #2 make the system loose data integrity, then it should roll back according to op #1.
To achieve what you want, just make a call to op #1 and #2, wrapper #2 on try/catch block, something like:
try {
start Tranaction ;
//pkA is primary key of A
Object pkA = insertA();
updateA(pkA);
try {
Object pkB = insertB(pkA);
updateB(pkB);
}
catch(Exception e) {
logger.ERROR("Error when inserting and updating B.Ignore. ",e);
}
commit transaction;
}
catch(Exception e) {
logger.ERROR(e);
rollback Transaction;
}
HTH.
I am trying to use Dapper support my data access for my server app.
My server app has another application that drops records into my database at a rate of 400 per minute.
My app pulls them out in batches, processes them, and then deletes them from the database.
Since data continues to flow into the database while I am processing, I don't have a good way to say delete from myTable where allProcessed = true.
However, I do know the PK value of the rows to delete. So I want to do a delete from myTable where Id in #listToDelete
Problem is that if my server goes down for even 6 mintues, then I have over 2100 rows to delete.
Since Dapper takes my #listToDelete and turns each one into a parameter, my call to delete fails. (Causing my data purging to get even further behind.)
What is the best way to deal with this in Dapper?
NOTES:
I have looked at Tabled Valued Parameters but from what I can see, they are not very performant. This piece of my architecture is the bottle neck of my system and I need to be very very fast.
One option is to create a temp table on the server and then use the bulk load facility to upload all the IDs into that table at once. Then use a join, EXISTS or IN clause to delete only the records that you uploaded into your temp table.
Bulk loads are a well-optimized path in SQL Server and it should be very fast.
For example:
Execute the statement CREATE TABLE #RowsToDelete(ID INT PRIMARY KEY)
Use a bulk load to insert keys into #RowsToDelete
Execute DELETE FROM myTable where Id IN (SELECT ID FROM #RowsToDelete)
Execute DROP TABLE #RowsToDelte (the table will also be automatically dropped if you close the session)
(Assuming Dapper) code example:
conn.Open();
var columnName = "ID";
conn.Execute(string.Format("CREATE TABLE #{0}s({0} INT PRIMARY KEY)", columnName));
using (var bulkCopy = new SqlBulkCopy(conn))
{
bulkCopy.BatchSize = ids.Count;
bulkCopy.DestinationTableName = string.Format("#{0}s", columnName);
var table = new DataTable();
table.Columns.Add(columnName, typeof (int));
bulkCopy.ColumnMappings.Add(columnName, columnName);
foreach (var id in ids)
{
table.Rows.Add(id);
}
bulkCopy.WriteToServer(table);
}
//or do other things with your table instead of deleting here
conn.Execute(string.Format(#"DELETE FROM myTable where Id IN
(SELECT {0} FROM #{0}s", columnName));
conn.Execute(string.Format("DROP TABLE #{0}s", columnName));
To get this code working, I went dark side.
Since Dapper makes my list into parameters. And SQL Server can't handle a lot of parameters. (I have never needed even double digit parameters before). I had to go with Dynamic SQL.
So here was my solution:
string listOfIdsJoined = "("+String.Join(",", listOfIds.ToArray())+")";
connection.Execute("delete from myTable where Id in " + listOfIdsJoined);
Before everyone grabs the their torches and pitchforks, let me explain.
This code runs on a server whose only input is a data feed from a Mainframe system.
The list I am dynamically creating is a list of longs/bigints.
The longs/bigints are from an Identity column.
I know constructing dynamic SQL is bad juju, but in this case, I just can't see how it leads to a security risk.
Dapper request the List of object having parameter as a property so in above case a list of object having Id as property will work.
connection.Execute("delete from myTable where Id in (#Id)", listOfIds.AsEnumerable().Select(i=> new { Id = i }).ToList());
This will work.