Deadlock while running multiple instances of a spring batch job [duplicate] - sql-server

Question

This question already has answers here:
Spring Batch Deadlock - Could not increment identity; nested exception is com.microsoft.sqlserver.jdbc.SQLServerException
(2 answers)
Closed 7 months ago.
I have a spring batch job that reads from a database and writes into a file after doing some processing, in a chunk based step.
My requirement is to run almost 16 instances of the job parallelly at the same time, just with different job parameters.
But I've been facing the a couple of issues while doing so.
1.
Could not open JDBC Connection for transaction. Nested exception is java.sql.SQLTransientConnectionException: Hikaripool -1 - Connection is not available.
Exception: could not increment identity. Nested Exception is com.microsoft.SQLserver.jdbc.SQLServerException: Transaction (process ID 124) was deadlocked on lock resources with other process, and has been chosen as the deadlock victim. Rerun the transaction.
I've tried the solutions provided in the link Github link, by setting the IsolationLevel and altering the metadata tables as shown below.
Set the IsolationLevelForCreate like this
JobRepositoryFactoryBean factory = new JobRepositoryFactoryBean();
factory.setIsolationLevelForCreate("ISOLATION_REPEATABLE_READ");
Have the DBA add indexes to each of the SEQ tables like this (JET is my schema that I put the repo tables in):
ALTER TABLE [JET].[BATCH_JOB_EXECUTION_SEQ]
ADD CONSTRAINT [BATCH_JOB_EXECUTION_SEQ_PK] PRIMARY KEY CLUSTERED ([ID] ASC)
GO
ALTER TABLE [JET].[BATCH_JOB_SEQ]
ADD CONSTRAINT [BATCH_JOB_SEQ_PK] PRIMARY KEY CLUSTERED ([ID] ASC)
GO
ALTER TABLE [JET].[BATCH_STEP_EXECUTION_SEQ]
ADD CONSTRAINT [BATCH_STEP_EXECUTION_SEQ_PK] PRIMARY KEY CLUSTERED ([ID] ASC)
GO
But I am still facing the issue.
PS: The spring batch has been deployed to AKS(Azure Kubernetes Services), and using Azure SQLServer as datasource.

Answer 1

Based on the discussion in https://github.com/spring-projects/spring-batch/issues/1448, the issue seems to be caused by the SqlServerMaxValueIncrementer from Spring Framework not using SQLServer's native sequences. Here is an excerpt from the Javadoc:
There should be one sequence table per table that needs an auto-generated key.
Example:
create table tab (id int not null primary key, text varchar(100))
create table tab_sequence (id bigint identity)
insert into tab_sequence default values
This could be due to SQLServer not supporting sequences until recently. But I guess that's why Spring Batch uses tables to emulate sequences for MS SQL Server.
I suggest you try to change the default DDL to use sequences instead of tables:
CREATE SEQUENCE BATCH_STEP_EXECUTION_SEQ ;
CREATE SEQUENCE BATCH_JOB_EXECUTION_SEQ ;
CREATE SEQUENCE BATCH_JOB_SEQ ;
This is the default sequences definition based on MS SQL Server's docs. This should work, but you can customize them if needed.
You might also need to provide a custom DataFieldMaxValueIncrementer that is based on sequences (since the one from Spring Framework uses tables) and register it in Spring Batch through a DataFieldMaxValueIncrementerFactory (See JobRepositoryFactoryBean#setIncrementerFactory).