Hi We are using tomcat 6 and context.xml is like below
<Context>
<Resource defaultAutoCommit="false" defaultReadOnly="false"
driverClassName="oracle.jdbc.driver.OracleDriver"
fairQueue="false" initialSize="${DBPool.initialPoolSize}"
jdbcInterceptors="ConnectionState;StatementFinalizer"
jmxEnabled="true" logAbandoned="false" maxActive="${DBPool.maxPoolSize}"
maxIdle="30" maxWait="30000"
minEvictableIdleTimeMillis="5000" minIdle="${DBPool.minPoolSize}"
name="jdbc/BankDBPool" password="${DBPool.bankPassword}"
removeAbandoned="true" removeAbandonedTimeout="60"
testOnBorrow="false" testOnReturn="true"
testWhileIdle="false" timeBetweenEvictionRunsMillis="5000"
type="javax.sql.DataSource"
url="${DBPool.jdbcUrl}"
factory="uk.co.xxxx.encryption.dbcp.DecryptingBasicDataSourceFactory"
useEquals="false" username="${DBPool.bankUser}" validationInterval="30000" validationQuery="select 1 from dual" />
</Context>
DBPool.maxPoolSize=400
DBPool.minPoolSize=15
DBPool.initialPoolSize=15
The issue is we have to set maxPoolSize to very high as it is giving connection not available Exception.
DB Monitor tool is showing the connections idle but seems they can not be reused. Traffic to this application is very low. Around 10000 hits in a day.
We are trying to figure out what might be an issue here.
All my service methods are all marked
#Transactional(propagation = Propagation.REQUIRED, readOnly = true or false)
DecryptingBasicDataSourceFactory only does job of returning datasource.
We are using spring and hibernate.
The issue was resolved. Apparently one of the method had #Transactional missing. Another change was reducing the batch-size in hibernate properties from 100 to 20. But mostly it was adding #Transactional which fixed the issue.
Related
currently my team and I are facing a strange problem. By now we have been spending 4 days searching for the solution.
We are developing a Java EE Web Application. We have been using Wildfly 14. Using Wildfly 14, there was no problem. Now we have upgraded Wildfly to Version 18 and afterwars to 19 and are now facing the following problem. As database we have an MS SQL.
The datasource looks like this:
<datasource jndi-name="java:/label" pool-name="Label">
<connection-url>jdbc:sqlserver://localhost:1433;DatabaseName=label</connection-url>
<driver>mssql-jdbc-8.2.2.jre8.jar</driver>
<security>
<user-name>label</user-name>
<password>label</password>
</security>
</datasource>
And the persistence.xml is the following:
<?xml version="1.0" encoding="UTF-8"?>
<persistence version="2.0"
xmlns="http://java.sun.com/xml/ns/persistence"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://java.sun.com/xml/ns/persistence http://java.sun.com/xml/ns/persistence/persistence_2_0.xsd">
<persistence-unit name="label-pu"
transaction-type="JTA">
<jta-data-source>java:/label</jta-data-source>
<properties>
<property name="hibernate.dialect"
value="org.hibernate.dialect.SQLServerDialect" />
</properties>
</persistence-unit>
We have a Bean in which we inject an EntityManager using the persistence-unit. When calling the bean's method we are getting an exception. The datasource is working. We checked that in the wildfly ui.
#Dependent
public class ConstructionService {
#PersistenceContext(unitName = "label-pu")
private EntityManager em;
public List<ConstructionRecordEntity> retrieveRecordForConstruction(int contructionId) {
String sqlSelectEmployeeId = "SELECT lfdnr, " + //
" erwkz1, " + //
" erwbe, " + //
" erwdat " + //
" FROM [label].[dbo].[aderw] where adnr = " + contructionId;//
return em//
.createNativeQuery(sqlSelectEmployeeId, ConstructionRecordEntity.class)//
.getResultList();
}
In the following you can see the stacktrace we are getting.
15:11:24,506 WARN [org.camunda.bpm.engine.rest.exception] (default task-1) ENGINE-REST-HTTP500 javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Unable to acquire JDBC Connection
at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:154)
at org.hibernate.query.internal.AbstractProducedQuery.list(AbstractProducedQuery.java:1515)
at org.hibernate.query.Query.getResultList(Query.java:132)
at de._._.ConstructionService.retrieveRecordForConstruction(ConstructionService.java:26)
...
Caused by: org.hibernate.exception.GenericJDBCException: Unable to acquire JDBC Connection
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:47)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:113)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:99)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:109)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getPhysicalConnection(LogicalConnectionManagedImpl.java:136)
at org.hibernate.engine.jdbc.internal.StatementPreparerImpl.connection(StatementPreparerImpl.java:50)
at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$5.doPrepare(StatementPreparerImpl.java:149)
at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$StatementPreparationTemplate.prepareStatement(StatementPreparerImpl.java:176)
at org.hibernate.engine.jdbc.internal.StatementPreparerImpl.prepareQueryStatement(StatementPreparerImpl.java:151)
at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:2082)
at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:2012)
at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1990)
at org.hibernate.loader.Loader.doQuery(Loader.java:949)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:351)
at org.hibernate.loader.Loader.doList(Loader.java:2787)
at org.hibernate.loader.Loader.doList(Loader.java:2770)
at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2604)
at org.hibernate.loader.Loader.list(Loader.java:2599)
at org.hibernate.loader.custom.CustomLoader.list(CustomLoader.java:338)
at org.hibernate.internal.SessionImpl.listCustomQuery(SessionImpl.java:2243)
at org.hibernate.internal.AbstractSharedSessionContract.list(AbstractSharedSessionContract.java:1069)
at org.hibernate.query.internal.NativeQueryImpl.doList(NativeQueryImpl.java:170)
at org.hibernate.query.internal.AbstractProducedQuery.list(AbstractProducedQuery.java:1506)
... 306 more
Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000457: Unchecked throwable in managedConnectionReconnected() cl=org.jboss.jca.core.connectionmanager.listener.TxConnectionListener#c18c68f[state=DESTROYED managed connection=org.jboss.jca.adapters.jdbc.local.LocalManagedConnection#51667cf2 connection handles=0 lastReturned=1594905084472 lastValidated=1594904382991 lastCheckedOut=1594905084468 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.OnePool#66e3a014 mcp=SemaphoreConcurrentLinkedQueueManagedConnectionPool#685b64a3[pool=Label] xaResource=LocalXAResourceImpl#671cf884[connectionListener=c18c68f connectionManager=bec01f1 warned=false currentXid=null productName=Microsoft SQL Server productVersion=15.00.2000 jndiName=java:/label] txSync=null]
at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:159)
at org.jboss.as.connector.subsystems.datasources.WildFlyDataSource.getConnection(WildFlyDataSource.java:64)
at org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:122)
at org.hibernate.internal.NonContextualJdbcConnectionAccess.obtainConnection(NonContextualJdbcConnectionAccess.java:35)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:106)
... 325 more
Caused by: javax.resource.ResourceException: IJ000457: Unchecked throwable in managedConnectionReconnected() cl=org.jboss.jca.core.connectionmanager.listener.TxConnectionListener#c18c68f[state=DESTROYED managed connection=org.jboss.jca.adapters.jdbc.local.LocalManagedConnection#51667cf2 connection handles=0 lastReturned=1594905084472 lastValidated=1594904382991 lastCheckedOut=1594905084468 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.OnePool#66e3a014 mcp=SemaphoreConcurrentLinkedQueueManagedConnectionPool#685b64a3[pool=Label] xaResource=LocalXAResourceImpl#671cf884[connectionListener=c18c68f connectionManager=bec01f1 warned=false currentXid=null productName=Microsoft SQL Server productVersion=15.00.2000 jndiName=java:/label] txSync=null]
at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.reconnectManagedConnection(AbstractConnectionManager.java:1055)
at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:792)
at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:151)
... 329 more
Caused by: javax.resource.ResourceException: IJ000461: Could not enlist in transaction on entering meta-aware object
at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.managedConnectionReconnected(TxConnectionManagerImpl.java:571)
at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.reconnectManagedConnection(AbstractConnectionManager.java:977)
... 331 more
Caused by: javax.transaction.SystemException: Error enlisting resource in transaction=Local transaction (delegate=TransactionImple < ac, BasicAction: 0:ffffc0a83801:34f7831:5f103dd7:8e1 status: ActionStatus.ABORT_ONLY >, owner=Local transaction context for provider JBoss JTA transaction provider)
at org.jboss.jca.core.connectionmanager.listener.TxConnectionListener$TransactionSynchronization.checkEnlisted(TxConnectionListener.java:957)
at org.jboss.jca.core.connectionmanager.listener.TxConnectionListener.enlist(TxConnectionListener.java:394)
at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.managedConnectionReconnected(TxConnectionManagerImpl.java:564)
... 332 more
Caused by: java.lang.Throwable: Failed to enlist
at org.jboss.jca.core.connectionmanager.listener.TxConnectionListener$TransactionSynchronization.enlist(TxConnectionListener.java:1001)
at org.jboss.jca.core.connectionmanager.listener.TxConnectionListener.enlist(TxConnectionListener.java:379)
... 333 more
I apreciate any hint.
We found out, that this is related to Camunda, when we first thought is was a pure Java EE Problem.
Using the mentioned Bean in a Rest Service works perfectly. Only when we inject the Bean into a Camunda JavaDelegate there is a problem.
We could solve this by adding
<system-properties>
<property name="com.arjuna.ats.arjuna.allowMultipleLastResources" value="true"/>
</system-properties>
to Wildfly Standalone.
It is working but we are getting a warning. Actually, this is necessary if you are using xa-datasource to have a transaction that surrounds multiple datasources. However, we do not have xa-datasource and we do not need a transaction that surrounds everything.
As far as we understand Camunda uses JTA to make the process jobs transaction safe. So when we are making a db query inside the JavaDelegate, there is a transaction inside a transaction. The first transaction is instantiated by Camunda the second one is instantiated by us by using JPA to query the database. Nevertheless, I do not understand. This scenario should be very common. So other people should have experienced this behavior, too? Or maybe it is related to the MS SQL Database?
The other thing that I do not understand is, that using JPA inside JavaDelegate worked with camunda 7.10 and in the latest version 7.13 it stopped working.
So, we found a workaround and we are not stuck anymore. Nonetheless, this workaround is not satisfying as we do not understand entirely what is going on underneath and we keep looking for an appropriate solution.
I was working with WildFly server long time ago so there is just my amateur thought:
There is something bad happened with transaction.
The issue point is connected with resource enlistement:
http://javadox.com/org.jboss.ironjacamar/ironjacamar-core-impl/1.2.7.Final/org/jboss/jca/core/connectionmanager/listener/TxConnectionListener.TransactionSynchronization.html#enlist()
According to the documentation, the database connection is also an "Transactional Resource". So, yes, we do not have an database connection for some reason.
May be someone close transaction for some reason? Try to check transaction status with injection of UserTransaction or with debugging of TxConnectionListener class (see the log)
If transaction is ok, then just debug from the point of error. Try to find the check that leads to an error. You can find the proper library to debug inside WildFly directory (with Far Manager, Double Commander and so on) and connect it to the IDE to be able to make a break point in the proper place.
I have a nice little Spring Boot JPA web application. It is deployed on Amazon Beanstalk and uses an Amazon RDS for persisting data. It is however not used that often and therefore fails after a while with this kind of exception:
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet successfully received from the server was 79,870,633 milliseconds ago.
The last packet sent successfully to the server was 79,870,634 milliseconds ago. is longer than the server configured value of 'wait_timeout'.
You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
I am not sure how to configure this setting and can not find information on it on http://spring.io (a very good site though). What are some ideas or pointers to information?
I assume that boot is configuring the DataSource for you. In this case, and since you are using MySQL, you can add the following to your application.properties up to 1.3
spring.datasource.testOnBorrow=true
spring.datasource.validationQuery=SELECT 1
As djxak noted in the comment, 1.4+ defines specific namespaces for the four connections pools Spring Boot supports: tomcat, hikari, dbcp, dbcp2 (dbcp is deprecated as of 1.5). You need to check which connection pool you are using and check if that feature is supported. The example above was for tomcat so you'd have to write it as follows in 1.4+:
spring.datasource.tomcat.testOnBorrow=true
spring.datasource.tomcat.validationQuery=SELECT 1
Note that the use of autoReconnect is not recommended:
The use of this feature is not recommended, because it has side effects related to session state and data consistency when applications don't handle SQLExceptions properly, and is only designed to be used when you are unable to configure your application to handle SQLExceptions resulting from dead and stale connections properly.
The above suggestions did not work for me.
What really worked was the inclusion of the following lines in the application.properties
spring.datasource.testWhileIdle = true
spring.datasource.timeBetweenEvictionRunsMillis = 3600000
spring.datasource.validationQuery = SELECT 1
You can find the explanation out here
Setting spring.datasource.tomcat.testOnBorrow=true in application.properties didn't work.
Programmatically setting like below worked without any issues.
import org.apache.tomcat.jdbc.pool.DataSource;
import org.apache.tomcat.jdbc.pool.PoolProperties;
#Bean
public DataSource dataSource() {
PoolProperties poolProperties = new PoolProperties();
poolProperties.setUrl(this.properties.getDatabase().getUrl());
poolProperties.setUsername(this.properties.getDatabase().getUsername());
poolProperties.setPassword(this.properties.getDatabase().getPassword());
//here it is
poolProperties.setTestOnBorrow(true);
poolProperties.setValidationQuery("SELECT 1");
return new DataSource(poolProperties);
}
I just moved to Spring Boot 1.4 and found these properties were renamed:
spring.datasource.dbcp.test-while-idle=true
spring.datasource.dbcp.time-between-eviction-runs-millis=3600000
spring.datasource.dbcp.validation-query=SELECT 1
whoami's answer is the correct one. Using the properties as suggested I was unable to get this to work (using Spring Boot 1.5.3.RELEASE)
I'm adding my answer since it's a complete configuration class so it might help someone using Spring Boot:
#Configuration
#Log4j
public class SwatDataBaseConfig {
#Value("${swat.decrypt.location}")
private String fileLocation;
#Value("${swat.datasource.url}")
private String dbURL;
#Value("${swat.datasource.driver-class-name}")
private String driverName;
#Value("${swat.datasource.username}")
private String userName;
#Value("${swat.datasource.password}")
private String hashedPassword;
#Bean
public DataSource primaryDataSource() {
PoolProperties poolProperties = new PoolProperties();
poolProperties.setUrl(dbURL);
poolProperties.setUsername(userName);
poolProperties.setPassword(password);
poolProperties.setDriverClassName(driverName);
poolProperties.setTestOnBorrow(true);
poolProperties.setValidationQuery("SELECT 1");
poolProperties.setValidationInterval(0);
DataSource ds = new org.apache.tomcat.jdbc.pool.DataSource(poolProperties);
return ds;
}
}
I have similar problem. Spring 4 and Tomcat 8. I solve the problem with Spring configuration
<bean id="dataSource" class="org.apache.tomcat.jdbc.pool.DataSource" destroy-method="close">
<property name="initialSize" value="10" />
<property name="maxActive" value="25" />
<property name="maxIdle" value="20" />
<property name="minIdle" value="10" />
...
<property name="testOnBorrow" value="true" />
<property name="validationQuery" value="SELECT 1" />
</bean>
I have tested. It works well! This two line does everything in order to reconnect to database:
<property name="testOnBorrow" value="true" />
<property name="validationQuery" value="SELECT 1" />
In case anyone is using custom DataSource
#Bean(name = "managementDataSource")
#ConfigurationProperties(prefix = "management.datasource")
public DataSource dataSource() {
return DataSourceBuilder.create().build();
}
Properties should look like the following. Notice the #ConfigurationProperties with prefix. The prefix is everything before the actual property name
management.datasource.test-on-borrow=true
management.datasource.validation-query=SELECT 1
A reference for Spring Version 1.4.4.RELEASE
As some people already pointed out, spring-boot 1.4+, has specific namespaces for the four connections pools. By default, hikaricp is used in spring-boot 2+. So you will have to specify the SQL here. The default is SELECT 1. Here's what you would need for DB2 for example:
spring.datasource.hikari.connection-test-query=SELECT current date FROM sysibm.sysdummy1
Caveat: If your driver supports JDBC4 we strongly recommend not setting this property. This is for "legacy" drivers that do not support the JDBC4 Connection.isValid() API. This is the query that will be executed just before a connection is given to you from the pool to validate that the connection to the database is still alive. Again, try running the pool without this property, HikariCP will log an error if your driver is not JDBC4 compliant to let you know. Default: none
For those who want to do it from YAML with multiple data sources, there is a great blog post about it: https://springframework.guru/how-to-configure-multiple-data-sources-in-a-spring-boot-application/
It basically says you both need to configure data source properties and datasource like this:
#Bean
#Primary
#ConfigurationProperties("app.datasource.member")
public DataSourceProperties memberDataSourceProperties() {
return new DataSourceProperties();
}
#Bean
#Primary
#ConfigurationProperties("app.datasource.member.hikari")
public DataSource memberDataSource() {
return memberDataSourceProperties().initializeDataSourceBuilder()
.type(HikariDataSource.class).build();
}
Do not forget to remove #Primary from other datasources.
I'm have a Java Maven Google App Engine project configured like follows:
I'm using EclipseLink as JPA persistence-manager for Cloud SQL. My object contains some simple fields (string, date, ...) and a ManyToMany relationship, which is configured as Lazy-Load
#Entity
#Table(name = "mytable")
public class MyObject1 {
private String nome;
private String descrizione;
#ManyToMany
#JoinTable
(
name = "myobject1_has_myobject2",
joinColumns = { #JoinColumn(name = "object1_id", referencedColumnName = "id") },
inverseJoinColumns = { #JoinColumn(name = "object2_id", referencedColumnName = "id") }
)
private List<MyObject2> relationshipObjects;
}
The project flow works like this:
- A query is made that retrieve x results of type MyObject1 (let's say 10 results)
- The query results list is iterated and each result is given to a different Thread for processing
- Each thread iterate the ManyToMany relationship (the relationshipObjects object), which is Lazy and this in confirmed because the code calls IndirectList.iterator, and do some processing for each MyObject2 item of the list
- When all the threads have finished, the query result of MyObject1 is iterated once again to create a request response
This kind of implementation is giving some trouble regarding the multi-thread implementation and a some sort of deadlock.
Here is the stacktrace
Caused by: Exception [EclipseLink-2001] (Eclipse Persistence Services - 2.6.4.v20160829-44060b6): org.eclipse.persistence.exceptions.ConcurrencyException
Exception Description: Wait was interrupted.
Message: [null]
at org.eclipse.persistence.exceptions.ConcurrencyException.waitWasInterrupted(ConcurrencyException.java:108)
at org.eclipse.persistence.internal.helper.ConcurrencyManager.acquireDeferredLock(ConcurrencyManager.java:187)
at org.eclipse.persistence.internal.identitymaps.CacheKey.acquireDeferredLock(CacheKey.java:210)
at org.eclipse.persistence.internal.identitymaps.AbstractIdentityMap.acquireDeferredLock(AbstractIdentityMap.java:84)
at org.eclipse.persistence.internal.identitymaps.IdentityMapManager.acquireDeferredLock(IdentityMapManager.java:146)
at org.eclipse.persistence.internal.sessions.IdentityMapAccessor.acquireDeferredLock(IdentityMapAccessor.java:81)
at org.eclipse.persistence.internal.sessions.AbstractSession.retrieveCacheKey(AbstractSession.java:5200)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:965)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildWorkingCopyCloneNormally(ObjectBuilder.java:899)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObjectInUnitOfWork(ObjectBuilder.java:852)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:735)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:689)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.buildObject(ObjectLevelReadQuery.java:805)
at org.eclipse.persistence.queries.ReadAllQuery.registerResultInUnitOfWork(ReadAllQuery.java:962)
at org.eclipse.persistence.queries.ReadAllQuery.executeObjectLevelReadQuery(ReadAllQuery.java:573)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1175)
at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:904)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1134)
at org.eclipse.persistence.queries.ReadAllQuery.execute(ReadAllQuery.java:460)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1222)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2896)
at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1857)
at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1839)
at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1804)
at org.eclipse.persistence.internal.jpa.QueryImpl.executeReadQuery(QueryImpl.java:258)
at org.eclipse.persistence.internal.jpa.QueryImpl.getResultList(QueryImpl.java:473)
I cannot reproduce all the time the problem, besides that I can give you all the information I can gather.
Looking inside the EclipseLink documentation I found a section related to this matter
Cache - If using a shared cache, EclipseLink requires locking the cache on reads and writes to ensure consistency.
You will see cache access, such as IdentityMapManager acquireLock or acquireDeferredLock, or WriteLockManager as the last call on the stack.
In my persistence-unit I did not configure the shared-cache behaviour, so it is running on default which is enabled.
Here is my persistence-unit properties
<properties>
<!-- configure the various connection pool properties -->
<!-- http://www.eclipse.org/eclipselink/documentation/2.5/jpa/extensions/p_connection_pool.htm -->
<property name="eclipselink.connection-pool.default.initial" value="1" />
<property name="eclipselink.connection-pool.default.min" value="64" />
<property name="eclipselink.connection-pool.default.max" value="64" />
<property name="eclipselink.connection-pool.default.shared" value="true" />
<!-- whether connections in EclipseLink read connection pool should be shared (not exclusive). Connection sharing means the same JDBC connection will be used concurrently for multiple reading threads. -->
<property name="eclipselink.jdbc.connection_pool.read.shared" value="true" />
<!-- specify if JDBC statements should be cached -->
<!-- http://www.eclipse.org/eclipselink/documentation/2.5/jpa/extensions/p_jdbc_cachestatements.htm -->
<property name="eclipselink.jdbc.cache-statements" value="true" />
<!-- the number of statements held when using internal statement caching -->
<!-- http://www.eclipse.org/eclipselink/documentation/2.5/jpa/extensions/p_jdbc_cachestatements_size.htm#CACBICGG -->
<property name="eclipselink.jdbc.cache-statements.size" value="100" />
</properties>
I can see in my stacktrace that there is indeed the IdentityMapManager.acquireDeferredLock(IdentityMapManager.java:146) row that is referred.
The thing is, this error is thrown by the App Engine request (see last line of stacktrace) when I call the getResultList method.
This call is made by the main request thread, other threads (one for query result) has not been launched yet.
So I started to looking for the shared-cache documentation and I found this part:
The shared cache exists for the duration of the persistence unit (EntityManagerFactory, or server) and is shared by all EntityManagers and users of the persistence unit
My EntityManagerFactory instance is instance-shared (I have a static variabile which is initialized at first query).
So at first access (for each App Engine instance) the variabile is initialized and then shared for all the http requests that will be server by the same instance.
I did this sort of "caching" because the deployment descriptor at the first access of EntityManagerFactory is very slow, and even if I can pre-warmup this object, opening a new EMF at every request costs about 1-2 seconds.
So, I open/close a new EntityManager at each flow (and each thread, because EntityManager is not thread-safe) but the EMF object is shared.
Also, there is another line which says
This is normally related to having relationships that do not use LAZY, ensure all relationship use LAZY.
My ManyToMany relationship is already Lazy, as pointed out before, so even this point cannot be the cause
Basing on that here, I tried to gather all together:
- EclipseLink requires locking the cache on reads and writes to ensure consistency, so the access to this cache is atomic and multiple threads are queued.
- The sharedcache is based on the EMF object
- The EMF object is shared between requests of the same instance
As suggested by EclipseLink documentation I tried to disable the shared-cache and all the flow appaers to works, but it is now very slow.
Anyway, this is another point that confirm the problem here is related to the shared-cache of JPA.
This solution is not suitable because, even w/o considering the speed problem, all those request and threads that concurr of get data from the DBMS (while iterating the Lazy list) consumes all the available connections and the DBMS starts on giving connection errors.
Another suggestion from the documentation
DeferredLockManager.SHOULD_USE_DEFERRED_LOCKS = false;
but the error is still the same, nothing changed (the error is on IdentityMapManager.acquireLock, so the deferredLock is not used anyway)
From the App Engine logs I can see that all these requests are killed after 60s timeout, so the Wait was interrupted message can be related to all the threads that were waiting to access the shared-cache, but at the end the App Engine deadline killed the request.
Because of that I tried to deploy on basic-scaling (which does not have the 60s deadline) to see if the request is only slower than the dealine or it is truly stuck on a deadlock
Inside the logs there is no error... but the longest requests does not even show. At this point I can think that the erroneous requests are stuck indefinitely and the request logs will not be shown at all.
Another test I made is reducing all the persistence-unit configuration, removing all the shared configuration, like follows
<properties>
<property name="eclipselink.connection-pool.default.initial" value="1" />
<property name="eclipselink.connection-pool.default.min" value="64" />
<property name="eclipselink.connection-pool.default.max" value="64" />
</properties>
But the error is still the same. So it is not related to a connection-pool sharing but the multi-thread itself
As ultimate test I tried to remove the multi-thread flow (each query result is processed one-by-one by the main thread) and leaving the shared-cache enabled.
This is working.
At this point I'm wondering... because the shared-cache is synchronized, so there is a "funnel" that block anyway the multi-thread process, should I use the monothread implementation anyway?
I'm seeing 100% utilisation of activemq's temp storage (configured to be 100mb), and the activemq client is blocking. This 100% usage remains permanently, and I have no idea what's going on
I have a camel route, which consumes from a queue (QUEUE.IN) using the JmsTransactionManager.
public final class RouteUnderTest extends RouteBuilder {
#Override
public void configure() throws Exception {
from("activemq-transacted:QUEUE.IN")
.bean(myBean)
.to("activemq:QUEUE.OUT");
}
}
While processing the message from this queue I'm invoking a spring-integration client (myBean) which is configured as follows
<int:gateway id="myBean" service-interface="MyBean">
<int:method name="request" request-channel="channel"/>
</int:gateway>
<int:chain input-channel="channel">
<int:transformer ref="transformedToJsonHere"/>
<jms:outbound-gateway request-destination-name="QUEUE.MYBEAN"
receive-timeout="5000"
explicit-qos-enabled="true"
time-to-live="5000"
delivery-persistent="false"/>
<int:transformer ref="transformedToAnObjectHere"/>
</int:chain>
My broker is configured to use LevelDB, and with the following usage limits:
<persistenceAdapter>
<levelDB directory="${activemq.data}/leveldb"/>
</persistenceAdapter>
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage percentOfJvmHeap="70"/>
</memoryUsage>
<storeUsage>
<storeUsage limit="500 mb"/>
</storeUsage>
<tempUsage>
<tempUsage limit="100 mb"/>
</tempUsage>
</systemUsage>
</systemUsage>
When my route consumes the message and then attempts to put a non-persistent message on QUEUE.OUT the client is blocked and my broker shows 100% usage of temp storage.
And I see the following activemq logs
2015-07-28 15:44:59,678 | INFO | Usage(default:temp:queue://QUEUE.MYBEAN:temp) percentUsage=0%, usage=104857600, limit=104857600, percentUsageMinDelta=1%;Parent:Usage(default:temp) percentUsage=100%, usage=104857600, limit=104857600, percentUsageMinDelta=1%: Temp Store is Full (0% of 104857600). Stopping producer (ID:orbit-vm-55561-1438094698190-1:1:3:1) to prevent flooding queue://QUEUE.MYBEAN. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 1s) | org.apache.activemq.broker.region.Queue | ActiveMQ NIO Worker 6
The queues look like (You can see that the QUEUE.IN message has been not been dequeued because it's still being processed transactionally, and no message has gone to QUEUE.MYBEAN)
I can fix this problem with any one of the following approaches:
Use KahaDB instead of LevelDB
Increase temp storage limit (150MB seems to do it but I haven't experimented a great deal)
Configure tempDataStore in activemq.xml (see below)
When configuring the tempDataStore it looks like:
<tempDataStore>
<bean xmlns="http://www.springframework.org/schema/beans" class="org.apache.activemq.leveldb.LevelDBStore">
<property name="directory" value="${activemq.data}/tmp" />
</bean>
</tempDataStore>
I should add, we were using KahaDB previously and this worked fine, but the upgrade to LevelDB has exposed this issue. Reverting to KahaDB is not an option.
I'm hoping someone could explain what we're seeing here, as the results are really difficult to understand. Why does using LevelDB necessitate a higher temp usage limit?, and why does configuring the tempDataStore explicitly also fix the problem?
I don't fully understand what's going on here so I'm worried that simply increasing the temp usage limit a little will just hide the problem until a later date.
Versions:
ActiveMQ: 5.11.1
Camel: 2.14.0
Spring: 4.0.8.RELEASE
Spring Integration: 4.0.5.RELEASE
We ran into exactly the same issue with ActiveMQ 5.13.2
The solution when using LevelDB is to explicitly configure a dedicated tempDataStore as you did.
If not, the broker uses the same store (LevelDB) for both persistent (persistent usage) and non-persistent messages (temp usage). You may therefore end-up in situations where the broker doesn't accept any non-persistent message anymore just because the store already holds persistent ones up to the configured tempUsage limit. It will however accept persistent ones if your storeUsage limit is set higher...
When using KahaDB, the broker automatically uses another store for the non-persistent messages (created in the tmp directory). So you don't have the problem...
Look at the following code for more indepth information: https://github.com/apache/activemq/blob/activemq-5.13.2/activemq-broker/src/main/java/org/apache/activemq/broker/BrokerService.java#L1739
When reading that code, remember LevelDBStore implements PListStore, but KahaDBStore doesn't...
I built an app with Silverlight4, RIA Services, and I'm using ASP.NET Membership for authentication/authorization.
My web.config has this:
<system.web>
<sessionState timeout="20"/>
<authentication mode="Forms">
<forms name="_ASPXAUTH" timeout="20"/>
</authentication>
I have read a number of different strategies on how to deal with auth/session timeout on the client side. That is: if the client is idle for x minutes (20 here), and then they do something with the UI that triggers a RIA/WCF call, I want to trap on that event and deal with appropriately (e.g. take them back to the login screen) -- in a nutshell: I need a way to differentiate from a bona-fide server side DomainException vs. an auth failure because the session timed out.
AFAIK: there is no typed exception or property that can determine this. The only way I've been able to determine this -- which seems like a hack: is to inspect the Error's Message string and look for something like "Access denied" or "denied". For example: something like this:
if (ex.Message.Contains("denied"))
// this is probably an auth failure b/c of a session timeout
So, this is what I'm currently doing, and it works if I run and debug either with the built-in server from VS2010, or if I run in localhost IIS. If I set the timeout to 1 minute, login, wait more than a minute and trigger another call, I breakpoint on the exception and enter the if code block above and all is well.
Then I deploy the app to a remote IIS7 server and I try the same test and it doesn't work. So, I added log tracing, and here's the event where the exception happened:
<E2ETraceEvent xmlns="http://schemas.microsoft.com/2004/06/E2ETraceEvent">
<System xmlns="http://schemas.microsoft.com/2004/06/windows/eventlog/system">
<EventID>131076</EventID>
<Type>3</Type>
<SubType Name="Error">0</SubType>
<Level>2</Level>
<TimeCreated SystemTime="2011-10-30T22:13:54.6425781Z" />
<Source Name="System.ServiceModel" />
<Correlation ActivityID="{20c26991-372f-430f-913b-1b72a261863d}" />
<Execution ProcessName="w3wp" ProcessID="4316" ThreadID="24" />
<Channel />
<Computer>TESTPROD-HOST</Computer>
</System>
<ApplicationData>
<TraceData>
<DataItem>
<TraceRecord xmlns="http://schemas.microsoft.com/2004/10/E2ETraceEvent/TraceRecord" Severity="Error">
<TraceIdentifier>http://msdn.microsoft.com/en-US/library/System.ServiceModel.Diagnostics.TraceHandledException.aspx</TraceIdentifier>
<Description>Handling an exception.</Description>
<AppDomain>/LM/W3SVC/1/ROOT/sla-2-129644844652558594</AppDomain>
<Exception>
<ExceptionType>System.ServiceModel.FaultException`1[[System.ServiceModel.DomainServices.Hosting.DomainServiceFault, System.ServiceModel.DomainServices.Hosting, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]], System.ServiceModel, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</ExceptionType>
<Message></Message>
<StackTrace>
at System.ServiceModel.DomainServices.Hosting.QueryOperationBehavior`1.QueryOperationInvoker.InvokeCore(Object instance, Object[] inputs, Object[]& outputs)
at System.ServiceModel.DomainServices.Hosting.DomainOperationInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs)
at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage31(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)
</StackTrace>
<ExceptionString>System.ServiceModel.FaultException`1[System.ServiceModel.DomainServices.Hosting.DomainServiceFault]: (Fault Detail is equal to System.ServiceModel.DomainServices.Hosting.DomainServiceFault).</ExceptionString>
</Exception>
</TraceRecord>
</DataItem>
</TraceData>
</ApplicationData>
</E2ETraceEvent>
The problem is that I don't have the string in the error message that indicates "denied" or "Access denied" - and I am unsure as to why this solution works in localhost IIS or VS2010 host but not in a remote IIS7 server. Is there some obscure configuration setting that I'm missing here? Is there a better way to do this in general?
You've probably gotten by this by now, but this article describes using the DomainOperationException and checking the error codes.
dex.ErrorCode == ErrorCodes.NotAuthenticated || dex.ErrorCode == ErrorCodes.Unauthorized
For convenient access (and in case the we loose access to the blog) here's the blog article by Josh Eastburn:
A question that comes up often from developers who are working with Silverlight and WCF RIA Services: why does my Silverlight application throw an exception when it has been idle for a period of time? As you might expect, it is due to the authenticated session timing out. But it isn’t quite that straightforward. Because Silverlight uses a client/server architecture, the client can operate independent of the server for an indefinite period of time. It is only when the Silverlight client makes a call to the server that the server-side timeout is realized. There are a few options to handle the client-server timeout issue (and you may be able to come up with a few more): If you aren’t concerned with the security implications of removing a session timeout, you can either increase the timeout setting in web.config, or create a DispatcherTimer in the Silverlight client that calls a simple method on the server to act as a "Keep Alive." Add a DispatcherTimer to the Silverlight client that stays in sync with the server-side timeout and warn/prompt the user keep the session active before the time expires or have them re-authenticate if it has already expired. However, this requires extra effort to keep the timers in sync when new server requests are made. Allow the server to handle the timeout as it normally would and handle the timeout gracefully on the Silverlight client. This means that the timeout is determined by server call activity, NOT activity confined the Silverlight client (i.e. accessing client-side data in the context). Of these three options, I find the third to be the best balance of security and usability while at the same time not adding unnecessary complexity to the application. In order to handle these server-side timeouts globally, you can add the following logic in either the Application_UnhandledException method in App.xaml.cs or in your global ViewModel loading construct if you have one:
// Check for Server-Side Session Timeout Exception
var dex = e.ExceptionObject as DomainOperationException;
if ((dex != null) && (dex.ErrorCode == ErrorCodes.NotAuthenticated || dex.ErrorCode == ErrorCodes.Unauthorized) && WebContext.Current.User.IsAuthenticated)
{
// A server-side timeout has occurred. Call LoadUser which will automatically
// authenticate if "Remember Me" was checked, or prompt for the user to log on again
WebContext.Current.Authentication.LoadUser(Application_UserLoaded, null);
e.Handled = true;
}
The following constants are defined within the ErrorCodes class:
public static class ErrorCodes
{
public const int NotAuthenticated = 0xA01;
public const int Unauthorized = 401;
}
When the server-side session times out, any subsequent calls will return a DomainOperationException. By inspecting the returned ErrorCode, you can determine if it is an authentication error and handle it accordingly. In my example, I am calling WebContext.Current.Authentication.LoadUser() which will attempt to re-authenticate the user if possible. Even if the user can not be automatically re-authenticated, it will call back to my Application_UserLoaded method. There I can check WebContext.Current.User.IsAuthenticated to determine whether to proceed with the previous operation or if I need to redirect back to the home page and reprompt for login. Here is an example of some code in the Appliation_UserLoaded callback that shows a login dialog if the user is not authenticated:
// Determine if the user is authenticated
if (!WebContext.Current.User.IsAuthenticated)
{
// Show login dialog automatically
LoginRegistrationWindow loginWindow = new LoginRegistrationWindow();
loginWindow.Show();
}
To test your code, you can set your timeout value in web.config to a
small value so timeouts occur quickly:
<authentication mode="Forms">
<forms name=".Falafel_ASPXAUTH" timeout="1" />
</authentication>
If you’d like to see all of this code in a working solution, check out our Silverlight RIA Template on CodePlex.