Getting below error from code deployed on WAS at:
org.springframework.messaging.MessageHandlingException:
error occurred in message handler [org.springframework.integration.aggregator.AggregatingMessageHandler#0];
nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException:
Could not get JDBC Connection;
nested exception is com.ibm.websphere.ce.cm.ConnectionWaitTimeoutException:
Connection not available, Timed out waiting for 180000
Detailed Trace:
Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is com.ibm.websphere.ce.cm.ConnectionWaitTimeoutException: Connection not available, Timed out waiting for 180000
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:630)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:695)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:727)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:752)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:762)
at org.springframework.integration.jdbc.JdbcMessageStore.getMessageGroup(JdbcMessageStore.java:431)
at org.springframework.integration.aggregator.AbstractCorrelatingMessageHandler.handleMessageInternal(AbstractCorrelatingMessageHandler.java:388)
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:78)
... 157 more
Caused by: com.ibm.websphere.ce.cm.ConnectionWaitTimeoutException: Connection not available, Timed out waiting for 180000
at com.ibm.ws.rsadapter.AdapterUtil.toSQLException(AdapterUtil.java:1684)
at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:686)
at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:636)
at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111)
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77)
... 165 more
Caused by: com.ibm.websphere.ce.j2c.ConnectionWaitTimeoutException: Connection not available, Timed out waiting for 180000
at com.ibm.ejs.j2c.FreePool.createOrWaitForConnection(FreePool.java:1729)
at com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:3329)
at com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:2610)
at com.ibm.ejs.j2c.ConnectionManager.allocateMCWrapper(ConnectionManager.java:1500)
at com.ibm.ejs.j2c.ConnectionManager.allocateConnection(ConnectionManager.java:1012)
at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:669)
... 168 more
Looking to your logs with the com.ibm.ejs.j2c.PoolManager, I'd recommend you to go to WAS support. Looks like your the Connection Pool is very small for your use-case. Especially having your clue:
It happens when i post too many requests on server.
Related
For a couple of month we have been sending reports to about 100 users.
We recently enabled MFA on our Office365 Tenant. I added the user sending reports to exceptions. But now the service only sends about 60 of the 100 emails. I have looked into the logs and cant find what the problem actually is.
I have added the error message below.
Than you!
emailextension!WindowsService_12!8850!01/03/2022-06:45:12:: e ERROR: Error sending email. Exception: System.AggregateException: One or more errors occurred. ---> System.Net.Mail.SmtpException: The server committed a protocol violation The server response was:
at System.Net.Mail.SendMailAsyncResult.End(IAsyncResult result)
at System.Net.Mail.SmtpClient.SendMailCallback(IAsyncResult result)
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at System.Threading.Tasks.Task.Wait(TimeSpan timeout)
at Microsoft.ReportingServices.EmailDeliveryProvider.EmailProvider.Deliver(Notification notification)
---> (Inner Exception #0) System.Net.Mail.SmtpException: The server committed a protocol violation The server response was:
at System.Net.Mail.SendMailAsyncResult.End(IAsyncResult result)
at System.Net.Mail.SmtpClient.SendMailCallback(IAsyncResult result)<---
. Additional Information: SmtpException StatusCode:GeneralFailure
notification!WindowsService_12!8850!01/03/2022-06:45:12:: e ERROR: Error occurred processing subscription bb732977-1cb9-4fef-8201-193e0e8b20b8: Failure sending mail: One or more errors occurred.Mail will not be resent.
I actually found the solution. For some reason part of the emails were being sent using TLS 1.0 and the rest were using TLS 1.2 I disabled TLS 1.0 through the registry and it went back to normal.
I have set up a flink standalone cluster, with one master and three slaves , all SESU Linux machines. In the master Dashboard http://flink-master:8081/ I can see 3 Task Managers and 3 task slots as I have set taskmanager.numberOfTaskSlots: 1 in flink-conf.yaml in all of the slaves.
When I run a flink built-in program,like the examples/streaming/Iteration.jar,I get exception often:
java.io.IOException: Connecting the channel failed: Connecting to remote task manager + 'ccr202/127.0.0.2:49651' has failed. This might indicate that the remote task manager has been lost.
at org.apache.flink.runtime.io.network.netty.PartitionRequestClientFactory$ConnectingChannel.waitForChannel(PartitionRequestClientFactory.java:197)
at org.apache.flink.runtime.io.network.netty.PartitionRequestClientFactory$ConnectingChannel.access$000(PartitionRequestClientFactory.java:132)
at org.apache.flink.runtime.io.network.netty.PartitionRequestClientFactory.createPartitionRequestClient(PartitionRequestClientFactory.java:84)
at org.apache.flink.runtime.io.network.netty.NettyConnectionManager.createPartitionRequestClient(NettyConnectionManager.java:59)
at org.apache.flink.runtime.io.network.partition.consumer.RemoteInputChannel.requestSubpartition(RemoteInputChannel.java:156)
at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.requestPartitions(SingleInputGate.java:480)
at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:502)
at org.apache.flink.streaming.runtime.io.BarrierTracker.getNextNonBlocked(BarrierTracker.java:93)
at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:214)
at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:69)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:264)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:718)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.flink.runtime.io.network.netty.exception.RemoteTransportException: Connecting to remote task manager + 'ccr202/127.0.0.2:49651' has failed. This might indicate that the remote task manager has been lost.
at org.apache.flink.runtime.io.network.netty.PartitionRequestClientFactory$ConnectingChannel.operationComplete(PartitionRequestClientFactory.java:220)
at org.apache.flink.runtime.io.network.netty.PartitionRequestClientFactory$ConnectingChannel.operationComplete(PartitionRequestClientFactory.java:132)
at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:424)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:268)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:284)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
... 1 more
Caused by: java.net.ConnectException: Connection refused: ccr202/127.0.0.2:49651
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.flink.shaded.netty4.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:224)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:281)
... 6 more
It seems that the network causes the problem,but sometimes the flink program can successfully finish.So what is the reason?
I also encounter this issue very frequently especially when there are many taskManagers. There are a few config I have tried to solve this issue. It's happened when the taskManager read the remote partition through netty connection. It timed out when request the connection. I increased the config "taskmanager.network.netty.server.numThreads", it solved the issue.
I'm using jmeter to stress test a GAE web service which uses CloudSQL and I'm getting intermittent communications link failure exceptions.
I've tried using direct connections and a connection pool, and I see exceptions in either scenario. The exceptions increase as the number of requests per second increase.
Note that we are using the highest tier of cloud sql, D32 and the tests are well under the max 3200 connections.
Here's a stack trace for reference:
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at sun.reflect.GeneratedConstructorAccessor48.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:33)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1117)
at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:350)
at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2413)
at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2450)
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2235)
at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:818)
at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)
at sun.reflect.GeneratedConstructorAccessor46.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:33)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:404)
at com.mysql.jdbc.GoogleNonRegisteringDriver$JdbcWrapper.getInstance(GoogleNonRegisteringDriver.java:276)
at com.mysql.jdbc.GoogleNonRegisteringDriver.connect(GoogleNonRegisteringDriver.java:246)
at java.sql.DriverManager.getConnection(DriverManager.java:571)
at java.sql.DriverManager.getConnection(DriverManager.java:215)
Update: I changed the connection pool settings to maxActive = 5 and maxIdle = 5 and the intermittent communications link exceptions went away. Note that I've tried commons dbcp and tomcat dbcp. I'm now seeing the following exceptions in the logs:
Caused by: java.sql.SQLException: java.lang.SecurityException: Unable to access gatherPerformanceMetrics
Caused by: java.sql.SQLException: java.lang.SecurityException: Unable to access includeThreadDumpInDeadlockExceptions
Caused by: java.sql.SQLException: java.lang.SecurityException: Unable to access nullNamePatternMatchesAll
From https://cloud.google.com/appengine/docs/java/cloud-sql/#Java_Size_and_access_limits
"Each App Engine instance cannot have more than 12 concurrent connections to a Google Cloud SQL instance."
Can you tell more about the test set-up? How many requests is jmeter sending to appengine and how many connections does the app instance open for each of those requests?
To everyone who are looking for why you might be getting "com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure" on a connection.
Make sure your IP is allowed if you are calling from a test server!
I was testing at a friends house, and this unhelpful error kept showing up.
We have an application using a Weblogic pooled non-XA SQL Server JDBC driver. Here is the flow of what is causing the issue.
Get connection from datasource.
Set auto commit to false.
Create CallableStatement.
Set parameters.
Call executeUpdate.
Inside stored proc call, the database raises an error with SET XACT_ABORT as ON
Application sees error code in out param of CallableStatement.
Closes CallableStatement
Calls rollback on the connection.
Exception is thrown:
2013-Oct-17 16:45:20.500
EDT||XXXXXXX|asyncDelivery9|XXX|XXX|XXXX-a3436e48-1672-4a26-88c0-50b9954f6a42|ERROR|XXXXX|rollBack|The
server failed to resume the transaction. Desc:4200000012.
com.microsoft.sqlserver.jdbc.SQLServerException: The server failed to
resume the transaction. Desc:4200000012.
at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:216)
~[sqljdbc4.jar:na]
at com.microsoft.sqlserver.jdbc.TDSTokenHandler.onEOF(tdsparser.java:254)
~[sqljdbc4.jar:na]
at com.microsoft.sqlserver.jdbc.TDSParser.parse(tdsparser.java:84)
~[sqljdbc4.jar:na]
at com.microsoft.sqlserver.jdbc.TDSParser.parse(tdsparser.java:39)
~[sqljdbc4.jar:na]
at com.microsoft.sqlserver.jdbc.SQLServerConnection$1ConnectionCommand.doExecute(SQLServerConnection.java:1756)
~[sqljdbc4.jar:na]
at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:5696)
~[sqljdbc4.jar:na]
at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:1715)
~[sqljdbc4.jar:na]
at com.microsoft.sqlserver.jdbc.SQLServerConnection.connectionCommand(SQLServerConnection.java:1761)
~[sqljdbc4.jar:na]
at com.microsoft.sqlserver.jdbc.SQLServerConnection.rollback(SQLServerConnection.java:1964)
~[sqljdbc4.jar:na]
at weblogic.jdbc.wrapper.PoolConnection_com_microsoft_sqlserver_jdbc_SQLServerConnection.rollback(Unknown
Source) ~[wlfullclient.jar:12.1.1.0]
Connection is closed.
Then the next time the application attempts to get a connection from the data source, the following error is thrown:
java.sql.SQLException: Failed to setAutoCommit to true for pool connection: First we got The server failed to resume the transaction. Desc:4200000012., then we got The server failed to resume the transaction. Desc:4200000012.
at weblogic.jdbc.wrapper.PoolConnection.init(PoolConnection.java:70) ~[wlfullclient.jar:12.1.1.0]
at weblogic.jdbc.common.internal.RmiDataSource.getPoolConnectionObj(RmiDataSource.java:639) ~[wlfullclient.jar:12.1.1.0]
at weblogic.jdbc.common.internal.RmiDataSource.getPoolConnection(RmiDataSource.java:478) ~[wlfullclient.jar:12.1.1.0]
at weblogic.jdbc.common.internal.RmiDataSource.getConnectionInternal(RmiDataSource.java:558) ~[wlfullclient.jar:12.1.1.0]
at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:518) ~[wlfullclient.jar:12.1.1.0]
at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:511) ~[wlfullclient.jar:12.1.1.0]
In the Weblogic log, the following is logged:
<Oct 15, 2013 3:07:50 PM EDT> <Warning> <JDBC> <BEA-001153> <Forcibly releasing inactive/harvested connection "weblogic.jdbc.wrapper.PoolConnection_com_microsoft_sqlserver_jdbc_SQLServerConnection#238607" back into the data source connec
tion pool "MASTERDB", currently reserved by: reclaimed because of init failure before user access. Not because of inactivity..>
It seems to me that the transaction on the connection was never ended. But a rollback was called which I would have thought would end the transaction. And now the next time Weblogic attempts to return the connection, it errors when initializing it. Any ideas as to what the issue is? We are not explicitly setting auto commit back to true at the end, could that be the issue?
Found this suggestion for same similar problem here.
"Make sure you have the datasource/pool configured to test-on-reserve, and seconds-to-trust = 0."
This should solve the issue of eliminating those "bad" connections from pool before returning to application.
I have a multi-threaded application in solrj 4. The threads (max 25) share one connection to HttpSolrServer. Each thread is running one query. This worked fine for a while, until it finally crashed with the following messages:
Jan 12, 2013 12:52:15 PM org.apache.http.impl.client.DefaultRequestDirector tryExecute
INFO: Retrying request
Jan 12, 2013 12:52:15 PM org.apache.http.impl.client.DefaultRequestDirector tryExecute
INFO: I/O exception (java.net.SocketException) caught when processing request: Connection reset
I would like to catch this exception and reset the connection to the server. I don't get the whole stack trace with the above message, so I'm not sure where to do this. The only place I reference the server is for making a query with:
server.query( query )
But the only exception this throws is SolrServerException, which I'm currently handling.
Any suggestions would be greatly appreciated.
FYI, this is how I'm setting up the initial server connection:
server = new HttpSolrServer(url);
server.setSoTimeout(0);
server.setDefaultMaxConnectionsPerHost(50);
server.setMaxTotalConnections(128);