I hope there is some expert on C3P0 that can help me answer the following question.
First, here's the general problem I'm trying to solve. We have an application connected to a database. When the database goes out, requests start taking several seconds to be processed, as opposed to a few milliseconds. This is because C3P0 will attempt to create new connections to the database. It will eventually timeout and the request will be rejected.
I came up with a proposal to fix it. Before grabbing a connection from the pool, I'll query C3P0's APIs to see if there are any connections in the pool. If there are none, we'll immediately drop the request. This way, our latency should remain in the milliseconds, instead of waiting until the timeout occurs. This solution works because C3P0 is capable of removing connections if it detects that they've gone bad.
Now, I set up a test with the values for "setTestConnectionOnCheckin" and "setTestConnectionOnCheckout" as "false". According to my understanding, this would mean that C3P0 would not test a connection (or, let's say, a connection in use, because there's also the idleConnectionTestPeriod setting). However, when I run my test, immediately after shutting off the database, C3P0 detects it and removes the connections from the pool. To give you a clearer picture, here's the execution's result:
14:48:01 - Request processed successfully. Processing time: 5 ms.
14:48:02 - Request processed successfully. Processing time: 4 ms.
14:48:03 - (Database is shut down at this point).
14:48:04 - java.net.ConnectException.
14:48:05 - Request rejected. Processing time: 258 ms.
14:48:06 - Request rejected. Processing time: 1 ms.
14:48:07 - Request rejected. Processing time: 1 ms.
C3P0 apparently knew that the database went down and removed the connections from the pool. It probably took a while, because the very first request after the database was shut off took longer than the others. I have run this test several times and that single request can take from 1 ms up to 3.5 seconds (which is the timeout time). This entry appears as many times as the number of connections I have defined for my pool. I have omitted all the rest for simplicity.
I think it's great that C3P0 is capable of removing the connections from the pool right away (well, as quickly as 258 ms. in the above example), but I'm having troubles explaining other people why that works. If "setTestConnectionOnCheckin" and "setTestConnectionOnCheckout" are set to "false", how is C3P0 capable of knowing that a connection went bad?
Even if they were set to "true", testing a connection is supposed to attempt executing a query on the database (something like "select 1 + 1 from dual"). We the database goes down, shouldn't the test timeout? In other words, shouldn't C3P0 take 3.5 seconds to determine that a connection has gone bad?
Thanks a lot, in advance.
(apologies... this'll be terse, i'm phonebound.)
1) even if no explicit Connection testing has been configured, c3p0 tests Connections that experience Exceptions while checked-out to determine whether they remain suitable for pooling.
2) a good JDBC driver will throw Exceptions quickly if the DBMS is unavailable. there's no reason why these internal Connection tests should be slow.
3) rather than polling for unused Connections to avoid waiting for checking / new acquisitions, you might consider just setting the config parameter checkoutTimeout.
good luck!
Related
The docs describe avg_wait_time as:
Time spent by clients waiting for a server in microseconds (average per second).
We see occasional spikes in avg_wait_time (normally it's 0). During those spikes, best I can tell, there are available/idle servers. What could be causing the wait time to be greater than 0 in these cases?
Reading the flow from hackernoon leads to that your connection pools are exhausted, and the new connections need to wait until a free spot becomes available for either connecting to a pool, either to get to the executing phase.
These server connections, that clients get linked to, are
“pooled” — limited in number and reused. Because of that it might
occur that while client sends some request (beginning a transaction or
performing a query) a corresponding server connection pool is
exhausted, i.e. pgbouncer oppened as many connections as were allowed
it and all of them are occupied by (linked to) some other clients.
PgBouncer in this scenario puts client into a queue, and this client’s
connection goes to a CL_WAITING state. This might happen as well while
client only logging in, so there’s CL_WAITING_LOGIN for that also:
I am using version 2.3.7 for Java 6. I have set maximumPoolSize to 200 and connectionTimeout to 30 s. I run into SQLTimeoutExceptions from BaseHikariPool.getConnection in one of our load test cases involving 70 simultaneous users uploading 10 files each. I turned on debug logging and obtained pool stats. So it would seem that the pool isn't being exhausted. Rather, HikariCP takes longer than connectionTimeout to create new connections. How can I debug this part of the process? The underlying data source is SQLServerDataSource version 4.1.
connectionTimeout is the maximum time to wait for obtaining connection from pool.
it is NOT time out for creating connection from data source. there is none.
You may want to consider reducing pool size. begin load testing with minimum and gradually increasing till SqlServer begins to take much longer to create connection.
check about pool size
HTH
It might be because in HikariCP opening a connection is a blocking call (
https://github.com/brettwooldridge/HikariCP/issues/1287)
You might find this option useful com.zaxxer.hikari.blockUntilFilled. With this option connection pool will open minimumIdle connections in parallel during initialization instead of lazy initializing connections.
We are trying to optimize the performance of our site. We are using glimpse for profiling and debugging. We noticed that the "connection opened" time in glimpse timeline is much higher when compared to the command execution time.
Load and number of connections in DB server seems to be normal. So we couldn't nail down the root cause.
As per Glimpse Timeline, the connection was opened for 381 ms. But the command got executed in 6 ms.
Total query execution time - 430 ms
Total connection open time - 19130 ms
Any idea why this happens?
Note :
Scope of DB Context is the scope of Http request
Number of records returned by these queries are < 1000 (approx)
.Net uses connection pooling and doesn't close connections. The only thing that matters is how long it took the EF method to return back to the caller. Do you have the DBContext instance in a "using" block to let .Net know to release it?
I've recently switched to running my development environment over our company's VPN using NetExtender. It would now seem that my database driven applications are now timing out the first time they try to hit the database. After the timeout (30 sec or so) and an additional 5-10 seconds, all DB calls succeed. During the 5-10 seconds the timeout error response is sent immediately. It seems to be related to when SQL Server needs to create a new database session for me. Each time I need to be assigned a new client process ID, I timeout. This is a huge problem when using Resharper + NUnit as a test harness as each time the tests are run, a new instance of resharper's unit test runner is created thusly causing me to timeout. Server timeout seems to be in the area of 30 seconds, which is certainly generous enough for a connection to be established.
It sounds to me like it could be a DNS issue. If the primary DNS is not properly configured and is inaccessible from the VPN client, it will timeout and pass on to the secondary.
Additionally, some VPNs allow you to access some local resources - this could put the DNS on your own, local network in play.
I think I'd try changing the DNS-order and see if that did the trick.
We have a java server connecting to a MySQL 5 database usingHibernate as our persistence layer which is using c3p0 for DB connection pooling.
I've tried following the c3p0 and hibernate documentation:
Hibernate - HowTo Configure c3p0 connection pool
C3P0 Hibernate properties
C3P0.properties configuration
We're getting an error on our production servers stating that:
... Caused by:
com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException:
No operations allowed after connection
closed.Connection was implicitly
closed due to underlying
exception/error:
BEGIN NESTED EXCEPTION
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException
MESSAGE: The last packet successfully
received from the server was45000
seconds ago.The last packet sent
successfully to the server was 45000
seconds ago, which is longer than the
server configured value of
'wait_timeout'. You should consider
either expiring and/or testing
connection validity before use in your
application, increasing the server
configured values for client timeouts,
or using the Connector/J connection
property 'autoReconnect=true' to avoid
this problem.
STACKTRACE:
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
The last packet successfully received
from the server was45000 seconds
ago.The last packet sent successfully
to the server was 45000 seconds ago,
which is longer than the server
configured value of 'wait_timeout'.
You should consider either expiring
and/or testing connection validity
before use in your application,
increasing the server configured
values for client timeouts, or using
the Connector/J connection property
'autoReconnect=true' to avoid this
problem.
We have our c3p0 connection pool properties setup as follows:
hibernate.c3p0.max_size=10
hibernate.c3p0.min_size=1
hibernate.c3p0.timeout=5000
hibernate.c3p0.idle_test_period=300
hibernate.c3p0.max_statements=100
hibernate.c3p0.acquire_increment=2
The default MySQL wait_timetout is set to 28800 seconds (8 hours), the reported error is saying that it's been over 45000 seconds (about 12.5 hours). Although the c3p0 configuration states that it will "timeout" idle connections that haven't been used after 5000 seconds and it will check every 300 seconds, thus an idle connection should never live longer than 5299 seconds right?
I've tested locally by setting my developer MySQL (my.ini on windows, my.cnf on Unix) wait_timeout=60 and lowering the c3p0 idle timeout values below 60 seconds, and it will properly timeout idle connections and create new ones. I also check to ensure that we're not leaking DB connections and holding onto a connection, and it doesn't appear we are.
Here's the c3p0.properties file I'm using to test in my developer environment to ensure c3p0 is properly handling connections.
hibernate.properties (testing with MySQL wait_timeout=60)
hibernate.c3p0.max_size=10
hibernate.c3p0.min_size=1
hibernate.c3p0.timeout=20
hibernate.c3p0.max_statements=100
hibernate.c3p0.idle_test_period=5
hibernate.c3p0.acquire_increment=2
c3p0.properties
com.mchange.v2.log.FallbackMLog.DEFAULT_CUTOFF_LEVEL=ALL
com.mchange.v2.log.MLog=com.mchange.v2.log.FallbackMLog
c3p0.debugUnreturnedConnectionStackTraces=true
c3p0.unreturnedConnectionTimeout=10
Make sure that c3p0 really is starting by examine the log. I, for some reason, had two versions of hibernate (hibernate-core3.3.1.jar and hibernate-3.2.6GA.jar) on my classpath. I also used hibernate annotatations version 3.4.0GA which is not compatible with 3.2.x. (dont know if that had something to do with the original problem).
After removal of one of the hibernate jar's (cant remember which i deleted, probably hibernate-3.2.6GA.jar) c3p0 finally started and i got rid of the annoying com.mysql.jdbc.exceptions.jdbc4.CommunicationsException that happend efter 8h inactivity.