I am using version 2.3.7 for Java 6. I have set maximumPoolSize to 200 and connectionTimeout to 30 s. I run into SQLTimeoutExceptions from BaseHikariPool.getConnection in one of our load test cases involving 70 simultaneous users uploading 10 files each. I turned on debug logging and obtained pool stats. So it would seem that the pool isn't being exhausted. Rather, HikariCP takes longer than connectionTimeout to create new connections. How can I debug this part of the process? The underlying data source is SQLServerDataSource version 4.1.
connectionTimeout is the maximum time to wait for obtaining connection from pool.
it is NOT time out for creating connection from data source. there is none.
You may want to consider reducing pool size. begin load testing with minimum and gradually increasing till SqlServer begins to take much longer to create connection.
check about pool size
HTH
It might be because in HikariCP opening a connection is a blocking call (
https://github.com/brettwooldridge/HikariCP/issues/1287)
You might find this option useful com.zaxxer.hikari.blockUntilFilled. With this option connection pool will open minimumIdle connections in parallel during initialization instead of lazy initializing connections.
Related
I have configured openliberty (version 21) with a database (oracle) connection as follows in the server.xml :
<dataSource jndiName="jdbc/myds" transactional="true">
<connectionManager maxPoolSize="20" minPoolSize="5" agedTimeout="120s" connectionTimeout="10s"/>
<jdbcDriver libraryRef="jdbcLib" />
<properties.oracle URL="jdbc:oracle:thin:#..." user="..." password="..."/>
</dataSource>
The server starts and I can make queries to the database via my rest api but I have noticed that I only use 1 active database connection and parallel http queries result in queuing databases queries over that 1 connection.
I have verified this by monitoring the active open database connections in combination with slow queries (I make several rest calls in parallel). Only 1 connection is opened and 1 query is processes after the other. How do I open a connection pool with for example 5-20 connections for parallel operation.
Based on your described usage, the connection pool should be creating connections as requests come in if there are no connections available in the free pool.
Your connectionTimeout is configured to be 10 seconds. To ensure that your test really is running in parallel would be to make two requests to the server. The server should create a connection, use it, wait 11 seconds, then close the connection.
If your requests are NOT running in parallel, you will not get any exception since the second request won't start until after the first one finished and that would be an issue with your test procedure.
If your requests are running in parallel, and you do not get any exception output from Liberty. Then Liberty likely is making multiple connections and that can be confirmed by enabling J2C trace.
See: https://openliberty.io/docs/21.0.0.9/log-trace-configuration.html
Enable: J2C=ALL
If your requests are running in parallel, and no more than one connection is being created, then you will get a ConnectionWaitTimeoutException. This could be caused by the driver not being able to create more than one connection, incorrect use of the Oracle Connection Pool (UCP), or a number of other factors. I would need more information to debug that issue.
I wanted to know which approach is better, to have a single odbc connection shared among a series o threads, using a mutex to make sure that there are no concurrence problems, or to have one per thread and not use mutex. I don't see any substantial advantage in any of them, but maybe there are things in the odbc implementation i'm missing.
I tried to look for similar questions but couldn't find anything, sorry if it's been answered before
The ODBC Driver Manager already provides Connection Pooling: https://msdn.microsoft.com/en-us/library/ms716319(v=vs.85).aspx
Connection pooling enables an application to use a connection from a
pool of connections that do not need to be reestablished for each use.
In short: If a Connection is released by your application, and connection pooling is enabled, the driver manager will maybe not close the connection immediately, but keep it for some time open. If your application requests a new connection to be opened, the driver manager might return such an already open connection from its pool.
So: I would not care about do all the locking within your threads, but just enable connection pooling and let each driver use its own connection.
Note that you can either enable connection pooling on a process level or for each environment handle - see the remarks in the documentation:
Enables connection pooling by calling SQLSetEnvAttr to set the
SQL_ATTR_CONNECTION_POOLING environment attribute to
SQL_CP_ONE_PER_DRIVER or SQL_CP_ONE_PER_HENV.
I think that it depends on what you want to achieve with your implementation. Having only one shared connection with a lot of threads will result in threads being blocked and waiting for the connection to be available. This can decrease the performance. If you have a large number of threads, I'd rather create a sort of connection pool that handles the creation, usage and closure of connections.
I hope there is some expert on C3P0 that can help me answer the following question.
First, here's the general problem I'm trying to solve. We have an application connected to a database. When the database goes out, requests start taking several seconds to be processed, as opposed to a few milliseconds. This is because C3P0 will attempt to create new connections to the database. It will eventually timeout and the request will be rejected.
I came up with a proposal to fix it. Before grabbing a connection from the pool, I'll query C3P0's APIs to see if there are any connections in the pool. If there are none, we'll immediately drop the request. This way, our latency should remain in the milliseconds, instead of waiting until the timeout occurs. This solution works because C3P0 is capable of removing connections if it detects that they've gone bad.
Now, I set up a test with the values for "setTestConnectionOnCheckin" and "setTestConnectionOnCheckout" as "false". According to my understanding, this would mean that C3P0 would not test a connection (or, let's say, a connection in use, because there's also the idleConnectionTestPeriod setting). However, when I run my test, immediately after shutting off the database, C3P0 detects it and removes the connections from the pool. To give you a clearer picture, here's the execution's result:
14:48:01 - Request processed successfully. Processing time: 5 ms.
14:48:02 - Request processed successfully. Processing time: 4 ms.
14:48:03 - (Database is shut down at this point).
14:48:04 - java.net.ConnectException.
14:48:05 - Request rejected. Processing time: 258 ms.
14:48:06 - Request rejected. Processing time: 1 ms.
14:48:07 - Request rejected. Processing time: 1 ms.
C3P0 apparently knew that the database went down and removed the connections from the pool. It probably took a while, because the very first request after the database was shut off took longer than the others. I have run this test several times and that single request can take from 1 ms up to 3.5 seconds (which is the timeout time). This entry appears as many times as the number of connections I have defined for my pool. I have omitted all the rest for simplicity.
I think it's great that C3P0 is capable of removing the connections from the pool right away (well, as quickly as 258 ms. in the above example), but I'm having troubles explaining other people why that works. If "setTestConnectionOnCheckin" and "setTestConnectionOnCheckout" are set to "false", how is C3P0 capable of knowing that a connection went bad?
Even if they were set to "true", testing a connection is supposed to attempt executing a query on the database (something like "select 1 + 1 from dual"). We the database goes down, shouldn't the test timeout? In other words, shouldn't C3P0 take 3.5 seconds to determine that a connection has gone bad?
Thanks a lot, in advance.
(apologies... this'll be terse, i'm phonebound.)
1) even if no explicit Connection testing has been configured, c3p0 tests Connections that experience Exceptions while checked-out to determine whether they remain suitable for pooling.
2) a good JDBC driver will throw Exceptions quickly if the DBMS is unavailable. there's no reason why these internal Connection tests should be slow.
3) rather than polling for unused Connections to avoid waiting for checking / new acquisitions, you might consider just setting the config parameter checkoutTimeout.
good luck!
I was wondering what is maxPoolSize for? what is minPoolSize for?
How do I know which property value to use for my database?
EDITED
I am using Oracle 10g, connecting with Hibernate and bitronix on windows OS.
minPoolSize is the minimum number of database connections being opened all the time, even when none of them are used. maxPoolSize on the other hand represents the maximum number of concurrent connection. Now:
Use minPoolSize to always keep few connections opened (even when idle) so that you don't have to wait for establishing a new network connection when request arrives to the system with low load. This is basically a set of connections waiting "for emergency"
maxPoolSize is much more important. If the system is under heavy load and some request tries to open maxPoolSize + 1 connection, the connection pool will refuse, in turns causing the whole request to be discarded. On the other hand setting this parameter to to high value shift the bottleneck from your application to the database server, as it has limited capacity.
I was wondering if anyone knows how to set up connection pooling for Sitecore 6 running on SQLServer 2005 ?
And is this a good idea to setup on a Sitecore solution? Or probably more correct, will there be any problems if setup incorrectly?
Any other comments or tips about this are also greatly appreciated.
The answers from Sitecores support:
By default, connection pooling is
enabled in ADO.NET (it means that we
don't need any connection string
parameters to enable it, parameters
are needed only to disable or to tune
the connection pooling). Thats why
there are no connection pooling
parameters in the default connection
strings. For more information please
refer to the following MSDN articles:
http://msdn.microsoft.com/en-us/library/system.data.sqlclient.sqlconnection.connectionstring.aspx
http://msdn.microsoft.com/en-us/library/8xx3tyca.aspx
If you wish to tune the Database
Connection Pooling, you can add
appropriate parameters to the
connection strings.
Second reply after I asked some more about it:
Please see
http://msdn.microsoft.com/en-us/library/8xx3tyca%28v=vs.80%29.aspx
When a connection is first opened, a
connection pool is created based on an
exact matching algorithm that
associates the pool with the
connection string in the connection.
If MinPoolSize is either not specified
in the connection string or is
specified as zero, the connections in
the pool will be closed after a period
of inactivity. However, if the
specified MinPoolSize is greater than
zero, the connection pool is not
destroyed until the AppDomain is
unloaded and the process ends.
Maintenance of inactive or empty pools
involves minimal system overhead.
Following connection strings
parameters can be applied.
# Max Pool Size - specifies the
maximum size of your connection pool.
Default is 100. Most Web sites do not
use more than 40 connections under the
heaviest load but it depends on how
long your database operations take to
complete. # Min Pool Size - initial
number of connections that will be
added to the pool upon its creation.
Default is zero; however, you may
chose to set this to a small number
such as 5 if your application needs
consistent response times even after
it was idle for hours. In this case
the first user requests won't have to
wait for those database connections to
establish. # Pooling - controls if
your connection pooling on or off.
Default as you may've guessed is true.
I hope this helps others.
The only mentioning about negative impacts I'm aware about is this.