We're using glassfish 3.1.2 and it seems very brittle in terms of maintaining datasource connection pools through a database server restart or a break in network connectivity with the database (even a slight one).
It doesn't seem like it's particularly adept and re-establishing those connections and moving on.
We're using com.mysql.jdbc.jdbc2.optional.MysqlDataSource.
We're using connection validation.
Is there something else we're missing? Does anyone else experience this?
Same here.
Have you tried to configure the JDBC connection with autoreconnect?
Example: jdbc:mysql://localhost:3306/foo?autoReconnect=true
Related
Can someone tell me why the code below performs very slow?
The last sentence takes more than 6 seconds.
I am trying to read data from a SQL server with C++.
std::string connectString("Provider=SQLOLEDB; Data Source=XXX;Initial Catalog=YYY;Integrated Security=SSPI;");
_ConnectionPtr Connection;
CoInitialize(NULL);
pConnection.CreateInstance(__uuidof(Connection));
HRESULT hr=Connection->Open(connectString.c_str(),"","",adConnectUnspecified);
Adding the portnumber did the trick!
std::string connectString("Provider=SQLOLEDB; Data Source=XXX,1430;Initial Catalog=YYY;Integrated Security=SSPI;");
Here are some points to check:
Between your datasource and your application lies a network connection, e.g. a SMB share. These tend to be slow.
Try to find out if some registry settings work against you, forcing too small net packets or other kinds of throttles. These things depend on the framework version.
A virus scanner wants to be your friend.
Here is an example how the firewall can disturb: Very slow connection to SQL Server 2005 only if using ADO.NET with SqlClient
As the firewall blocks the requests, the system chooses some pipe streaming after a timeout.
I hope one of those will help you :-)
For me the answer was to enable TCP/IP using Sql Server Configuration Manager (I'm not sure how it managed to connect before; only Shared Memory was enabled ¯\_ (ツ)_/¯ )
I have 3 servers set up for SQL mirroring and automatic failover using a witness server. This works as expected.
Now my application that connects to the database, seems to have a problem when a failover occurs - I need to manually intervene and change connection strings for it to connect again.
The best solution I've found so far involves using Failover Partner parameter of the connection string, however it's neither intuitive nor complete: Data Source="Mirror";Failover Partner="Principal" found here.
From the example in the blog above (scenario #3) when the first failover occurs, and principal (failover partner) is unavailable, data source is used instead (which is the new principal). If it fails again (and I only tried within a limited period), it then comes up with an error message. This happens because the connection string is cached, so until this is refreshed, it will keep coming out with an error (it seems connection string refreshes ~5 mins after it encounters an error). If after failover I swap data source and failover partner, I will have one more silent failover again.
Is there a way to achieve fully automatic failover for applications that use mirroring databases too (without ever seeing the error)?
I can see potential workarounds using custom scripts that would poll currently active database node name and adjust connection string accordingly, however it seems like an overkill at the moment.
Read the blog post here
http://blogs.msdn.com/b/spike/archive/2010/12/15/running-a-database-mirror-setup-with-the-sqlbrowser-service-off-may-produce-unexpected-results.aspx
It explains what is happening, the failover partner is actually being read from the sql server not from your config. Run the query in that post to find out what is actually being used as the failover server. It will probably be a machine name that is not discoverable from where your client is running.
You can clear the application pool in the case a failover has happened. Not very nice I know ;-)
// ClearAllPools resets (or empties) the connection pool.
// If there are connections in use at the time of the call,
// they are marked appropriately and will be discarded
// (instead of being returned to the pool) when Close is called on them.
System.Data.SqlClient.SqlConnection.ClearAllPools();
We use it when we change an underlying server via SQL Server alias, to enforce a "refresh" of the server name.
http://msdn.microsoft.com/en-us/library/system.data.sqlclient.sqlconnection.clearallpools.aspx
The solution is to turn connection pooling off Pooling="false"
Whilst this has minimal impact on small applications, I haven't tested it with applications that receive hundreds of requests per minute (or more) and not sure what the implications are. Anyone care to comment?
Try this connectionString:
connectionString="Data Source=[MSSQLPrincipalServerIP,MSSQLPORT];Failover Partner=[MSSQLMirrorServerIP,MSSQLPORT];Initial Catalog=DatabaseName;Persist Security Info=True;User Id=userName; Password=userPassword.; Connection Timeout=15;"
If you are using .net development, you can try to use ObjAdoDBLib or PigSQLSrvLib and PigSQLSrvCoreLib, and the code will become simple.
Example code:
New object
ObjAdoDBLib
Me.ConnSQLSrv = New ConnSQLSrv(Me.DBSrv, Me.MirrDBSrv, Me.CurrDB, Me.DBUser, Me.DBPwd, Me.ProviderSQLSrv)
PigSQLSrvLib or PigSQLSrvCoreLib
Me.ConnSQLSrv = New ConnSQLSrv(Me.DBSrv, Me.MirrDBSrv, Me.CurrDB, Me.DBUser, Me.DBPwd)
Execute this method to automatically connect to the online database after the mirror database fails over.
Me.ConnSQLSrv.OpenOrKeepActive
For more information, see the relevant links.
https://www.nuget.org/packages/ObjAdoDBLib/
https://www.nuget.org/packages/PigSQLSrvLib/
https://www.nuget.org/packages/PigSQLSrvCoreLib/
We have a very strange intermittent issue which has started coming up over the last month or so whereby some connections to mssql server fail with the error:
System.Data.SqlClient.SqlException: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server)
The error does not bring down the site, nor does it require a db restart - if you simply rerun the same query will work the second time. This means a lot of users will hit an error every now and then and have to refresh the error page for things to work.
Now, my initial knee-jerk reaction was this could be due to:
Resource related issue - so I started running SQL profiler and perfmon, but did not find any issues with the serve struggling to keep up with the number of connections / sec. I've been looking at MSSQL:SQL Errors, MSSQL:Wait Statistics, MSSQL:Exec Statistics, MSSQL:Locks. Does anyone have any guidance on other stats I should be poking and prodding here?
Unclosed DB connections - I ruled this one out after going through all the data-tier code. We have all the fail safes in place to stop this from happening.
Connection / Network related issue: our SQL server sits on a separate server (MS SQL Server Standard 2008) to our application server (running ASP.Net on IIS7) - both servers run on xlarge Amazon EC2 instances with all security policies configured (as per Amazons direction). Anyone got guidance on how to test the connectivity between the two servers or if this could be the issue?
Is it a possible issue with the IIS connection string? I have not tested this but should we be fully qualifying the server with the computer name we are connecting to (just thought of it)? We use a connection string in the format: server=xxxxx;Database=xxxx;uid=xxxx;password=xxx;
Your thoughts and insight is very much appreciated!
Thanks in advance
Solved. After testing almost every possible performance metric and examining every piece of code, I discovered that the error was caused by a bit of deprecated database code. The main issue was being caused by code using:
SqlConnection.ClearPools;
For future reference, any other developers looking to debug their code and manage connection pools, an excellent resource can be found here: http://www.codeproject.com/KB/dotnet/ADONET_ConnectionPooling.aspx
Try changing the connection string to the FQDN+port
server=xxxxx.domain.tld,1234;
Note: you don't need any instance name if you use port
On our global corporate intranet... we had a similar issue that happened to remote clients: more often if they were further away, never in the same building as the server.
After some poking around, chatting to the DBAs and MS, it was said to be caused by timing/Kerberos/too many firewalls etc. Adding FQDN+port removed all our issues.
It may be solved by switching to TCP/IP instead of Named Pipes, if you can.
Perhaps you can test this by changing the server name to the server IP address.
I use server=tcp:servername in my connection string to force TCP.
KB313295
It seems like connection are not being closed correctly, and after some time you can't open any more new connections. As the total allowed connections to database is a constant digit.
If you are using C#/VB.net
Are you using "Using" statements to open the connections ?
using (System.Data.SqlClient.SqlConnection con = new SqlConnection("YourConnection string"))
{
con.Open();
}
Context: The Cloud
We have a java-based web application that we normally host on our own servers. Recently we used Amazon Web Services (AWS EC2) cloud to host an instance.
This "cloud setup" matches our typical "on site" setup: one server for the app server, another server for the database server. (Several app servers point to the same database server)
The problem
In this cloud setup, we receive intermittent "connection reset by peer errors" between the database and the jdbc driver, where at (seemingly) random intervals and at random points in the codebase, the database connection fails.
Here are a few error excerpts for the log
Stack Trace Example 1:
at com.participate.pe.genericdisplay.client.taglib.GenDisplayViewTag.doStartTag(GenDisplayViewTag.java:77)
... 75 more
Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: The connection is closed.
at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDriverError(SQLServerException.java:170)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.checkClosed(SQLServerConnection.java:304)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.getMetaData(SQLServerConnection.java:1734)
at org.jboss.resource.adapter.jdbc.WrappedConnection.getMetaData(WrappedConnection.java:354)
Stack Trace Example 2
at java.lang.Thread.run(Thread.java:619)
Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: Connection reset
at com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:1368)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:1355)
at com.microsoft.sqlserver.jdbc.TDSChannel.read(IOBuffer.java:1532)
at com.microsoft.sqlserver.jdbc.TDSReader.readPacket(IOBuffer.java:3274)
at com.microsoft.sqlserver.jdbc.TDSCommand.startResponse(IOBuffer.java:4437)
at com.microsoft.sqlserver.jdbc.TDSCommand.startResponse(IOBuffer.java:4389)
at com.microsoft.sqlserver.jdbc.SQLServerConnection$1ConnectionCommand.doExecute(SQLServerConnection.java:1457)
at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:4026)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:1416)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.connectionCommand(SQLServerConnection.java:1462)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.setAutoCommit(SQLServerConnection.java:1610)
at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnection.checkTransaction(BaseWrapperManagedConnection.java:429)
Technical Environment
Jboss 4.2.2.GA (Jboss-Web 2.0/ Tomcat 6)
MSSQL 2005 2.0 jdbc driver
Some points
We have never seen this problem in
our own environment (i.e. own data centers) running the application for several years
This led me to conclude "something funny is going on with Amazon network environment". I may be wrong/missing something/etc.
This problem only occurs with our application. We have other java and php applications which have not had this problem. The other java application uses a different jdbc driver (jtds, afaik)
It doesn't seem like a simple connection timeout
Questions
-Has anyone seen this before?
-If it's an EC2 "known issue", can we configure our way around the problem (i.e. make sure everything is on its own subnet or virtual private cloud (vpc) ?
-Any jdbc driver settings to get past this problem?
** Update **
I've extended and increased the bounty on this question.
On extra bit of information: the two virtual servers (database and application server) were on different subnets--i.e. one hop between the two servers.
In a non-cloud environment we have "zero hops" bewtewn the two servers.
Our hosting admins said we had no control over the subnets of our EC2 instances. This made me wonder if virtual private cloud would help.
thanks in advance
will
Not sure if this is related or not. We experienced something similar with an app that we were running in the EC2 environment. Same symptom, that the database connection would intermittently close. We were using MSSQL 1.2 driver. Also, we would see the errors usually after a delay or idle time with the connection. Our assumption (never proven) was that something in the network layer was closing the connection and the client wasn't detecting it, so it became stale.
We were able to work around it because we were using commons connection pools, and had the pool recreate the connection on failure. We eventually moved the application out of EC2 and didn't see the issue again.
Just a word of caution on usind DBCP/connection pool features to mitigate the issue - the more you enable 'testOnBorrow' and other features, the more you can introduce latency or other performance changing affects on the system. I don't know if DBCP still does this or not, but a few years ago it would generate actual test queries to test the connection - full stack, database responses - not just at the network layer. The above link from Brian brings back horrific memories from the early 2000s on surrounding re-try logic for JDBC connection management.
Anyway, it's tough to really root cause this, other than gather evidence and eliminate the 'seemingly random' to a specific set of conditions:
You could try to throw up a Wireshark/PCAP trace, find when it happens, and send the results to both Amazon and Microsoft to see if they can root cause it
You could try the above with certain test harnesses to isolate the problem (JMeter tests to get concurrency up), bounce the network connection, watch for recovery, etc
You could try alternative versions of SQL Server to discount a SQL Server/JDBC driver bug that has since been fixed.
If DNS is used in connection strings, could use IP addresses to validate nslookup issues
I'm not a SQL Server expert, but another route for research could be within the related products domain - e.g. see if anyone experienced similar issues with TFS/Sharepoint (e.g. such as http://nickhoggard.wordpress.com/2009/12/07/further-experiences-with-tfs-2010-beta-2-on-amazon-ec2/ )
I have seen this issue in both the EC2 environment and the Windows Azure environment. I think connection retry logic needs to be a standard part of your design when working in a distributed computing environment.
This article is for SQL Azure - but I think it equally applies to EC2 and all drivers.
I can also confirm that this happens and will spin up a lower priority investigation since it's not production critical.
Our production servers are in our data center. We use developer laptops to run our applications. Neither of these get this issue once we configured c3p0 connection pool timeouts and test period (see article: http://www.codefin.net/2007/05/hibernate-and-mysql-connection-timeouts.html).
However, we do have a development staging server that is in EC2 and it does indeed happen there. If I find something that seems to work, I'll ping back. Also, I'm using mysql. I see that you are using MS SQL Server so it is across database vendors.
// MS SQL Server 2008.
My application has several different connections to sql server database (C#, ADO .NET). Each connection uses the same user name and password. Some connections may execute simultaneously.
Is there any problem with that ? Should I add some settings to support that functionality ?
Is there any settings in connection string that allow/restrict simultaneous connections ? Is there any settings in SQL Server that allow/restrict that functionality ?
No this is normal. Using the exact same connection string also helps with pooling the connections (In the background your connections to the DB are kept open and reused rather than completely closing and opening them).
No. Absolutely not. I is quite normal (in many scenarios you use an app server that uses a specific logon on the sql server - not one per user) and may have thousand of connections open at the same time.
There is no restrictun, no setting. it is considered normal use.
MS SQL Server is designed to handle simultaneous connections, no matter which user they come from. I can't see how you'd have a problem, unless the connections are working on the same data.