I've recently switched to running my development environment over our company's VPN using NetExtender. It would now seem that my database driven applications are now timing out the first time they try to hit the database. After the timeout (30 sec or so) and an additional 5-10 seconds, all DB calls succeed. During the 5-10 seconds the timeout error response is sent immediately. It seems to be related to when SQL Server needs to create a new database session for me. Each time I need to be assigned a new client process ID, I timeout. This is a huge problem when using Resharper + NUnit as a test harness as each time the tests are run, a new instance of resharper's unit test runner is created thusly causing me to timeout. Server timeout seems to be in the area of 30 seconds, which is certainly generous enough for a connection to be established.
It sounds to me like it could be a DNS issue. If the primary DNS is not properly configured and is inaccessible from the VPN client, it will timeout and pass on to the secondary.
Additionally, some VPNs allow you to access some local resources - this could put the DNS on your own, local network in play.
I think I'd try changing the DNS-order and see if that did the trick.
Related
i am using sql server 2012 with desktop application as client , the application get errors after period of time when no activity on it , i googled about this issue all solutions points me to AUTO_CLOSE option on database but it's already set to false .
i thing is something missing in connection string (ADO Extension)
To be honest, if you have long running connections, you can hit these errors regardless due to firewalls / routers closing connections, etc. The correct solution is to instantiate a connection when you need it, use the connection and release it. With connection pooling, this is not really a performance problem.
If your long-running application is "bursty", it is sometimes convenient to open the connection, do a number of commands -- then when you go idle, release the connection and wait the next burst of activity.
I hope there is some expert on C3P0 that can help me answer the following question.
First, here's the general problem I'm trying to solve. We have an application connected to a database. When the database goes out, requests start taking several seconds to be processed, as opposed to a few milliseconds. This is because C3P0 will attempt to create new connections to the database. It will eventually timeout and the request will be rejected.
I came up with a proposal to fix it. Before grabbing a connection from the pool, I'll query C3P0's APIs to see if there are any connections in the pool. If there are none, we'll immediately drop the request. This way, our latency should remain in the milliseconds, instead of waiting until the timeout occurs. This solution works because C3P0 is capable of removing connections if it detects that they've gone bad.
Now, I set up a test with the values for "setTestConnectionOnCheckin" and "setTestConnectionOnCheckout" as "false". According to my understanding, this would mean that C3P0 would not test a connection (or, let's say, a connection in use, because there's also the idleConnectionTestPeriod setting). However, when I run my test, immediately after shutting off the database, C3P0 detects it and removes the connections from the pool. To give you a clearer picture, here's the execution's result:
14:48:01 - Request processed successfully. Processing time: 5 ms.
14:48:02 - Request processed successfully. Processing time: 4 ms.
14:48:03 - (Database is shut down at this point).
14:48:04 - java.net.ConnectException.
14:48:05 - Request rejected. Processing time: 258 ms.
14:48:06 - Request rejected. Processing time: 1 ms.
14:48:07 - Request rejected. Processing time: 1 ms.
C3P0 apparently knew that the database went down and removed the connections from the pool. It probably took a while, because the very first request after the database was shut off took longer than the others. I have run this test several times and that single request can take from 1 ms up to 3.5 seconds (which is the timeout time). This entry appears as many times as the number of connections I have defined for my pool. I have omitted all the rest for simplicity.
I think it's great that C3P0 is capable of removing the connections from the pool right away (well, as quickly as 258 ms. in the above example), but I'm having troubles explaining other people why that works. If "setTestConnectionOnCheckin" and "setTestConnectionOnCheckout" are set to "false", how is C3P0 capable of knowing that a connection went bad?
Even if they were set to "true", testing a connection is supposed to attempt executing a query on the database (something like "select 1 + 1 from dual"). We the database goes down, shouldn't the test timeout? In other words, shouldn't C3P0 take 3.5 seconds to determine that a connection has gone bad?
Thanks a lot, in advance.
(apologies... this'll be terse, i'm phonebound.)
1) even if no explicit Connection testing has been configured, c3p0 tests Connections that experience Exceptions while checked-out to determine whether they remain suitable for pooling.
2) a good JDBC driver will throw Exceptions quickly if the DBMS is unavailable. there's no reason why these internal Connection tests should be slow.
3) rather than polling for unused Connections to avoid waiting for checking / new acquisitions, you might consider just setting the config parameter checkoutTimeout.
good luck!
We are running a website on a vps server with sql server 2008 x64 r2. We are being bombarded with 17886 errors - namely:
The server will drop the connection, because the client driver has
sent multiple requests while the session is in single-user mode. This
error occurs when a client sends a request to reset the connection
while there are batches still running in the session, or when the
client sends a request while the session is resetting a connection.
Please contact the client driver vendor.
This causes sql statements to return corrupt results. I have tried pretty much all of the suggestions I have found on the net, including:
with mars, and without.
with pooling and without
with async=true and without
we only have one database and it is absolutely multi-user.
Everything has been installed recently so it is up to date. They may be correlated with high cpu (though not exclusively according to the monitors I have seen). Also correlated with high request rates from search engines. However, high cpu/requests shouldn't cause sql connections to reset - at worst we should have high response times or iis refusing to send response.
Any suggestions? I am only a developer not dba - do i need a dba to solve this problem?
Not sure but some of your queries might cause deadlocks on the server.
At the point you detect this error again
Open Management Studio (on the server, install it if necessary)
Open a new query window
Run sp_who2
Check the blkby column which is short for Blocked By. If there is any data in that column you have a deadlock problem (Normally it should be like the screenshot I attached, completely empty).
If you have a deadlock then we can continue with next steps. But right now please check that.
To fix the error above, ”MultipleActiveResultSets=True” needs to be added to the connection string.
via Event ID 17886 MSSQLServer – The server will drop the connection
I would create an eventlog task to email you whenever 17886 is thrown. Then go immediately to the db and execute the sp_who2, get the blkby spid and run a dbcc inputbuffer. Hopefully the eventinfo will give you something a bit more tangible to go on.
sp_who2
DBCC INPUTBUFFER(62)
GO
Use a "Instance Per Request" strategy in your DI-instantiation code and your problem will be solved
Most probably you are using dependency injection. During web development you have to take into account the possibility of concurrent requests. Therefor you have to make sure every request gets new instances during DI, otherwise you will get into concurrency issues. Don't be cheap by using ".SingleInstance" for services and contexts.
Enabling MARS will probably decrease the number of errors, but the errors that are encountered will be less clear. Enabling MARS is always never the solution, do not use this unless you know what you're doing.
We have a java server connecting to a MySQL 5 database usingHibernate as our persistence layer which is using c3p0 for DB connection pooling.
I've tried following the c3p0 and hibernate documentation:
Hibernate - HowTo Configure c3p0 connection pool
C3P0 Hibernate properties
C3P0.properties configuration
We're getting an error on our production servers stating that:
... Caused by:
com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException:
No operations allowed after connection
closed.Connection was implicitly
closed due to underlying
exception/error:
BEGIN NESTED EXCEPTION
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException
MESSAGE: The last packet successfully
received from the server was45000
seconds ago.The last packet sent
successfully to the server was 45000
seconds ago, which is longer than the
server configured value of
'wait_timeout'. You should consider
either expiring and/or testing
connection validity before use in your
application, increasing the server
configured values for client timeouts,
or using the Connector/J connection
property 'autoReconnect=true' to avoid
this problem.
STACKTRACE:
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
The last packet successfully received
from the server was45000 seconds
ago.The last packet sent successfully
to the server was 45000 seconds ago,
which is longer than the server
configured value of 'wait_timeout'.
You should consider either expiring
and/or testing connection validity
before use in your application,
increasing the server configured
values for client timeouts, or using
the Connector/J connection property
'autoReconnect=true' to avoid this
problem.
We have our c3p0 connection pool properties setup as follows:
hibernate.c3p0.max_size=10
hibernate.c3p0.min_size=1
hibernate.c3p0.timeout=5000
hibernate.c3p0.idle_test_period=300
hibernate.c3p0.max_statements=100
hibernate.c3p0.acquire_increment=2
The default MySQL wait_timetout is set to 28800 seconds (8 hours), the reported error is saying that it's been over 45000 seconds (about 12.5 hours). Although the c3p0 configuration states that it will "timeout" idle connections that haven't been used after 5000 seconds and it will check every 300 seconds, thus an idle connection should never live longer than 5299 seconds right?
I've tested locally by setting my developer MySQL (my.ini on windows, my.cnf on Unix) wait_timeout=60 and lowering the c3p0 idle timeout values below 60 seconds, and it will properly timeout idle connections and create new ones. I also check to ensure that we're not leaking DB connections and holding onto a connection, and it doesn't appear we are.
Here's the c3p0.properties file I'm using to test in my developer environment to ensure c3p0 is properly handling connections.
hibernate.properties (testing with MySQL wait_timeout=60)
hibernate.c3p0.max_size=10
hibernate.c3p0.min_size=1
hibernate.c3p0.timeout=20
hibernate.c3p0.max_statements=100
hibernate.c3p0.idle_test_period=5
hibernate.c3p0.acquire_increment=2
c3p0.properties
com.mchange.v2.log.FallbackMLog.DEFAULT_CUTOFF_LEVEL=ALL
com.mchange.v2.log.MLog=com.mchange.v2.log.FallbackMLog
c3p0.debugUnreturnedConnectionStackTraces=true
c3p0.unreturnedConnectionTimeout=10
Make sure that c3p0 really is starting by examine the log. I, for some reason, had two versions of hibernate (hibernate-core3.3.1.jar and hibernate-3.2.6GA.jar) on my classpath. I also used hibernate annotatations version 3.4.0GA which is not compatible with 3.2.x. (dont know if that had something to do with the original problem).
After removal of one of the hibernate jar's (cant remember which i deleted, probably hibernate-3.2.6GA.jar) c3p0 finally started and i got rid of the annoying com.mysql.jdbc.exceptions.jdbc4.CommunicationsException that happend efter 8h inactivity.
Suppose the following:
I have a database set up on database.mywebsite.com, which resolves to IP 111.111.1.1, running from a local DNS server on our network.
I have countless ASP, ASP.NET and WinForms applications that use a connection string utilising database.mywebsite.com as the server name, all running from the internal network.
Then the box running the database dies, and I switch over to a new box with an IP of 222.222.2.2.
So, I update the DNS for database.mywebsite.com to point to 222.222.2.2.
Will all the applications and computers running them have cached the old resolved IP address?
I'm assuming they will have.
Any suggestions along the lines of "don't have your IP change each time you switch box" are not too welcome as I cannot control this aspect of the situation, unfortunately. We are currently using the machine name of the box, which changes every time it dies and all apps etc. have to be updated with the new machine name. It hurts.
Even if the DNS is not cached local to the machine, it will likely be cached somewhere along the DNS chain between the machine and the name servers, at least for a short while. My understanding is this situation would usually be handled with IP takeover where you just make the new machine 111.111.1.1.
Probably a question for serverfault.
You're looking for DNS TTL (Time To Live) I guess.. In my opinion applications may cache the IP for at most the value of the TTL. I'm afraid however that some applications/technologies might actually cache it longer (agian in my opinion completely wrong)
Each machine will cache the ip address.
The length of time it is cached is the TTL (Time To Live). This is a setting on your DNS server, if you set it very low say 5 mins, then you show be up and running fairly quikly. A bit of a hack but it should work.
Yes, the other comments are correct in that what controls this is the DNS TTL set for the hostname database.mywebsite.com.
You'll have to decide what the maximum amount of time you're willing to wait for if you have a failure on your primary address (111.111.1.1) after you make the switch to the secondary address. Lower settings will give you a quicker recovery time, but will also increase the load and bandwidth to your DNS server because clients will have to re-query it to refresh their cache more often.
You can use nslookup using the -d option from your cmd prompt to see what your default TTL times and remaining TTL times are for the DNS server you are querying.
%> nslookup -d google.com
You should assume that they are cashed for two reasons not clearly mentioned before:
1- Many "modern" versions of OS families do DNS caching.
2- Many applications do DNS caching or have poor error/failure detection on live connections and/or opening new connections. This would possibly include your database client.
Also, this is probably not well documented. I did some googling, and found this for MySQL:
http://dev.mysql.com/doc/refman/5.0/en/connector-net-programming-connecting-connection-string.html#connector-net-programming-connecting-errors
It does not clearly explain its behavior in this regard.
I had a similar issue with a web site that disables the application pool recycling features and runs for weeks on end. Sometimes, a clustered SQL Server box would restart and for some reason, my SqlConnection's were not reconnecting. I was getting the error:
A network-related or instance-specific
error occurred while establishing a
connection to SQL Server. The server
was not found or was not accessible.
Verify that the instance name is
correct and that SQL Server is
configured to allow remote
connections. (provider: Named Pipes
Provider, error: 40 - Could not open a
connection to SQL Server)
The server was there - and running - in fact, if I just recycled the app pool, the app would work fine - but I don't like recycling app pools!
The connections that were being held in the connection pool were somehow using old connection information, and that could have been old IP addresses. This is what seems so similar to the poster's question, that it appears to be cached DNS information, because as soon as some sort of a cache is cleared, the app works fine.
This is how I solved it - by forcing all of the connections in the pool to be re-created:
Try
' Example: SqlDependency, but this could also be any SqlConnection.Open call
Dim result As Boolean = SqlClient.SqlDependency.Start(ConnStr)
Catch sqlex As SqlClient.SqlException
SqlClient.SqlConnection.ClearAllPools()
End Try
The code sample is just the boiled-down basics - it should be tweaked for your situation!
The DNS gets cached, but for any server that resolves to the wrong ip address, you can update the HOSTS file of the server and the ip should be updated immediately. This could be a solution if you have a limited amount of servers accessing your database server.