We are building an application which makes every week a very large amount of request over the database, concurrently.
We have ~15-20 threads which query the database concurrently.
We are actually encountering a lot of problems:
On the database side(not enough RAM): being resolved.
But on the client too. We have Exception when trying to get a connection or execute commands. Those commande are mades through entity framework.
The application has two part: one website and one console application.
So can anyone tell me how to increase the following values?
Connection Timeout
Command Timeout
Connection pool size
I think that there several things that have to be done on the server side(SQL Server or IIS), but I can't find where?
Command timeout can be set on ObjectContext instance. Connect timeout and connection pool size is configured in connection string but if you have only 15-20 threads your problem will most probably be somewhere else because default connection pool size is 100.
Enclose your objectContext in a using block so the context disposes after you have done your work.
you can make a method to pass in thread which uses your entity context to do the work you want and then dispose the connection after the work is finished, you can use the stateinfo object variable to pass in different parameters to use during the life of your method.
void DoContextWork(object stateInfo)
{
// wrap your context in a using clause
using(var objectContext = new YourEntity()
{
// Do work here
}
}
you can have multiple threads call this method and each time your connection gets called you can do your work on your DB without getting the issues you mentioned above.
Related
I am using c3p0 for my connection pooling. The ComboPooledDataSource I use is configured as below.
#Bean
public DataSource dataSource() {
ComboPooledDataSource dataSource = new ComboPooledDataSource();
dataSource.setUser("user");
dataSource.setDriverClass("oracle.jdbc.OracleDriver");
dataSource.setJdbcUrl("test");
dataSource.setPassword("test");
dataSource.setMinPoolSize("10");
dataSource.setMaxPoolSize("20");
dataSource.setMaxStatements("100");
return dataSource;
}
I am facing some issues with this. I get warnings that this might leak connections. Also the below error from time to time,
as all the connections are being used up.
java.sql.SQLException: Io exception: Got minus one from a read call
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:112)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:146)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:255)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:387)
at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:439)
at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:165)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:35)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:801)
at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:135)
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection
(WrapperConnectionPoolDataSource.java:182)
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection
(WrapperConnectionPoolDataSource.java:171)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource
(C3P0PooledConnectionPool.java:137)
at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)
at com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)
at com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810)
at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
And from the DB stat, could see almost 290 inactive connections. I am having around 8 applications deployed in two servers,
connecting to the same DB.
My queries are
How do I make sure th connections are closed and not to have these many inactive connections?
Would configuring idle time and timeout resolve this issue?
What would happen if the server is brought down/tomcat is shutdown, will the connections remain open?
Connections are mainly used during startup to load cache, so is there a way of not using these connections afterwards?
What should I do for the existing inactive connections?
Given maxPoolSize of 20 and eight deployments, you should expect to see up to 180 Connections, which may be inactive if the application has seen periods of traffic which has now subsided. You have configured nothing to encourage a fast scaling down of your pools -- set maxIdleTime and/or maxIdleTimeExcessConnections and/or maxConnectionAge.
You should probably tell Spring how to close the DataSource you've defined. Use #Bean(destroyMethodName="close") instead of #Bean alone above your dataSource() method.
You have not configured any sort of Connection testing, so even broken Connections might remain in the pool. Please see Simple Advice On Connection Testing.
If the issue were a Connection leak, clients would eventually hang indefinitely, as the pool would be out of Connections to check out, but would already have reached maxPoolSize, and so wouldn't be able to acquire more from the DBMS. Are you seeing clients hang like that?
The way you avoid Connection leaks is, post-Java7, to always acquire Connections from your DataSource via try-with-resources. That is, use...
try ( Connection conn = myDataSource.getConnection() ) {
...
}
rather than just calling getConnection() in a method that might throw an Exception or in a try block. If you are using an older version of Java, you need to use the robust resource cleanup idiom, that is, acquire the Connection in a try block and be sure that conn.close() is always closed in the finally block, regardless of any other failures in the finally block. If you are not working with the DataSource directly, but letting Spring utilities work with it, hopefully those utilities are doing the right thing. But you should post whatever warning you are receiving that warns you of potential Connection leaks!
If your application has little use for Connections after it has "warmed up", and you want to minimize the resource footprint, set minPoolSize to a very low number, and use maxIdleTime and/or maxIdleTimeExcessConnections and/or maxConnectionAge as above to ensure that the pool promptly scales down when Connections are no longer in demand. Alternatively you might close() the DataSource when you are done with its work, but you are probably leaving that to Spring.
Our scheduled jobs started failing since yesterday with the following error message:
CustomUpdate.Execute - System.NullReferenceException: Object reference
not set to an instance of an object. at
System.Web.Security.Roles.GetRolesForUser(String username) at
EPiServer.Security.PrincipalInfo.CreatePrincipal(String username)
The scheduled job uses anonymous execution and logs in programmatically using the following call:
if (PrincipalInfo.CurrentPrincipal.Identity.Name == string.Empty)
{
PrincipalInfo.CurrentPrincipal = PrincipalInfo.CreatePrincipal(ApplicationSettings.ScheduledJobUsername);
}
I have put in some more logging around PrincipalInfo.CreatePrincipal call which is in Episerver.Security and noticed that PrincipalInfo.CreatePrincipal calls System.Web.Security.Roles.GetRolesForUser(username) and Roles.GetRolesForUser(username) returns an empty string array.
There were no changes code wise or on the server (updates, etc).
I checked that the user name used to run the task is in the database and has roles associated with it.
I checked that applicationname is set up correctly and is associated with the user
If i run the job manually using the same user it executes with no issues (i know there is a difference between running the job manually and using the scheduler)
I also tried creating a new user, that didn’t work either.
Has anyone come across the same or similar issue? Any thoughts how to resolve this issue?
I have finally found a problem - application pool running with more than one worker processes (in my instance I had two worker processes). Once I set worker processes limit to one everything started to work again.
We are having an issue when using NHibernate with distributed transactions.
Consider the following snippet:
//
// There is already an ambient distributed transaction
//
using(var scope = new TransactionScope()) {
using(var session = _sessionFactory.OpenSession())
using(session.BeginTransaction()) {
using(var cmd = new SqlCommand(_simpleUpdateQuery, (SqlConnection)session.Connection)) {
cmd.ExecuteNonQuery();
}
session.Save(new SomeEntity());
session.Transaction.Commit();
}
scope.Complete();
}
Sometimes, when the server is under extreme load, we'll see the following:
The query executed with cmd.ExecuteNonQuery is chosen as a deadlock victim (we can see it in SQL Profiler), but no exception is raised.
session.Save fails with the error message, "The operation is not valid for the state of the transaction."
Every time this code is executed after that, session.BeginTransaction fails. The first few times, the inner exception varies (sometimes it is the deadlock exception that should have been raised in step 1). Eventually it stabilizes to "The server failed to resume the transaction. Desc:3800000177." or "New request is not allowed to start because it should come with valid transaction descriptor."
If left alone, the application will eventually (after seconds or minutes) recover from this condition.
Why is the deadlock exception not being reported in step 1? And if we can't resolve that, then how can we prevent our application from temporarily becoming unusable?
The issue has been reproduced in the following environments
Windows 7 x64 and Windows Server 2003 x86
SQL Server 2005 and 2008
.NET 4.0 and 3.5
NHibernate 3.2, 3.1 and 2.1.2
I've created a test fixture which will sometimes reproduce the issue for us. It is available here: http://wikiupload.com/EWJIGAECG9SQDMZ
We've finally narrowed this down to a cause.
When opening a session, if there is an ambient distributed transaction, NHibernate attaches an event handler to the Transaction.TransactionCompleted, which closes the session when the distributed transaction is completed. This appears to be subject to a race condition wherein the connection may be closed and returned to the pool before the deadlock error propagates across, leaving the connection in an unusable state.
The following code will reproduce the error for us occasionally, even without any load on the server. If there is extreme load on the server, it becomes more consistent.
using(var scope = new TransactionScope()) {
//
// Force promotion to distributed transaction
//
TransactionInterop.GetTransmitterPropagationToken(Transaction.Current);
var connection = new SqlConnection(_connectionString);
connection.Open();
//
// Close the connection once the distributed transaction is
// completed.
//
Transaction.Current.TransactionCompleted +=
(sender, e) => connection.Close();
using(connection.BeginTransaction())
//
// Deadlocks but sometimes does not raise exception
//
ForceDeadlockOnConnection(connection);
scope.Complete();
}
//
// Subsequent attempts to open a connection with the same
// connection string will fail
//
We have not settled on a solution, but the following things will eliminate the problem (while possibly having other consequences):
Turning off connection pooling
Using NHibernate's AdoNetTransactionFactory instead of AdoNetWithDistributedTransactionFactory
Adding error handling that calls SqlConnection.ClearPool() when the "server failed to resume the transaction" error occurs
According to Microsoft (https://connect.microsoft.com/VisualStudio/feedback/details/722659/), the SqlConnection class is not thread-safe, and that includes closing the connection on a separate thread. Based on this response we have filed a bug report for NHibernate (http://nhibernate.jira.com/browse/NH-3023).
not a definitive answer, but i suspect you have some problems with session management and that you are using the same session across multiple calls to handlers. i don't think it's actually the connection that is in a bad state, but rather the nhibernate session. this doesn't seem to jive with you not seeing the problem with connection pooling turned off, so i may be off base, but i still suspect it has to do with reusing sessions.
the first thing i would suggest is that you try to confirm this by logging the hashcode of the session and the hashcode of session.GetSessionImplementation() (my understanding of using the castle nhibernate facility is that you will see the same instance of session, even though it is actually a different session and the session implementation will actually show a difference). see if you are seeing the same hashcodes being used in handling different messages.
if it is a question of session management, try using a nservicebus module to manage your sessions for your handlers. here is a post from andreas about doing that. i don't think his edit about having a way to do this built in on the trunk was in the 2.5 release, so you probably want to go ahead with this. (i could be wrong about that.)
http://andreasohlund.net/2010/02/03/nhibernate-session-management-in-nservicebus/
This doesn't exactly solve your problem, but you could make your IPreInsertEventListener just send a NSB message, and then have the receiver of the message invoke the stored procedure. I've done that with problematic pre-and post event listeners while using NHibernate and NSB in the past.
Another thought is have your pre-event listener create its own connection object wrapped in a nice using statement, then it won't touch NHibernate's connection. If it deadlocks, then just do a throw an make sure you've disposed of any object's in scope.
It is an NHibernate issue. NHibernate is not opening and closing the connection on the same thread, which is not supported by ADO.NET. You can work around it by opening and closing the connection yourself. NHibernate will not close the connection unless it has also opened it.
Workaround
var connection = ((SessionFactoryImpl)_sessionFactory).ConnectionProvider.GetConnection();
using(var session = _sessionFactory.OpenSession(connection))
{
//do database stuff
}
connection.Close();
I am having an issue with the DirectoryEntry object where it's taking a long time trying to connect to to a dead AD server and eventually failing. Is it possible to set a timeout so that if its not able to connect within a specific time, it just comes out to try the next one?
There is no timeout option for DirectoryEntry directly.
You can use DirectorySearcher and set the ClientTimeout (even if you're only looking for one object by path). Or do your directory operation on a new thread or BackgroundWorker and control your own timeout.
I suggest you create your own LdapConnection to the server. This will allow you to specify a timeout and finely control which method you are using.
Also note that without going to this lower level, the .NET classes will attempt to use LDAP+SSL, then Kerberos, and finally RPC. You may be experiencing delays/timeouts during this process.
We are using ASP.NET MVC with LINQ to SQL. We added some features and tested them all to perfection on our QA box. We are using Windows Server 2003 and SQL Server 2005. So when we pushed out changes to the Live web server we also used Red Gate SQL Compare to push new database changes to the LIVE database. We tested again between the few of us, no problems. Time for bed.
The morning comes and users are starting to hit the app, and BOOM. We have no idea why this would happen as we have not been doing any new types of code things that we were not doing before. However we did notice that during the SQL Compare sync the names of all the foreign keys were different between the two databases, not the IDs in the tables, FK_AssetAsset_A0EB67 to FK_AssetAsset_B67EF8 (for example, don't remember the exact number of trailing mixed characters during the SQL Compare), we are not sure why but that is another variable in this problem.
Strangely once this was all pushed out we could then replicate the errors on QA, but not before everything was pushed to LIVE.
QA and LIVE databases are on the same SQL Server, but the apps are on different instances of Windows Server 2003.
Errors generated:
Index was outside the bounds of the array.
Invalid attempt to call FieldCount when reader is closed.
Server failed to resume the transaction.
There is already an open DataReader associated with this Command which must be closed first.
A transport-level error has occurred when sending the request to the server.
A transport-level error has occurred when receiving results from the server.
Invalid attempt to call Read when reader is closed.
Invalid attempt to call MetaData when reader is closed.
Count must be positive and count must refer to a location within the string/array/collection. Parameter name: count
ExecuteReader requires an open and available Connection. The connection's current state is connecting.
Any one have any idea what the heck could have happened?
EDIT: Since we were able to replicate the errors all of a sudden on QA, it might not be a user load issue... Needless to say we all feel really screwed here.
Concurrency always brings bugs out of the woodwork. I'd recommend you check for objects that could be shared among requests (such as static members and singletons) and refactor your code so that as little as possible is shared.
As far as specifics go, for the error "There is already an open DataReader associated with this Command which must be closed first," you may want to try adding MultipleActiveResultSets=True to your connection strings.
It sounds like you're crossing the streams a bit and trying to share DataContexts across requests. My suggestion would be to wire in a dependancy injection framework that creates a new instance of the dependancy for each request.
I use Castle's IoC and wire it into the controller factory so that when it sees a dependancy on a repository it creates a new instance of that repository for each request. If you go this route let me know and I can shoot you a few more resources.