We have a client deployment of our software that is showing intermittent SQL server connection failures, and we are struggling to understand them.
Our system consists of a SQL Server DB (2012) and 14 identical engines, each installed on a Windows 2012 VM. Each of these was created from the same template so they should be identical. The engines consist of a Windows service that connects to the DB on startup by reading a single row from a table. If the connection fails they will wait a few seconds and try again, until they get a connection.
In this particular case, the VMs were all rebooted due to a Windows Update. (The SQL server had the update/reboot about 12 hours before). They came online within a few minutes of each other. 12 of the engines started up without any problem. Two of them, however, failed to connect to the DB with:
"The underlying provider failed on Open."
Those two engines then started to poll, and continued to get this error for many hours. The rest of the engines had started up and were fine. We have a broker service too that was accessing the DB throughout and showed no connection issues.
When the client noticed this issue, they restarted the engine services on the two problem VMs, and the two engines connected to the DB just fine.
We are trying to understand what could have happened here. I guess my main questions are:
What could be an explanation of why 12 connections succeed and two fail? There's absolutely no difference as far as we know between the engines. The query itself is very simple.
Why did the connection continue to fail for those two engines until the service was restarted? This suggests to me that there is some process-level failed state that is only cleared when restarting the services. I've looked at the code to see if it was reusing the connections. It uses Entity Framework to read the single table row, and we create a fresh DbContext each time. I don't understand how this could go wrong.
We noted that there was a CheckDb operation proceeding on the DB around the time the services were coming up, and we wondered if this could be related to the issue. However, the client says that this runs every night and hasn't caused problems in the past. And it wouldn't explain why the engines didn't come back up again.
Thanks in advance for any help.
Related
So I've had a Windows Workflow with persistence enabled and all has been well for a month now. This morning, things came crashing down with the error below. The problem is, a persistence store HAS been configured and working perfectly (in SQL Server). Absolutely nothing has changed. The database where the persistence information is stored is still there and healthy. We can find no problems with the database itself. No matter what we do, the list of available servers that should show up in the "Sql Server Store" dropdown remains empty.
It gets worse. The error isn't restricted to one server. It has happened organization-wide on several machines (we're using AppFabric with persistence enabled on multiple servers here). We've tried creating new databases on different servers running separate instances of SQL Server--same result. Our theory, such as it is, is that Microsoft pushed out some strange update via automatic updates overnight.
Clues, anyone? TIA
Background
To make the story short, our company is facing the task of making our application redudant and more resilent to heavy loads by load-balancing. The task is on my desk and I've been doing some research as I've never done it before.
Fact
Today we host our application on 1 server and the goal is to have another one to even out serverload with load-balacing.
Issue
I've been doing some research and got stuck on how to setup the MSSQL Replication. If one server goes down the other one must be in sync as the users will be redirected there insted by the load-balancer
The tenthousand view to the solution goes something like: Have a Publisher/Distributor on same server and then add subscriber databases and they will sync between eachother.
Question
What happens if the Publisher/Distrubtor server goes down? Suddenly the system isn't redudant at all. Do we have to setup a Publisher/Distrubtor on each subscriber server to take over the role? I've been search around and haven't found a good answer.
Just hint if the explanation is confusing and I'll fill in the blanks..
Thanks in advance!
Background:
I have a SQL Server 2005 setup with master, slave1, slave2 replication set up as a pull replication from slaves. The distribution database resides on the slave1 machine, both slaves pull.
A problem began today where the replication on slave1 simply stops running. It claims that it completed successfully, but it does not restart, and manually starting the process finishes in roughly one minute, again without an error message.
Replication is running fine on slave2, but I can't seem to figure out what's wrong on slave1. I've tried the obvious Windows debugging 101: "restart the machine" technique, but to no avail.
Has anyone encountered this before Does anyone have an idea of what I could check or change to get it working again? I'm especially at a loss as SQL Server claims that the job is just finishing successfully.
Though I'm unsure of why this began occurring. It appears to be due to the use of a custom SQL Server Replication Agent profile. Switching to using the default got it working again.
We are having some strange performance issues and I was hoping somebody may be able to point us in the right direction. Our scenario is an ASP.NET MVC C# website using EF4 POCO in IIS 7 (highly specced servers, dedicated just for this application).
Obviously it's slow on application_startup which is to be expected, but once that has loaded you can navigate the site and everything is nice and snappy 0.5ms page loads (we are using Mini-Profiler). Now if you stop using the site for say 5 - 10 minutes (we have the app pool recycle set to 2 hours and we are logging so we know that it hasn't been recycled) then the first page load is ridiculously slow, 10 - 15 seconds, but then you can navigate around again without issue (0.5ms).
This is not SQL queries that are slow as all queries seem to work fine after the first page hit even if they haven't been run yet so not caching anywhere either.
We have done a huge amount of testing and I can't figure this out. The main thing I have tried so far is to Pre generate EF views but this has not helped.
It seems after looking at Sql Server Profiler after 5 minutes give or take 30 seconds with no activity in Sql Server Profiler and no site interaction a couple of "Audit Logout" entries appear for the application and as soon as that happens it then seems to take 10 - 15 seconds to refresh the application. Is there an idle timeout on Sql Server?
This should work if you use the below in your connection string:
server=MyServer;database=MyDatabase;Min Pool Size=1;Max Pool Size=100
It will force your connection pool to always maintain at least one connection. I must say I don't recommend this (persistant connection) but it will solve your problem.
Are you using LMHOSTS file? We had same issue. LMHOSTS file cache expires after 10 minutes by default. After system has been sitting idle for 10 minutes the host would use Broadcast message before reloading the LMHOSTS file causing the delay.
It seems after looking at Sql Server Profiler after 5 minutes give or
take 30 seconds with no activity in Sql Server Profiler and no site
interaction a couple of "Audit Logout" entries appear for the
application and as soon as that happens it then seems to take 10 - 15
seconds to refresh the application. Is there an idle timeout on Sql
Server?
This is telling me that the issue most likely lies with your SQL server and/or the connection to it rather than with your application. SQL server uses connection pooling and SQL will scavange these pools every so often and clean them up. The delay you appear to be experiencing is when your connection pools have been cleaned up (the audit logouts) and you are having to establish a new connection. I would talk/work with your SQL database people.
For testing, do you have access to a local/dev copy of the database not running on the same SQL server as your production app? If not, try and get one setup and see if you suffer from the same issues.
I would run "SQL Server Profiler" against SQL Server and capture a new trace while reproducing the problem by accessing the site after being idle 5-10 mins. After that look at the trace. Specifically, look for entries in which ApplicationName starts with "EntityFramework...". This will tell you what is EF doing at that moment. There could be some issue with custom caching on top of EF, or some session state that is expiring (check sessionState timeout in web.config)
Since it is the first run (per app?) that is slow, you may be experiencing the compilation of the EDMX or the LINQ into SQL.
Possible solutions:
Use precompiled views and precompiled queries (may require a lot of refactoring).
http://msdn.microsoft.com/en-us/library/bb896240.aspx
http://blogs.msdn.com/b/dmcat/archive/2010/04/21/isolating-performance-with-precompiled-pre-generated-views-in-the-entity-framework-4.aspx
http://msdn.microsoft.com/en-us/library/bb896297.aspx
http://msdn.microsoft.com/en-us/magazine/ee336024.aspx
Dry run all your queries on app start (prior to first request is received).
You can run queries with a fake input (e.g. non existing zero keys) on a default connection string (can be to an empty database). Just make sure you don't throw exceptions (use SingleOrDefault() instead of Single() and handle null results and 0-length list results on .ToList()).
Create a simple webpage that accesses the SQL Server with a trivial query like "Select getDate()" or some other cheap query. Then use an external service like Pingdom or other monitor to hit that page every 30 seconds or so. That should keep the connections warm.
Try to overwrite your timeout in the web.config like in this example:
Data Source=mydatabase;Initial Catalog=Match;Persist Security Info=True
;User ID=User;Password=password;Connection Timeout=120
If it works, this is not a solution..just a work around.
In our case the application was hosted in Azure App Service Plan and was having similar problem. Turned out to be a problem of not configuring virtual network. See the question/answer here - EF Core 3.1.14 Recurring Cold Start
In our college, we are conducting a contest, which is in the form of Multiple choice questions.
For that we are using VB as a front-end , MS Access as a back-end.
Ref:
The Application VB 6 runs with no problem and the participant entry is added into the database along with their scores, when 1 or 2 clients access the database simultaneously.
Problem:
But when more than 2 clients access the database simultaneously, the Application crashes.
1.In some clients, we are getting runtimeerror printing some large negative value with a
message "Operations query cannot modify the database".
2.In some clients, the VB 6.0 crashes and closes suddenly.
These errors occurs when we are tried to access the database using
OK,SUBMIT buttons.
Could you tell me why this error occurs and how can I correct it.
My Questions are..
1.Is giving the burden of all clients to a single laptop is the problem here?If there is some other problem please explain it.
2.Why I am getting the error as "Operation query cannot modify database", If so then how it works when 1 or 2 clients access the database simultaneously.
Access databases (and other directly file-based DBs) are not really built for multi-access. There are some facilities in place to help it work, but in my experience, it is quite unreliable.
You need a database server running, which can allow multiple clients to the same database simultaneously. A free option is MySQL. There is also a free version of Microsoft SQL Server available.
"Operation query cannot modify database"
Badly designed Access databases can have issues with users trying to modify records if they don't have defined Primary keys. This is especially true when multiple people are accessing because it literally can't tell which record to modify if two people try to do the same thing. Sometimes it will let you insert but not update.
Further if you are looking for performance, Access is just the wrong tool. It has very little in the way of performance tuning options or abilities. SQl Server Express or mySQl would have more things available to diagnose and fix a performance issue.