I have a winforms app where I have a Telerik dropdownchecklist that lets the user select a group of state names.
Using EF and the database is stored in Azure SQL.
The code then hits a database of about 17,000 records and filters the results to only include states that are checked.
Works fine. I am wanting to update a count on the screen whenever they change the list box.
This is the code, in the itemCheckChanged event:
var states = stateDropDownList.CheckedItems.Select(i => i.Value.ToString()).ToList();
var filteredStops = (from stop in aDb.Stop_address_details where states.Contains(stop.Stop_state) select stop).ToArray();
ExportInfo_tb.Text = "Current Stop Count: " + filteredStops.Count();
It works, but it is slow.
I tried to load everything into a memory variable then querying that vs the database but can't seem to figure out how to do that.
Any suggestions?
Improvement:
I picked up a noticeable improvement by limiting the amount of data coming down by:
var filteredStops = (from stop in aDb.Stop_address_details where states.Contains(stop.Stop_state) select stop.Stop_state).ToList();
And better yet --
int count = (from stop in aDb.Stop_address_details where
states.Contains(stop.Stop_state)
select stop).Count();
ExportInfo_tb.Text = "Current Stop Count: " + count.ToString();
The performance of you query, actually, has nothing to do with Contiains, in this case. Contains is pretty performant. The problem, as you picked up on in your third solution, is that you are pulling far more data over the network than required.
In your first solution you are pulling back all of the rows from the server with the matching stop state and performing the count locally. This is the worst possible approach. You are pulling back data just to count it and you are pulling back far more data than you need.
In your second solution you limited the data coming back to a single field which is why the performance improved. This could have resulted in a significant improvement if your table is really wide. The problem with this is that you are still pulling back all the data just to count it locally.
In your third solution EF will translate the .Count() method into a query that performs the count for you. So the count will happen on the server and the only data returned is a single value; the result of count. Since network latency CAN often be (but is not always) the longest step when performing a query, returning less data can often result in significant gains in query speed.
The query translation of your final solution should look something like this:
SELECT COUNT(*) AS [value]
FROM [Stop_address_details] AS [t0]
WHERE [t0].[Stop_state] IN (#p0)
I use a SqlTransaction in my C# project, and I use a Delete statement with an EcexuteNonQuery call.
This works very well and I have always the same amount of rows to delete, but 95% of the time, this needs 1 ms and approx 5% of the time, it is between 300 - 500 ms.
My code:
using (SqlTransaction DbTrans = conn.BeginTransaction(IsolationLevel.ReadCommitted))
{
SqlCommand dbQuery = conn.CreateCommand();
dbQuery.Transaction = DbTrans;
dbQuery.CommandType = CommandType.Text;
dbQuery.CommandText = "delete from xy where id = #ID";
dbQuery.Parameters.Add("ID", SqlDbType.Int).Value = x.ID;
dbQuery.ExecuteNonQuery();
}
Is something wrong with my code?
Read Understanding how SQL Server executes a query and How to analyse SQL Server performance to get you started on troubleshooting such issues.
Of course I assume you have an index on xy.id. Your DELETE is likely blocking from time to time. This an be caused by many causes:
data locks from other queries
IO block from your hardware
log growth events
etc
The gist of it is that using the techniques in the articles linked above (specially the second one) you can identify the cause and address it appropriately.
Changes to your C# code will have little impact, if any at all. Using a stored procedure is
not going to help. You need to root cause the problem.
I hope this is the correct site to post this on. I wasn't sure if I should post here or Server Fault, but seeing as this involves the website perspective, I thought perhaps this community might be a little more accurate, but I'm not 100% on that.
I have been banging my head against the wall for over half a year trying to figure out just what's going on here. I would be ecstatic if I could track down why AJAX calls are slow when going through our Site Server.
I have built a small web-app for the organization I work for and it is pretty much set up like this:
The site itself (WebMatrix IIS Express site) resides on the Site Server, but (with the help of C#) it uses SQL queries to query a (considerably large) database on our Database Server.
The problem is that when my site performs the AJAX (simple jQuery $.ajax() calls) that requires it to query the database, the response takes over 5 seconds, each!
(Chrome Network Details):
(You'll see that some of the responses are really quick. These responses contain no or a lot less data than the other responses. Maybe there's a data limit somewhere that's causing the Site Server to analyze them?)
Now here's the kicker:
On the Development machine, the local machine the site is developed on, cuts out the Site Server, has the same code, and queries the same database, but the lag doesn't persist in this scenario. The responses in this scenario are in the low millisecond, just what I would expect it to be.
Here's what the Chrome Network Details look like from the development machine:
(None even close to 1 second, let alone 5).
Some More Specifics
When launching this site straight from the Site Server, the lag persists.
WebMatrix uses SQL Server CE, while the SQL Installed on the Database Server is SQL Server 2005 (I really don't think this makes a difference, as the query itself isn't anything special, plus it's the same code that's used in either scenario).
The Site Server has been tested to see if the RAM, Processor, and Bandwidth are maxing out, but the truth is that running this web-app doesn't even touch the Site Server's resources. The same has been found for the Database Server, as well.
The connection to the database is readonly (doubt this matters, just trying to give as much detail as possible).
We have indexed the database on the Database Server, but it helped, virtually, none at all.
Even though it is just an Intranet site, I am told that putting the site directly on the Database Server is not an option.
At the moment, the AJAX requests are not asynchronous, but it should still not take this long (especially considering that it only lags from the Site Server and not from the Development Machine, even though the code is 100% identical in both cases).
Probably doesn't make any difference, but I am in an ASP.NET WebPages using WebMatrix with C# environment.
The Operating System on the Site Server is: Windows Server 2008 R2
The Operating System on the Database Server is: Windows Server 2003
What could make this app work well from my local machine but not from the Site Server? I think the problem has to be the Site Server, given this, but none of its resources are maxing out or anything. It seems to only lag by about 5 seconds per request if the data being returned is over a certain amount (an amount that seems pretty low, honestly).
Truth is, I am hopelessly stuck here. We have tried everything over the past several months (we are having a similar problem with another Intranet site where the AJAX calls lag there, too, we have just lived with it for a while).
I don't know what else to even look into anymore.
In case anybody wants to see some code
jQuery (one of the AJAX requests, they are all just repeats of this with different parameters)
$.ajax({
url: '/AJAX Pages/Get_Transactions?dep=1004',
async: false,
type: 'GET',
dataType: "json",
contentType: "application/json",
success: function (trans) {
for (var i = 0; i < trans.length; i++) {
trans[i][0] = getTimeStamp(trans[i][0]);
}
jsonObj1004 = trans;
},
error: function (jqXHR, textStatus, error) {
alert("Oops! It appears there has been an AJAX error. The Transaction chart may not work properly. Please try again, by reloading the page.\n\nError Status: " + textStatus + "\nError: " + error);
}
});
C# Server Side Code (With Razor)
#{
Layout = "";
if (IsAjax)
{
var db = Database.Open("OkmulgeeCIC");
Dictionary<string, double> dataList = new Dictionary<string, double>();
var date = "";
var previousDate = "";
double amount = 0;
string jsonString = "[";
string queryDep = "SELECT ba_trans_entered AS transDate, (ba_trans_amount * -1) AS transAmount FROM BA_VTRANS WHERE ba_trans_year >= 2011 AND ba_trans_operator = 'E' AND ba_trans_system = 'AP' AND ba_trans_ledger LIKE #0 + '%' ORDER BY ba_trans_entered ASC";
string dep = Request.QueryString["dep"];
foreach (var row in db.Query(queryDep, dep))
{
date = row.transDate.ToString();
date = date.Substring(0, date.IndexOf(" "));
amount = Convert.ToDouble(row.transAmount);
if (date == previousDate)
{
dataList[date] = dataList[date] + amount;
}
else
{
dataList.Add(date, amount);
}
previousDate = date;
}
foreach (var item in dataList)
{
jsonString += "[";
jsonString += Json.Encode(item.Key) + ", ";
jsonString += Json.Encode(item.Value) + "],";
}
//jsonString += Json.Encode(date);
jsonString = jsonString.TrimEnd(',');
jsonString += "]";
#Html.Raw(jsonString)
}
else
{
Context.RedirectLocal("~/");
}
}
ADDITONAL INFO FROM SQL SERVER PROFILER
From Development Machine
From User Machine (lag)
Just looking over your code two things jumped out for me
1) You're not closing your db connection, this is very bad. Either wrap your connection object in a using block (preferred) or add a call to .Close() at the end of your data work
using(var db = Database.Open())
{
//do work
}
2) Doing string concatenation in a loop like that is a terrible thing to do and very slow. Either use a StringBuilder. Or since you're outputting JSON anyway just bundle your objects into a list or something and pass that to JSON.Encode() (preferred)
It seem to me this problem come from your Site Server but anyway you can try this:
1/ Publish your site to any Internet web server. if it is still slow => your code problem -> Check your code.
If not go to 2/ Check your configuration Site Server and Database Server. It might be Firewall or TCP/IP:Port or NETBIOS/Domain name between two Server.
I do not know if this has any relevance to this problem because i cannot see how you are calling your application. But I have multiple times experienced about 5 sec lag on IIS using C# when i use domain names to call other servers (this can also be localhost). Instead the ip should be used.
It could be nice if you tried playing around with this using IP instead of using a domain
I had something similar when working with AjAX and JSF site.
Jquery loading taking excessive time
Since you already have it working from dev machine, it might not be a problem in your case.
But to rule out any such scenario, can you develop the page without using jquery ?
I had a similar issue where I would call 20 sprocs using a for loop, they were not large sprocs mind you but a sproc that would return 5 values.
It would work fine but time to time it would almost like lag out and would not be able to load any of the sprocs or a very small amount until timing out completely.
That is when I discovered Parameter Sniffing for SQL Server.
To fix it I added in sproc Parameters equal to the incoming parameters from my C# code.
OLD CODE:
CREATE PROC [dbo].[sp_procname_proc]
(
#param1 int,
#param2 int,
#param3 varchar(5),
--..... etc .....
)AS
BEGIN
-- select from db
END
NEW CODE
CREATE PROC [dbo].[sp_procname_proc]
(
#param1 int,
#param2 int,
#param3 varchar(5),
--..... etc .....
)AS
BEGIN
#localParam1 INT = #param1
#localParam2 INT = #param2
#localParam3 varchar(5) = #param3
-- select from db using new parameters
END
My data model has an entity Person with 3 related (1:N) entities Jobs, Tasks and Dates.
My query looks like
var persons = (from x in context.Persons
select new {
PersonId = x.Id,
JobNames = x.Jobs.Select(y => y.Name),
TaskDates = x.Tasks.Select(y => y.Date),
DateInfos = x.Dates.Select(y => y.Info)
}).ToList();
Everything seems to work fine, but the lists JobNames, TaskDates and DateInfos are not all filled.
For example, TaskDates and DateInfos have the correct values, but JobNames stays empty. But when I remove TaskDates from the query, then JobNames is correctly filled.
So it seems that EF can only handle a limited number of these "subqueries"? Is this correct? If so, what is the max. number of these "subqueries" for a single statement? Is there a way to work around these issue without having to make more than one call to the database?
(ps: I'm not entirely sure, but I seem to remember that this query worked in LINQ2SQL - could it be?)
UPDATE
I'm getting crazy about this. I tried to repro the issue from ground up using a fresh, simple project (to post the entire piece of code here, not only an oversimplified example) - and I found I wasn't able to repro it. It still happens within our existing code base (apparently there's more behind this problem, but I cannot share this closed code base, unfortunately).
After hours and hours of playing around I found the weirdest behavior:
It works great when I don't SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED; before calling the LINQ statement
It also works great (independent of the above) when I don't use a .Take() to only get the first X rows
It also works great when I add an additional .Where() statements to cut the the number of rows returned from SQL Server
I didn't find any comprehensible reason why I see this behavior, but I started to look at the SQL: Although EF generates the exact same SQL, the execution plan is different when I use READ UNCOMMITTED. It returns more rows on a specific index in the middle of the execution plan, which curiously ends in less rows returned for the entire SQL statement - which in turn results in the missing data, that is the reason for my question to begin with.
This sounds very confusing and unbelievable, I know, but this is the behavior I see. I don't know what else to do, I don't even know what to google for at this point ;-).
I can fix my problem (just don't use READ UNCOMMITTED), but I have no idea why it occurs and if it is a bug or something I don't know about SQL Server. Maybe there's some "magic max number of allowed results in sub-queries" in SQL Server? At least: As far as I can see, it's not an issue with EF itself.
A little late, but does calling ToList() on each subquery produce the required effect?
var persons = (from x in context.Persons
select new {
PersonId = x.Id,
JobNames = x.Jobs.Select(y => y.Name.ToList()),
TaskDates = x.Tasks.Select(y => y.Date).ToList(),
DateInfos = x.Dates.Select(y => y.Info).ToList()
}).ToList();
My application makes about 5 queries per second to a SQL Server database. Each query results in 1500 rows on average. The application is written on C++/QT, database operations are implemented using QODBC driver. I determined that query processing takes about 25 ms, but fetching the result - 800 ms. Here is how code querying the data base looks like
QSqlQuery query(db)
query.prepare(queryStr);
query.setForwardOnly(true);
if(query.exec())
{
while( query.next() )
{
int v = query.value(0).toInt();
.....
}
}
How to optimize result fetching?
This does not directly answer your question as I haven't used QT in years. In the actual ODBC API you can often speed up the retrieval of rows by setting SQL_ATTR_ROW_ARRAY_SIZE to N then each call to SQLFetch returns N rows at once. I took a look at SqlQuery in qt and could not see a way to do this but it may be something you could look in to with QT or simply write to the ODBC API directly. You can find an example at Preparing to Return Multiple Rows