I'm confronted with a problem I cannot get my mind around. We're running SQL Server 2012. I have run into a pair of essentially identical queries which yield different execution plans and dramatically different times to execute (1 sec vs 40+ sec)... and they even return the exact same records. The only difference between them is the category the records are queried by.
This query runs in 1 second:
SELECT P.idProduct, P.sku, P.description, P.price, P.listhidden, P.listprice, P.serviceSpec, P.bToBPrice, P.smallImageUrl,P.noprices,P.stock, P.noStock,P.pcprod_HideBTOPrice,P.pcProd_BackOrder,P.FormQuantity,P.pcProd_BTODefaultPrice,cast(P.sDesc as varchar(8000)) sDesc, 0, 0, P.pcprod_OrdInHome, P.sales, P.pcprod_EnteredOn, P.hotdeal, P.pcProd_SkipDetailsPage
FROM products P INNER JOIN categories_products CP ON P.idProduct = CP.idProduct
WHERE CP.idCategory=494 AND active=-1 AND configOnly=0 and removed=0 AND formQuantity=0
AND ((SELECT TOP 1 SP.stock FROM products SP WHERE SP.pcprod_ParentPrd = P.idProduct AND SP.description LIKE N'%(9-12 Months)' AND SP.removed=0) > 0)
ORDER BY P.description Asc
The second runs 40 seconds or more, but the ONLY difference is the idCategory queried:
SELECT P.idProduct, P.sku, P.description, P.price, P.listhidden, P.listprice, P.serviceSpec, P.bToBPrice, P.smallImageUrl,P.noprices,P.stock, P.noStock,P.pcprod_HideBTOPrice,P.pcProd_BackOrder,P.FormQuantity,P.pcProd_BTODefaultPrice,cast(P.sDesc as varchar(8000)) sDesc, 0, 0, P.pcprod_OrdInHome, P.sales, P.pcprod_EnteredOn, P.hotdeal, P.pcProd_SkipDetailsPage
FROM products P INNER JOIN categories_products CP ON P.idProduct = CP.idProduct
WHERE CP.idCategory=628 AND active=-1 AND configOnly=0 and removed=0 AND formQuantity=0
AND ((SELECT TOP 1 SP.stock FROM products SP WHERE SP.pcprod_ParentPrd = P.idProduct AND SP.description LIKE N'%(9-12 Months)' AND SP.removed=0) > 0)
ORDER BY P.description Asc
They even return the exact same records in the exact same order.
Execution plan for the 1st query:
Execution plan for the 2nd query:
[EDIT] The plans here are the actual, not the estimated, execution plans.
The categories_products table is a simple lookup table with only the two fields idCategory and idProduct. Even the records returned are exactly the same (it just happens to be that for SP.description LIKE N'%(9-12 Months)', the same products are assigned to these 2 categories). The only other difference between the two is that CP.idCategory 628 was just created this morning (but i don't see what difference that could make).
[EDIT: but that's exactly what did make the difference]
How can this be? How can simply changing the CP.idCategory queried here yield a different execution plan, and even more importantly: how is it that one takes some 40 times as long to execute?
Ultimately, I'm at a loss to figure out how to improve the dreadful performance of the 2nd query given that there's no essential difference between the two that I can understand.
This problem is description column. The length of description of idCategory[628] longer than idCategory[494]. Because you are using SP.description LIKE N'%(9-12 Months)'. Length of description is too long then you get slowly.
Also you are using ORDER BY P.description Asc.
Related
Update: I will get query plan as soon as I can.
We had a poor performing query that took 4 minutes for a particular organization. After the usual recompiling the stored proc and updating statistics didn't help, we re-wrote the if Exists(...) to a select count(*)... and the stored procedure when from 4 minutes to 70 milliseconds. What is the problem with the conditional that makes a 70 ms query take 4 minutes? See the examples
These all take 4+ minutes:
if (
SELECT COUNT(*)
FROM ObservationOrganism omo
JOIN Observation om ON om.ObservationID = omo.ObservationMicID
JOIN Organism o ON o.OrganismID = omo.OrganismID
JOIN ObservationMicDrug omd ON omd.ObservationOrganismID = omo.ObservationOrganismID
JOIN SIRN srn ON srn.SIRNID = omd.SIRNID
JOIN OrganismDrug od ON od.OrganismDrugID = omd.OrganismDrugID
WHERE
om.StatusCode IN ('F', 'C')
AND o.OrganismGroupID <> -1
AND od.OrganismDrugGroupID <> -1
AND (om.LabType <> 'screen' OR om.LabType IS NULL)) > 0
print 'records';
-
IF (EXISTS(
SELECT *
FROM ObservationOrganism omo
JOIN Observation om ON om.ObservationID = omo.ObservationMicID
JOIN Organism o ON o.OrganismID = omo.OrganismID
JOIN ObservationMicDrug omd ON omd.ObservationOrganismID = omo.ObservationOrganismID
JOIN SIRN srn ON srn.SIRNID = omd.SIRNID
JOIN OrganismDrug od ON od.OrganismDrugID = omd.OrganismDrugID
WHERE
om.StatusCode IN ('F', 'C')
AND o.OrganismGroupID <> -1
AND od.OrganismDrugGroupID <> -1
AND (om.LabType <> 'screen' OR om.LabType IS NULL))
print 'records'
This all take 70 milliseconds:
Declare #recordCount INT;
SELECT #recordCount = COUNT(*)
FROM ObservationOrganism omo
JOIN Observation om ON om.ObservationID = omo.ObservationMicID
JOIN Organism o ON o.OrganismID = omo.OrganismID
JOIN ObservationMicDrug omd ON omd.ObservationOrganismID = omo.ObservationOrganismID
JOIN SIRN srn ON srn.SIRNID = omd.SIRNID
JOIN OrganismDrug od ON od.OrganismDrugID = omd.OrganismDrugID
WHERE
om.StatusCode IN ('F', 'C')
AND o.OrganismGroupID <> -1
AND od.OrganismDrugGroupID <> -1
AND (om.LabType <> 'screen' OR om.LabType IS NULL);
IF(#recordCount > 0)
print 'records';
It doesn't make sense to me why moving the exact same Count(*) query into an if statement causes such degradation or why 'Exists' is slower than Count. I even tried the exists() in a select CASE WHEN Exists() and it is still 4+ minutes.
Given that my previous answer was mentioned, I'll try to explain again because these things are pretty tricky. So yes, I think you're seeing the same problem as the other question. Namely a row goal issue.
So to try and explain what's causing this I'll start with the three types of joins that are at the disposal of the engine (and pretty much categorically): Loop Joins, Merge Joins, Hash Joins. Loop joins are what they sound like, a nested loop over both sets of data. Merge Joins take two sorted lists and move through them in lock-step. And Hash joins throw everything in the smaller set into a filing cabinet and then look for items in the larger set once the filing cabinet has been filled.
So performance wise, loop joins require pretty much no set up and if you're only looking for a small amount of data they're really optimal. Merge are the best of the best as far as join performance for any data size, but require data to be already sorted (which is rare). Hash Joins require a fair amount of setup but allow large data sets to be joined quickly.
Now we get to your query and the difference between COUNT(*) and EXISTS/TOP 1. So the behavior you're seeing is that the optimizer thinks that rows of this query are really likely (you can confirm this by planning the query without grouping and seeing how many records it thinks it will get in the last step). In particular it probably thinks that for some table in that query, every record in that table will appear in the output.
"Eureka!" it says, "if every row in this table ends up in the output, to find if one exists I can do the really cheap start-up loop join throughout because even though it's slow for large data sets, I only need one row." But then it doesn't find that row. And doesn't find it again. And now it's iterating through a vast set of data using the least efficient means at its disposal for weeding through large sets of data.
By comparison, if you ask for the full count of data, it has to find every record by definition. It sees a vast set of data and picks the choices that are best for iterating through that entire set of data instead of just a tiny sliver of it.
If, on the other hand, it really was correct and the records were very well correlated it would have found your record with the smallest possible amount of server resources and maximized its overall throughput.
I am having problem in fetching a number of records from while joining tables. Please see the below query:
SELECT
H.EIN,
H.OUC,
(
SELECT
COUNT(1)
FROM
tbl_Checks C
INNER JOIN INFM_People_OR.dbo.tblHierarchy P
ON P.EIN = C.EIN_Checked
WHERE
(H.EIN IN (P.L1, P.L2)
OR H.EIN = C.EIN_Checked)
AND C.[Read] = 1
) AS [Read]
FROM
INFM_People_OR.dbo.tblHierarchy H
LEFT JOIN tbl_Checks C
ON H.EIN = C.EIN_Checked
WHERE
H.L1 = #EIN
GROUP BY
H.EIN,
H.OUC,
C.Check_Date
Even if there are just 100 records this query takes a much more time(around 1 min).
Please suggest a solution to tune up this query as it is throwing error in front end
Given just the query there are a few things that stick out as being non-optimal:
Any use of OR will be slower:
WHERE
(H.EIN IN (P.L1, P.L2)
OR H.EIN = C.EIN_Checked)
AND C.[Read] = 1
If there's any way to rework this based off of your data set so that both the IN and the OR are replaced with ANDs that would help.
Also, use of a local variable in the WHERE clause will not work well with the optimizer:
WHERE
H.L1 = #EIN
Finally, make sure you have indexes (and hopefully these are integer fields) where you are doing your joins and group bys (H.EIN, H.OUC, C.Check_Date
The size of the result set (100 records) doesn't matter as much as the size of the joined tables and whether or not they have appropriate indexes.
The Estimated number of rows affected is 1196880 is very high resulting in high execution time of query. I have also tried to join the tables only once but that it giving different output.
Please suggest any other solution than creating indices as I have already created non-clustered index for the table tbl_checks but it doesn't make any difference.
Below is the SQl execution plan.
I've read many articles about parameter sniffing, but it's not clear if this is good or bad. Can anyone explain this with a simple example.
Is there a way to automatically detect that wrong plan was assigned to a specific statement?
Thanks in advance.
It is good but can be bad sometimes.
Parameter sniffing is about the query optimizer using the value of the provided parameter to figure out the best query plan possible. One of many choices and one that is pretty easy to understand is if the entire table should be scanned to get the values or if it will be faster using index seeks. If the value in your parameter is highly selective the optimizer will probably build a query plan with seeks and if it is not the query will do a scan of your table.
The query plan is then cached and reused for consecutive queries that have different values. The bad part of parameter sniffing is when the cached plan is not the best choice for one of those values.
Sample data:
create table T
(
ID int identity primary key,
Value int not null,
AnotherValue int null
);
create index IX_T_Value on T(Value);
insert into T(Value) values(1);
insert into T(Value)
select 2
from sys.all_objects;
T is a table with a couple of thousand rows with a non clustered index on Value. There is one row where value is 1 and the rest has the value 2.
Sample query:
select *
from T
where Value = #Value;
The choices the query optimizer has here is either to do a Clustered Index Scan and check the where clause against every row or use an Index Seek to find to rows that match and then do a Key Lookup to get the values from the columns asked for in the column list.
When the sniffed value is 1 the query plan will look like this:
And when the sniffed value is 2 it will look like this:
The bad part of parameter sniffing in this case happens when the query plan is built sniffing a 1 but executed later on with the value of 2.
You can see that the Key Lookup was executed 2352 times. A scan would clearly be the better choice.
To summarize I would say that parameter sniffing is a good thing that you should try to make happen as much as possible by using parameters to your queries. Sometimes it can go wrong and in those cases it is most likely due to skewed data that is messing with your statistics.
Update:
Here is a query against a couple of dmv's that you can use to find what queries are most expensive on your system. Change to order by clause to use different criteria on what you are looking for. I think that TotalDuration is a good place to start.
set transaction isolation level read uncommitted;
select top(10)
PlanCreated = qs.creation_time,
ObjectName = object_name(st.objectid),
QueryPlan = cast(qp.query_plan as xml),
QueryText = substring(st.text, 1 + (qs.statement_start_offset / 2), 1 + ((isnull(nullif(qs.statement_end_offset, -1), datalength(st.text)) - qs.statement_start_offset) / 2)),
ExecutionCount = qs.execution_count,
TotalRW = qs.total_logical_reads + qs.total_logical_writes,
AvgRW = (qs.total_logical_reads + qs.total_logical_writes) / qs.execution_count,
TotalDurationMS = qs.total_elapsed_time / 1000,
AvgDurationMS = qs.total_elapsed_time / qs.execution_count / 1000,
TotalCPUMS = qs.total_worker_time / 1000,
AvgCPUMS = qs.total_worker_time / qs.execution_count / 1000,
TotalCLRMS = qs.total_clr_time / 1000,
AvgCLRMS = qs.total_clr_time / qs.execution_count / 1000,
TotalRows = qs.total_rows,
AvgRows = qs.total_rows / qs.execution_count
from sys.dm_exec_query_stats as qs
cross apply sys.dm_exec_sql_text(qs.sql_handle) as st
cross apply sys.dm_exec_text_query_plan(qs.plan_handle, qs.statement_start_offset, qs.statement_end_offset) as qp
--order by ExecutionCount desc
--order by TotalRW desc
order by TotalDurationMS desc
--order by AvgDurationMS desc
;
Yes, sometime it is good or bad.
The Many time query optimizer chooses old query plan for execution because it stores this plan into the cache for frequently running queries.
Now what happened when old query plan has table scan parameter which is require to change for index scanning after increasing records.
I found that in my situation query optimizer use old query plan instead of to create a new query plan.The query optimizer was using an old query plan from the query cache.
I have created very interesting post on Parameter Sniffing. Please visit this url:
http://www.dbrnd.com/2015/05/sql-server-parameter-sniffing/
I am writting a stored procedure that will get data for a sales report. Queries are like this:
INSERT INTO #FirstQuery
SELECT t1.*, t2.*
FROM t1
LEFT JOIN t2 ON t1.idT1 = t2.idT1
LEFT JOIN t3 ON t3.idT2 = t2.idT2
WHERE t1.nonIndexedField = #parameter1
AND (t2.idT2 IS NULL
OR
(#parameter2 = 'XXX' AND t1.indexedField1 = #parameter3)
OR
(#parameter2 = 'YYY' AND t3.indexedField1 = #parameter3)
)
With those results, I then fill a second table variable:
INSERT INTO #SecondQuery
SELECT u1.*, u2.*
FROM u1
INNER JOIN u2 ON u2.idU1 = u1.idU1
WHERE u1.NONindexedField in (SELECT someField FROM #FirstQuery)
As it was being quite slow on QA environment, I watched the execution plans. First I took a look at the Estimated plan. It saw that SecondQuery was taking really long, and I realized that u1.NONindexedField didn't have an index and was taking an estimated 97% of the total cost.
But then I took a look at the Actual plan, and it said that FirstQuery was takin 100% of total cost. I checked the estimated rows calculated on the estimated plan and it many places it estimated very few rows where the actual plan showed around a hundred thouthand (100K). I thought it was because the field missing an index was in a table with not so many rows (17K), but I created the index anyway. To my surprise, the query time was reduced from 500 to 15 seconds.
So, my question is... why such a difference in execution plans, and how come the actual plan was way off? I know that Estimated plan doesn't really mean "an estimation of the plan" but rather "a plan with estimated row counts", but that doesn't explain the difference, and certanly doesn't explain why actual plan told me that all the cost was on a query that didn't need optimizing...
BTW, I compared relative times and the second query indeed takes around 97% of total execution time.
Thanks for reading!
If the actual plan was way off this could be caused by outdated statistics.
Have you tried updating statistics on the tables in the query.
If the statistics are out of date, this can cause the execution plans to be off. With out updated statistics your queries may be very inefficient.
the syntax for updating statistics is:
update statistics tablename;
Try updating the statistics and see if you are getting more accurate execution plans.
During query optimization I encounted a strange behaviour of sql server (Sql Server 2008 R2 Enterprise). I created several indexes on tables, as well as some indexed views. I have two queries, for example:
select top 10 N0."Oid",N1."ObjectType",N1."OptimisticLockField" from ((("dbo"."Issue" N0
inner join "dbo"."Article" N1 on (N0."Oid" = N1."Oid"))
inner join "dbo"."ProductLink" N2 on (N1."ProductLink" = N2."Oid"))
inner join "dbo"."Technology" N3 on (N2."Technology" = N3."Oid"))
where (N1."GCRecord" is null and (N0."IsPrivate" = 0) and ((N0."HasMarkedAnswers" = 0) or N0."HasMarkedAnswers" is null) and (N3."Name" = N'Discussions'))
order by N1."ModifiedOn" desc
and
select top 30 N0."Oid",N1."ObjectType",N1."OptimisticLockField" from ((("dbo"."Issue" N0
inner join "dbo"."Article" N1 on (N0."Oid" = N1."Oid"))
inner join "dbo"."ProductLink" N2 on (N1."ProductLink" = N2."Oid"))
inner join "dbo"."Technology" N3 on (N2."Technology" = N3."Oid"))
where (N1."GCRecord" is null and (N0."IsPrivate" = 0) and ((N0."HasMarkedAnswers" = 0) or N0."HasMarkedAnswers" is null) and (N3."Name" = N'Discussions'))
order by N1."ModifiedOn" desc
both queries are the same, except first starts with select top 10 and second with select top 30. Both queries returns the same result set - 6 rows. But the second query is 5 times faster then the first one! I looked at the actual execution plans for both queries, and of course, they differs. Second query uses indexed view, and performs great, and the first query denies to use it, using indexes on tables instead. I repeat myself - both queries are the same, to the same table, at the same server, they differs only by number in "top" part.
I tried to force optimizer to use indexed view in the first query by updating statistics, destroing indexes it used and so on. No matter how I try actual execution do not use indexed view for the first query and always use it for the second one.
I am really intrested in the reasons causing such behavior. Any suggestions?
Update I am not sure that it can help without decribing corresponding indexes and view, but this is actual execution plan diagramms:
for select top 19:
for select top 18:
another confusing fact is that for the select top 19 query sometimes indexed view is used, sometimes not
The only thing I can think of is perhaps the optimizer in the first query concluded that the specifying criteria is not selective enough for the "better" execution plan to be used.
If you are still investigating this see if TOP 60, 90, 100, ... produces the second execution plan and performs well. You could also tinker with it to see what the threshold is for the optimizer to select the second plan in this case.
Also try the queries without the order by statement to see if that is affecting the selection of the query plan (check the index on that field, etc)
Beyond that, you said you can't use index hints so perhaps a re-write where you select top X from your Article table (N1) with a bunch of exists statements in your where clause would provide better performance for you.