I've been battling this one for a while now. I have a stored proc that takes in 3 parameters that are used to filter. If a specific value is passed in, I want to filter on that. If -1 is passed in, give me all.
I've tried it the following two ways:
First way:
SELECT field1, field2...etc
FROM my_view
WHERE
parm1 = CASE WHEN #PARM1= -1 THEN parm1 ELSE #PARM1 END
AND parm2 = CASE WHEN #PARM2 = -1 THEN parm2 ELSE #PARM2 END
AND parm3 = CASE WHEN #PARM3 = -1 THEN parm3 ELSE #PARM3 END
Second Way:
SELECT field1, field2...etc
FROM my_view
WHERE
(#PARM1 = -1 OR parm1 = #PARM1)
AND (#PARM2 = -1 OR parm2 = #PARM2)
AND (#PARM3 = -1 OR parm3 = #PARM3)
I read somewhere that the second way will short circuit and never eval the second part if true. My DBA said it forces a table scan. I have not verified this, but it seems to run slower on some cases.
The main table that this view selects from has somewhere around 1.5 million records, and the view proceeds to join on about 15 other tables to gather a bunch of other information.
Both of these methods are slow...taking me from instant to anywhere from 2-40 seconds, which in my situation is completely unacceptable.
Is there a better way that doesn't involve breaking it down into each separate case of specific vs -1 ?
Any help is appreciated. Thanks.
I read somewhere that the second way will short circuit and never eval the second part if true. My DBA said it forces a table scan.
You read wrong; it will not short circuit. Your DBA is right; it will not play well with the query optimizer and likely force a table scan.
The first option is about as good as it gets. Your options to improve things are dynamic sql or a long stored procedure with every possible combination of filter columns so you get independent query plans. You might also try using the "WITH RECOMPILE" option, but I don't think it will help you.
if you are running SQL Server 2005 or above you can use IFs to make multiple version of the query with the proper WHERE so an index can be used. Each query plan will be placed in the query cache.
also, here is a very comprehensive article on this topic:
Dynamic Search Conditions in T-SQL by Erland Sommarskog
it covers all the issues and methods of trying to write queries with multiple optional search conditions
here is the table of contents:
Introduction
The Case Study: Searching Orders
The Northgale Database
Dynamic SQL
Introduction
Using sp_executesql
Using the CLR
Using EXEC()
When Caching Is Not Really What You Want
Static SQL
Introduction
x = #x OR #x IS NULL
Using IF statements
Umachandar's Bag of Tricks
Using Temp Tables
x = #x AND #x IS NOT NULL
Handling Complex Conditions
Hybrid Solutions – Using both Static and Dynamic SQL
Using Views
Using Inline Table Functions
Conclusion
Feedback and Acknowledgements
Revision History
If you pass in a null value when you want everything, then you can write your where clause as
Where colName = IsNull(#Paramater, ColName)
This is basically same as your first method... it will work as long as the column itself is not nullable... Null values IN the column will mess it up slightly.
The only approach to speed it up is to add an index on the column being filtered on in the Where clause. Is there one already? If not, that will result in a dramatic improvement.
No other way I can think of then doing:
WHERE
(MyCase IS NULL OR MyCase = #MyCaseParameter)
AND ....
The second one is more simpler and readable to ther developers if you ask me.
SQL 2008 and later make some improvements to optimization for things like (MyCase IS NULL OR MyCase = #MyCaseParameter) AND ....
If you can upgrade, and if you add an OPTION (RECOMPILE) to get decent perf for all possible param combinations (this is a situation where there is no single plan that is good for all possible param combinations), you may find that this performs well.
http://blogs.msdn.com/b/bartd/archive/2009/05/03/sometimes-the-simplest-solution-isn-t-the-best-solution-the-all-in-one-search-query.aspx
Related
I'm trying to work on SQL Server with some parameters that can be NULL, where NULL means "ignore this parameter".
Then I have the column where the middle name is stored and can contain nulls.
I have the following conditions that works really fast:
T.tr_ben_name = ISNULL(#BenFirstName, T.tr_send_name) AND
T.tr_ben_middle = ISNULL(#BenMiddleName, T.tr_send_middle) AND
T.tr_ben_last = ISNULL(#BenLastName, T.tr_send_last) AND
T.tr_ben_last2 = ISNULL(#BenSecondLastName, T.tr_send_last2 )
But for some reason if the middle name value and the corresponding parameter are both NULL the record will be skipped, even if I turn off ANSI NULLS.
Then I came up with this other version that works well but 4 times slower:
(T.tr_ben_name = #BenFirstName OR #BenFirstName IS NULL) AND
(T.tr_ben_middle = #BenMiddleName OR #BenMiddleName IS NULL) AND
(T.tr_ben_last = #BenLastName OR #BenLastName IS NULL) AND
(T.tr_ben_last2 = #BenSecondLastName OR #BenSecondLastName IS NULL)
Can anyone explain what is the difference between these 2 approaches?
This is a short summary of why the queries perform differently, and what you can do to help the performance. For more details, see Catch-All Queries and Revisiting Catch-all Queries by Gail Shaw. For an exhaustive analysis, see Dynamic Search Conditions in T‑SQL By Erland Sommarskog.
Basically, the query with conditional parameters tries to create a query plan that works for all possible combinations of parameters passed in. This means that it doesn't default to lookup by index seek if you pass in the seekable parameters only. Instead, it uses some query plan that is usable for any combination of parameters.
Basic fixes for this issue are
Add OPTION (RECOMPILE) on the end of your query (SQL 2008 R2 SP1, 2008 SP3, or higher only)
Use Dynamic SQL. Only add the conditions being checked with non-null parameters.
If you want the full detailed whys and wherefores, the articles above are very good.
I am dealing with some legacy code that looks like this:
Declare #PolicyId int
;
Select top 1 #PolicyId = policyid from policytab
;
Select col002
From SOMETAB
Where (cast(Col001 as int) = #PolicyId)
;
The code is actually in a loop, but the problem is the same. col001 is a CHAR(10)
Is there a way to specify an index on SOMETAB.Col001 that would speed up this code?
Are there other ways to speed up this code without modifying it?
The context of that question is that I am guessing that a simple index on col001 will not speed up the code because the select statement doing a cast on the column.
I am looking for a solution that does not involve changing this code because this technique was used on several tables and in several scripts for each table.
Once I determine that it is hopeless to speed up this code without changing it I have several options. I am bringing that so this post can stay on the topic of speeding up the code without changing the code.
Shortcut to hopeless if you can not change the code, (cast(Col001 as int) = #PolicyId) is not SARGable.
Sargable
SARGable functions in SQL Server - Rob Farley
SARGable expressions and performance - Daniel Hutmachier
After shortcut, avoid loops when possible and keep your search arguments SARGable. Indexed persisted computed columns are an option if you must maintain the char column and must compare to an integer.
If you cannot change the table structure, cast your parameter to the data type you are searching on in your select statement.
Cast(#PolicyId as char(10)) . This is a code change, and a good place to start looking if you decide to change code based on sqlZim's answer.
Zim's advice is excellent, and searching on int will always be faster than char. But, you may find this method an acceptable alternative to any schema changes.
Is policy stored as an int in PolicyTab and char in SomeTab for certain?
I have a parameterized query with optional parameters.
Multiple tables are joined.
A part of the WHERE clause looks like this:
and ((x.a = #arg1) OR (#arg1 IS NULL))
and ((y.b = #arg2) OR (#arg2 IS NULL))
and ((z.c = #arg3) OR (#arg3 IS NULL))
So, the idea is: A parameter can either be used for applying a filter, or, if the parameter is NULL, then no filtering will be applied.
However, I found that the execution plan is not good for this code.
When a parameter is actually set, then it is much better to write
and x.a = #arg1
instead of
and ((x.a= #arg1) OR (#arg1 IS NULL))
Actually, I have in total 8 tables which are joined together. In both statements, all 8 tables are joined, and there are the same index seeks/scans applied on all these tables. However, the join order is different, thus resulting in different execution speeds.
Is there a way to rewrite the above statement, such that the execution plan can work optimal? Possibly with some query hints ?
Or is there no way around writing dynamic SQL? I want to avoid the latter, because
dynamic SQL is hard to read,
SSMS does not show dependencies,
passing the parameters into the dynamic SQL is awful
try using COALESCE(#arg1, x.a) instead of ((x.a = #arg1) OR (#arg1 IS NULL))
and x.a = COALESCE(#arg1, x.a)
and y.b = COALESCE(#arg2, xyb)
and z.c = COALESCE(#arg3, z.c)
Unfortunately the optimiser will tend to stick with whatever plan suits the first invocation. Your best options are to write a stored procedure for each combination (or at least the major ones) or try OPTION(RECOMPILE) which will take the time to evaluate the actual parameters, but at some cost.
To answer your question about "Is there a way to rewrite the above statement, such that the execution plan can work optimal?", the answer is probably not. There are too many variations to come up with a single plan with so many things that may not occur (your null variables).
I'm using dapper-dot-net as an ORM and it produces the following, slow-executing (1700ms), SQL code.
exec sp_executesql N'SELECT TOP 5 SensorValue FROM "Values"
WHERE DeviceId IN (#id1,#id2) AND SensorId = #sensor
AND SensorValue != -32768 AND SensorValue != -32767',N'#id1
bigint,#id2 bigint,#sensor int',#id1=139,#id2=726,#sensor=178
When I modify this code by removing the parameters the query executes blazingly fast (20ms). Should the lack of these parameters actually make this big difference and why?
exec sp_executesql N'SELECT TOP 5 SensorValue FROM "Values"
WHERE DeviceId IN (139,726) AND SensorId = 178
AND SensorValue != -32768 AND SensorValue != -32767'
Add OPTION (RECOMPILE) to the end
... AND SensorValue != -32767 OPTION (RECOMPILE)
I suspect you are experiencing "parameter sniffing"
If that's the case we can leave it with the OPTION or consider alternatives
Update 1
The following article will introduce you to "parameter sniffing" http://pratchev.blogspot.be/2007/08/parameter-sniffing.html
I advice that you get to know the ins and out because it will make you much better in understanding sql server internals (that can bite).
If you understand it you will know that the tradeoff with option recompile can be a performance decrease if the statement is executed very often.
I personally add option recompile after I know the root cause is parameter sniffing and leave it in unless there is a performance issue. Rewriting a statement to avoid bad parameter sniffing leads to loss of intent and this lowers maintainability. But there are cases when the rewrite is justified (use good comments when you do).
Update 2
The best read I had on the subject was in chapter 32 called
"Parameter sniffing: your best friend... except when it isn't by " by GRANT FRITCHEY
It's recommended.
SQL Server MVP Deep Dives, Volume 2
I Recently ran into the same issue. The first thing I did was adding a NonClustered Covering Index on the columns in my where statement.
This improved the execution time on SQL, but when dapper was performing the query it was still slow, in fact it was timing out.
Then i realized that the query generated by dapper was passing in the parameter as nvarchar(4000) where as my db table column was a varchar(80) this caused it to perform an index scan instead of a seek (I suggest you read up on indexes if that does not make sense to you.). upon realizing this I updated my dapper where statement to be like this:
WHERE Reference = convert(varchar(80),#Reference)
Executing the with the where statement above resulted in an index seek, and 100% performance improvement.
Just to Add: The Option(Recompile) did not work for me.
And after all this song and dance, there is a way to tell dapper to do this for you by default:
Dapper.SqlMapper.AddTypeMap(typeof(string),
System.Data.DbType.AnsiString);
This will by default map any string parameters to a varchar(4000) rather than a nvarchar(4000). If you do need Unicode string comparison, then you can explicitly do the conversion on the parameter.
I'm writing a fairly complex stored procedure to search an image library.
I was going to use a view and write dynamic sql to query the view, but I need to use a full text index, and my view needs outer joins (MS-SQL 2005 full-text index on a view with outer joins)
So, I'm back to a stored procedure.
I need to search on (all optional):
a general search query that uses the full text index (or no search terms)
one or more categories (or none)
a single tag (or none)
Is there a way to do a conditional FREETEXT in the 'WHERE' clause? The query may be empty, in which case I want to ignore this, or just return all FTI matches.
...AND FREETEXT(dbo.MediaLibraryCultures.*, '"* "') doesn't seem to work. Not sure how a case statement would work here.
Am I better off inserting the category/tag filter results into a temp table/table variable, then joining the FTI search results? That way I can only do the join if the search term is supplied.
Thoughts?
I know it's a year later and a newer version of SQL but FYI...
I am using SQL Server 2008 and have tried to short circuit using
AND ( #searchText = '' OR freetext(Name, #searchText))
and I receive the message "Null or empty full-text predicate" when setting #searchText = ''. I guess something in 2008 has changed that keeps short circuiting from working in this case.
You could add a check for the empty search string like
where ...
AND (FREETEXT(dbo.MediaLibraryCultures.*, #FreeTextSearchFor) OR #FreeTextSearchFor = '')
(I have a feeling that freetext searches can't have null passed into them, so I'm comparing to an empty string)
If the term to search for is empty, the whole clause will evaluate to true, so no restrictions will be applied (by this clause) to the rows returned, and of course since its a constant being compared to a variable - I would think the optimizer would come into play and not perform that comparison for each row.
Hmm, I thought there was no short-circuiting in sql server?
AND (#q = '' OR FREETEXT(dbo.MediaLibraryCultures.*, #q))
seems to work just fine!
Strangely, the full text scan is still part of the execution plan.
Doesn't work on SQL Server 2014. I tried the suggested short circuit in one of my stored procedures, but it keeps evaluating the FREETEXT expression. The only solution I have found is the following:
IF ISNULL(#Text, N'') = N'' SET #Text = N'""'
SELECT ...
WHERE ...
AND (#Text = '""' OR FREETEXT([Data], #Text)