I'm trying to work on SQL Server with some parameters that can be NULL, where NULL means "ignore this parameter".
Then I have the column where the middle name is stored and can contain nulls.
I have the following conditions that works really fast:
T.tr_ben_name = ISNULL(#BenFirstName, T.tr_send_name) AND
T.tr_ben_middle = ISNULL(#BenMiddleName, T.tr_send_middle) AND
T.tr_ben_last = ISNULL(#BenLastName, T.tr_send_last) AND
T.tr_ben_last2 = ISNULL(#BenSecondLastName, T.tr_send_last2 )
But for some reason if the middle name value and the corresponding parameter are both NULL the record will be skipped, even if I turn off ANSI NULLS.
Then I came up with this other version that works well but 4 times slower:
(T.tr_ben_name = #BenFirstName OR #BenFirstName IS NULL) AND
(T.tr_ben_middle = #BenMiddleName OR #BenMiddleName IS NULL) AND
(T.tr_ben_last = #BenLastName OR #BenLastName IS NULL) AND
(T.tr_ben_last2 = #BenSecondLastName OR #BenSecondLastName IS NULL)
Can anyone explain what is the difference between these 2 approaches?
This is a short summary of why the queries perform differently, and what you can do to help the performance. For more details, see Catch-All Queries and Revisiting Catch-all Queries by Gail Shaw. For an exhaustive analysis, see Dynamic Search Conditions in T‑SQL By Erland Sommarskog.
Basically, the query with conditional parameters tries to create a query plan that works for all possible combinations of parameters passed in. This means that it doesn't default to lookup by index seek if you pass in the seekable parameters only. Instead, it uses some query plan that is usable for any combination of parameters.
Basic fixes for this issue are
Add OPTION (RECOMPILE) on the end of your query (SQL 2008 R2 SP1, 2008 SP3, or higher only)
Use Dynamic SQL. Only add the conditions being checked with non-null parameters.
If you want the full detailed whys and wherefores, the articles above are very good.
Related
Sorry if I'm using the wrong terminology - I'm not into MSSQL or VBScript.
I have a given script implementing a SQL query containing
.. AND (rep.is_primary_replica is null or rep.is_primary_replica = ''True'' or rep.is_primary_replica = ''False'') ..
which returns no results on a German server only because rep.is_primary_replica seems to contain ''Wahr'' and ''Falsch'' instead of ''True'' and ''False''
This question is about an at least similar problem.
Unfortunately there is no wtf flag.
Is there a way to do that correctly?
Can I disable localization of string conversions? (in MS SQL Server? VBS?)
Don't let printed values in the script confuse you. There is an implicit conversion from original value to string which depends on the client's (VBScript in your case I think) locale.
Type of the field is_primary_replica here must be bit if the rep is an instance of sys.dm_hadr_database_replica_states view.
If this is the case, it's pointless to check a bit field is Null, or its value equal to True or False, there's no other possibility anyway.
It appears that you can safely remove this condition from the query.
If you insist to include it, use number literals instead.
AND (rep.is_primary_replica is null or rep.is_primary_replica = 1 or rep.is_primary_replica = 0)
This is the proper way to query a bit field. With this way the server's or client's locale configuration won't cause any problems.
I have a parameterized query with optional parameters.
Multiple tables are joined.
A part of the WHERE clause looks like this:
and ((x.a = #arg1) OR (#arg1 IS NULL))
and ((y.b = #arg2) OR (#arg2 IS NULL))
and ((z.c = #arg3) OR (#arg3 IS NULL))
So, the idea is: A parameter can either be used for applying a filter, or, if the parameter is NULL, then no filtering will be applied.
However, I found that the execution plan is not good for this code.
When a parameter is actually set, then it is much better to write
and x.a = #arg1
instead of
and ((x.a= #arg1) OR (#arg1 IS NULL))
Actually, I have in total 8 tables which are joined together. In both statements, all 8 tables are joined, and there are the same index seeks/scans applied on all these tables. However, the join order is different, thus resulting in different execution speeds.
Is there a way to rewrite the above statement, such that the execution plan can work optimal? Possibly with some query hints ?
Or is there no way around writing dynamic SQL? I want to avoid the latter, because
dynamic SQL is hard to read,
SSMS does not show dependencies,
passing the parameters into the dynamic SQL is awful
try using COALESCE(#arg1, x.a) instead of ((x.a = #arg1) OR (#arg1 IS NULL))
and x.a = COALESCE(#arg1, x.a)
and y.b = COALESCE(#arg2, xyb)
and z.c = COALESCE(#arg3, z.c)
Unfortunately the optimiser will tend to stick with whatever plan suits the first invocation. Your best options are to write a stored procedure for each combination (or at least the major ones) or try OPTION(RECOMPILE) which will take the time to evaluate the actual parameters, but at some cost.
To answer your question about "Is there a way to rewrite the above statement, such that the execution plan can work optimal?", the answer is probably not. There are too many variations to come up with a single plan with so many things that may not occur (your null variables).
I'm writing a fairly complex stored procedure to search an image library.
I was going to use a view and write dynamic sql to query the view, but I need to use a full text index, and my view needs outer joins (MS-SQL 2005 full-text index on a view with outer joins)
So, I'm back to a stored procedure.
I need to search on (all optional):
a general search query that uses the full text index (or no search terms)
one or more categories (or none)
a single tag (or none)
Is there a way to do a conditional FREETEXT in the 'WHERE' clause? The query may be empty, in which case I want to ignore this, or just return all FTI matches.
...AND FREETEXT(dbo.MediaLibraryCultures.*, '"* "') doesn't seem to work. Not sure how a case statement would work here.
Am I better off inserting the category/tag filter results into a temp table/table variable, then joining the FTI search results? That way I can only do the join if the search term is supplied.
Thoughts?
I know it's a year later and a newer version of SQL but FYI...
I am using SQL Server 2008 and have tried to short circuit using
AND ( #searchText = '' OR freetext(Name, #searchText))
and I receive the message "Null or empty full-text predicate" when setting #searchText = ''. I guess something in 2008 has changed that keeps short circuiting from working in this case.
You could add a check for the empty search string like
where ...
AND (FREETEXT(dbo.MediaLibraryCultures.*, #FreeTextSearchFor) OR #FreeTextSearchFor = '')
(I have a feeling that freetext searches can't have null passed into them, so I'm comparing to an empty string)
If the term to search for is empty, the whole clause will evaluate to true, so no restrictions will be applied (by this clause) to the rows returned, and of course since its a constant being compared to a variable - I would think the optimizer would come into play and not perform that comparison for each row.
Hmm, I thought there was no short-circuiting in sql server?
AND (#q = '' OR FREETEXT(dbo.MediaLibraryCultures.*, #q))
seems to work just fine!
Strangely, the full text scan is still part of the execution plan.
Doesn't work on SQL Server 2014. I tried the suggested short circuit in one of my stored procedures, but it keeps evaluating the FREETEXT expression. The only solution I have found is the following:
IF ISNULL(#Text, N'') = N'' SET #Text = N'""'
SELECT ...
WHERE ...
AND (#Text = '""' OR FREETEXT([Data], #Text)
I've been battling this one for a while now. I have a stored proc that takes in 3 parameters that are used to filter. If a specific value is passed in, I want to filter on that. If -1 is passed in, give me all.
I've tried it the following two ways:
First way:
SELECT field1, field2...etc
FROM my_view
WHERE
parm1 = CASE WHEN #PARM1= -1 THEN parm1 ELSE #PARM1 END
AND parm2 = CASE WHEN #PARM2 = -1 THEN parm2 ELSE #PARM2 END
AND parm3 = CASE WHEN #PARM3 = -1 THEN parm3 ELSE #PARM3 END
Second Way:
SELECT field1, field2...etc
FROM my_view
WHERE
(#PARM1 = -1 OR parm1 = #PARM1)
AND (#PARM2 = -1 OR parm2 = #PARM2)
AND (#PARM3 = -1 OR parm3 = #PARM3)
I read somewhere that the second way will short circuit and never eval the second part if true. My DBA said it forces a table scan. I have not verified this, but it seems to run slower on some cases.
The main table that this view selects from has somewhere around 1.5 million records, and the view proceeds to join on about 15 other tables to gather a bunch of other information.
Both of these methods are slow...taking me from instant to anywhere from 2-40 seconds, which in my situation is completely unacceptable.
Is there a better way that doesn't involve breaking it down into each separate case of specific vs -1 ?
Any help is appreciated. Thanks.
I read somewhere that the second way will short circuit and never eval the second part if true. My DBA said it forces a table scan.
You read wrong; it will not short circuit. Your DBA is right; it will not play well with the query optimizer and likely force a table scan.
The first option is about as good as it gets. Your options to improve things are dynamic sql or a long stored procedure with every possible combination of filter columns so you get independent query plans. You might also try using the "WITH RECOMPILE" option, but I don't think it will help you.
if you are running SQL Server 2005 or above you can use IFs to make multiple version of the query with the proper WHERE so an index can be used. Each query plan will be placed in the query cache.
also, here is a very comprehensive article on this topic:
Dynamic Search Conditions in T-SQL by Erland Sommarskog
it covers all the issues and methods of trying to write queries with multiple optional search conditions
here is the table of contents:
Introduction
The Case Study: Searching Orders
The Northgale Database
Dynamic SQL
Introduction
Using sp_executesql
Using the CLR
Using EXEC()
When Caching Is Not Really What You Want
Static SQL
Introduction
x = #x OR #x IS NULL
Using IF statements
Umachandar's Bag of Tricks
Using Temp Tables
x = #x AND #x IS NOT NULL
Handling Complex Conditions
Hybrid Solutions – Using both Static and Dynamic SQL
Using Views
Using Inline Table Functions
Conclusion
Feedback and Acknowledgements
Revision History
If you pass in a null value when you want everything, then you can write your where clause as
Where colName = IsNull(#Paramater, ColName)
This is basically same as your first method... it will work as long as the column itself is not nullable... Null values IN the column will mess it up slightly.
The only approach to speed it up is to add an index on the column being filtered on in the Where clause. Is there one already? If not, that will result in a dramatic improvement.
No other way I can think of then doing:
WHERE
(MyCase IS NULL OR MyCase = #MyCaseParameter)
AND ....
The second one is more simpler and readable to ther developers if you ask me.
SQL 2008 and later make some improvements to optimization for things like (MyCase IS NULL OR MyCase = #MyCaseParameter) AND ....
If you can upgrade, and if you add an OPTION (RECOMPILE) to get decent perf for all possible param combinations (this is a situation where there is no single plan that is good for all possible param combinations), you may find that this performs well.
http://blogs.msdn.com/b/bartd/archive/2009/05/03/sometimes-the-simplest-solution-isn-t-the-best-solution-the-all-in-one-search-query.aspx
By definition (at least from what I've seen) sargable means that a query is capable of having the query engine optimize the execution plan that the query uses. I've tried looking up the answers, but there doesn't seem to be a lot on the subject matter. So the question is, what does or doesn't make an SQL query sargable? Any documentation would be greatly appreciated.
For reference: Sargable
The most common thing that will make a query non-sargable is to include a field inside a function in the where clause:
SELECT ... FROM ...
WHERE Year(myDate) = 2008
The SQL optimizer can't use an index on myDate, even if one exists. It will literally have to evaluate this function for every row of the table. Much better to use:
WHERE myDate >= '01-01-2008' AND myDate < '01-01-2009'
Some other examples:
Bad: Select ... WHERE isNull(FullName,'Ed Jones') = 'Ed Jones'
Fixed: Select ... WHERE ((FullName = 'Ed Jones') OR (FullName IS NULL))
Bad: Select ... WHERE SUBSTRING(DealerName,4) = 'Ford'
Fixed: Select ... WHERE DealerName Like 'Ford%'
Bad: Select ... WHERE DateDiff(mm,OrderDate,GetDate()) >= 30
Fixed: Select ... WHERE OrderDate < DateAdd(mm,-30,GetDate())
Don't do this:
WHERE Field LIKE '%blah%'
That causes a table/index scan, because the LIKE value begins with a wildcard character.
Don't do this:
WHERE FUNCTION(Field) = 'BLAH'
That causes a table/index scan.
The database server will have to evaluate FUNCTION() against every row in the table and then compare it to 'BLAH'.
If possible, do it in reverse:
WHERE Field = INVERSE_FUNCTION('BLAH')
This will run INVERSE_FUNCTION() against the parameter once and will still allow use of the index.
In this answer I assume the database has sufficient covering indexes. There are enough questions about this topic.
A lot of the times the sargability of a query is determined by the tipping point of the related indexes. The tipping point defines the difference between seeking and scanning an index while joining one table or result set onto another. One seek is of course much faster than scanning a whole table, but when you have to seek a lot of rows, a scan could make more sense.
So among other things a SQL statement is more sargable when the optimizer expects the number of resulting rows of one table to be less than the tipping point of a possible index on the next table.
You can find a detailed post and example here.
For an operation to be considered sargable, it is not sufficient for it to just be able to use an existing index. In the example above, adding a function call against an indexed column in the where clause, would still most likely take some advantage of the defined index. It will "scan" aka retrieve all values from that column (index) and then eliminate the ones that do not match to the filter value provided. It is still not efficient enough for tables with high number of rows.
What really defines sargability is the query ability to traverse the b-tree index using the binary search method that relies on half-set elimination for the sorted items array. In SQL, it would be displayed on the execution plan as a "index seek".