I have this quite simple query:
IF EXISTS (SELECT 1 FROM system.RawEvent_pool1 WHERE ProcessedDate IS NULL)
EXECUTE [system].[usp_ProcessAuditData] '1'
I have a filtered index on system.RawEvent_pool1.
It is filtered on ProcessedDate IS NULL
(ProcessedDate is included in the index)
So the index should be a perfect match for this query.
From the actual execution plan, I can tell, that when executed from Management Studio the index is used as expected.
But when executing the exact same query on the same database from a SQL Job Agent, the index is not used.
Instead, the query plan uses the clustered index on the primary key for the table.
This I can see in the Query Store.
Additional info: The job is executed under a different account.
Is there any logical explanation for this behavior?
Then I would love to hear it :-)
Found the reason.
The SQL Job Agent has 'QUOTED_IDENTIFIER' set to 'OFF' by default.
Executing 'SET QUOTED_IDENTIFIER ON' will make the statement use the expected execution plan
But I still have no clue why this works.
Related
On my test server there is a large query that is running (which is okay), but it is a multi-line query. E.g. in SSMS I told it to run something like:
begin transaction;
query;
query;
query;
query;
commit;
I want to see which query within the list is executing. Selecting text from sys.dm_exec_sql_text returns the entire statement, not the particular command that is executing within the list. Is there a way to view the individual commands that are being processed?
In case it matters (sometimes it does), this is running on a SQL Azure instance.
Use DBCC INPUTBUFFER
DBCC INPUTBUFFER(your session id)
It will display the query that is executing in your session
Here you can find my complete set o queries useful to show transactions running, time-wait events and open transactions with plan and sql texts:
http://zaboilab.com/sql-server-toolbox/queries-to-see-rapidly-what-your-sql-server-is-doing-now
So I have a stored procedure in SQL Server. I've simplified its code (for this question) to just this:
CREATE PROCEDURE dbo.DimensionLookup as
BEGIN
select DimensionID, DimensionField from DimensionTable
inner join Reference on Reference.ID = DimensionTable.ReferenceID
END
In SSIS on SQL Server 2012, I have a Lookup component with the following source command:
EXECUTE dbo.DimensionLookup WITH RESULT SETS (
(DimensionID int, DimensionField nvarchar(700) )
)
When I run this procedure in Preview mode in BIDS, it returns the two columns correctly. When I run the package in BIDS, it runs correctly.
But when I deploy it out to the SSIS catalog (the same server the database is on), point it to the same data sources, etc. - it fails with the message:
EXECUTE statement failed because its WITH RESULT SETS clause specified 2 column(s) for result set number 1, but the statement sent
3 column(s) at run time.
Steps Tried So Far:
Adding a third column to the result set - I get a different error, VS_NEEDSNEWMETADATA - which makes sense, kind of proof there's no third column.
SQL Profiler - I see this:
exec sp_prepare #p1 output,NULL,N'EXECUTE dbo.DimensionLookup WITH RESULT SETS ((
DimensionID int, DimensionField nvarchar(700)))',1
SET FMTONLY ON exec sp_execute 1 SET FMTONLY OFF
So it's trying to use FMTONLY to get the result set data ... needless to say, running SET FMTONLY ON and then running the command in SSMS myself yields .. just the two columns.
SET NOTCOUNT ON - Nothing changed.
So, two other interesting things:
I deployed it out to my local SQL 2012 install and it worked fine, same connections, etc. So it may be a server / database configuration. Not sure what if anything it is, I didn't install the dev server and my own install was pretty much click through vanilla.
Perhaps the most interesting thing. If I remove the join from the procedure's statement so it just becomes
select DimensionID, DimensionField from DimensionTable
It goes back to just sending 2 columns in the result set! So adding a join, without adding any additional output columns, ups the result set to 3 columns. Even if I add 6 more joins, just 3 columns. So one guess is its some sort of metadata column that only gets activated when there's a join.
Anyway, as you can imagine, it's driving me kind of mad. I have a workaround to load the data into a temp table and just return that, but why won't this work? What extra column is being sent back? Why only when I add a join?
Gah!
So all credit to billinkc: The reason is because of a patch.
In Version 11.0.2100.60, SSIS Lookup SQL command metadata is gathered using the old SET FMTONLY method. Unfortunately, this doesn't work in 2012, as the Books Online entry on SET FMTONLY helpfully notes:
Do not use this feature. This feature has been replaced by sp_describe_first_result_set.
Too bad they didn't follow their own advice!
This has been patched as of version 11.0.2218.0. Metadata is correctly gathered using the sp_describe_first_result_set system stored procedure.
This can happen if the specified WITH results set in SSIS identifies that there are more columns than being returned by the stored proc being called. Check your stored proc and ensure that you have the correct number of output columns as the WITH results set.
I have been asked to help with performance issue of a SQL server installation. I am not a SQL Server expert, but I decided to take a look. We are using a closed source application that appears to work OK. However after a SQL Server upgrade from 2000 to 2005, application performance has reportedly suffered considerably. I ran SQL profiler and caught the following query (field names changed to protect the innocent) taking about 30 seconds to run. My first thought was that I should optimize the query. But that is not possible, given that the application is closed source and the vendor is not helpful. So I am left, trying to figure out how to make this query run fast without changing it. It is also not clear to me how this query ran faster on the older SQL server 2000 product. Perhaps there was some sort of performance tuning applied to on that instance that did not carry over or does not work on the new SQL server. DBCC PINTABLE comes to mind.
Anyway, here is the offending query:
select min(row_id) from Table1 where calendar_id = 'Test1'
and exists
(select id from Table1 where calendar_id = 'Test1' and
DATEDIFF(day, '12/30/2010 09:21', start_datetime) = 0
)
and exists
(select id from Table1 where calendar_id = 'Test1' and
DATEDIFF(day, end_datetime, '01/17/2011 09:03') = 0
);
Table1 has about 6200 entries and looks like this. I have tried creating various indices to no effect.
id calendar_id start_datetime end_datetime
int, primary key varchar(10) datetime datetime
1 Test1 2005-01-01... 2005-01-01...
2 Test1 2005-01-02... 2005-01-02...
3 Test1 2005-01-03... 2005-01-03...
...
I would be very grateful if somebody could help resolve this mystery.
Thanks in advance.
The one thing that should help is a covering index on calendar_id:
create index <indexname>
on table (calendar_id, id)
include (start_datetime, end_datetime);
This will satisfy the calendar_id = 'Test1' predicates, the min(row_id) sort and will provide the material to evaluate the non-SARG-able DATEFIFF predicates. If there are no other columns in the table, then this is probably the clustered index you need and the id primary key should be a non-clustered one.
Make sure the indexes made the conversion. Then update statistics.
Check the differences between the execution plan on the old sql server and the new one. http://www.sql-server-performance.com/tips/query_execution_plan_analysis_p1.aspx
About the other only thing you can do beyond Remus Rusanu's index suggestion, is to upgrade to the Enterprise edition which has a more advanced scan feature (on both SQL Server 2005 and 2008 Enterprise Edition) which allows multiple tasks to share full table scans.
Beyond that, I do not think there is anything you can do if you cannot change the query. The reason is that the query is doing a comparison against a function result in the Where clause. That means it will force SQL Server to do a table scan on Table1 each time it is executed.
Reading Pages (more info about Advanced Scanning)
I have a LINQ to SQL query that generates the following SQL :
exec sp_executesql N'SELECT COUNT(*) AS [value]
FROM [dbo].[SessionVisit] AS [t0]
WHERE ([t0].[VisitedStore] = #p0) AND (NOT ([t0].[Bot] = 1)) AND
([t0].[SessionDate] > #p1)',N'#p0 int,#p1 datetime',
#p0=1,#p1='2010-02-15 01:24:00'
(This is the actual SQL taken from SQL Profiler on SQL Server 2008.)
The query plan generated when I run this SQL from within Query Analyser is perfect.
It uses an index containing VisitedStore, Bot, SessionDate.
The query returns instantly.
However when I run this from C# (with LINQ) a different query plan is used that is so inefficient it doesn't even return in 60 seconds. This query plan is trying to do a key lookup on the clustered primary key which contains a couple million rows. It has no chance of returning.
What I just can't understand though is that the EXACT same SQL is being run - either from within LINQ or from within Query Analyser yet the query plan is different.
I've ran the two queries many many times and they're now running in isolation from any other queries. The date is DateTime.Now.AddDays(-7), but I've even hardcoded that date to eliminate caching problems.
Is there anything i can change in LINQ to SQL to affect the query plan or try to debug this further? I'm very very confused!
This is a relatively common problem that surprised me too when I first saw it. The first thing to do is ensure your statistics are up to date. You can check the age of statistics with:
SELECT
object_name = Object_Name(ind.object_id),
IndexName = ind.name,
StatisticsDate = STATS_DATE(ind.object_id, ind.index_id)
FROM SYS.INDEXES ind
order by STATS_DATE(ind.object_id, ind.index_id) desc
Statistics should be updated in a weekly maintenance plan. For a quick fix, issue the following command to update all statistics in your database:
exec sp_updatestats
Apart from the statistics, another thing you can check is the SET options. They can be different between Query Analyzer and your Linq2Sql application.
Another possibility is that SQL Server is using an old cached plan for your Linq2Sql query. Plans can be cached on a per-user basis, so if you run Query Analyser as a different user, that can explain different plans. Normally you could add Option (RECOMPILE) to the application query, but I guess that's hard with Linq2Sql. You can clear the entire cache with DBCC FREEPROCCACHE and see if that speeds up the Linq2Sql query.
switched to a stored procedure and the same SQL works fine. would really like to know what's going on but can't spend any more time on this now. fortunately in this instance the query was not too dynamic.
hopefully this at least helps anyone in the same boat as me
Do all SQL server versions rebuild indexes automatically or have a default rebuild criteria? I understand statistics are rebuilt automatically but not sure if indexes do as well.
Rebuilding of indexes is not supported automatically in any version of Microsoft SQL Server - the reason being is that rebuilding indexes can potentially be very expensive operations, and so need careful scheduling and planning.
In many environments special scripts will be written to handle this, for example:
http://weblogs.sqlteam.com/tarad/archive/2008/09/03/Defragmenting-Indexes-in-SQL-Server-2005.aspx
Note that whilst SQL can automatically update statistics for you in many cases there is a performance gain to be had by managing these more carefully as well.
As people have mentioned here, your indexes do not automatically rebuild. This is quite a big problem in SQL Server, as your indexes will fragment over time. Your could find your indexes are 95% plus fragmented, affecting query performance badly.
Here is a simple query to check fragmentation on your existing indexes:
DECLARE #DBNAME VARCHAR(130);
SET #DBNAME = 'MYDBNAME';
DECLARE #DBID INT;
SET #DBID = DB_ID(#DBNAME);
SELECT
OBJECT_ID AS objectID
, index_id AS indexID
, avg_fragmentation_in_percent AS fragmentation
, page_count
INTO #indexDefragList
FROM
sys.dm_db_index_physical_stats
(#DBID, NULL, NULL , NULL, N'Limited')
WHERE
index_id > 0
OPTION (MaxDop 1);
SELECT
i.[name] as indexname,
d.fragmentation,
d.page_count
FROM
#indexDefragList d
INNER JOIN sys.indexes i
ON d.objectid = i.object_id
ORDER BY
d.fragmentation DESC
DROP TABLE #indexDefragList
This will return a list of all indexes in your current DB with their fragmentation in %.
You can easily build a script to automatically rebuild or reorganise them. There is a great article from SQLFool on this including a complete pre-made script.
To expand on what Chris has said:
With regard to statistics, columns not covered by an index will not have their statistics updated by rebuilding all indexes. They may periodically be updated by SQL Server however you may need to do this yourself with the UPDATE STATISTICS statement.
SQL Server 2005 determines whether to update statistics automatically based on changes to column modification counters (colmodctrs).
A statistics object is considered out of date in the following cases:
1.The table size has gone from 0 to >0 rows.
2.The number of rows in the table when the statistics were gathered was 500 or less, and the colmodctr of the leading column of the statistics object has changed by more than 500 since then.
3.The table had more than 500 rows when the statistics were gathered, and the colmodctr of the leading column of the statistics object has changed by more than 500 + 20% of the number of rows in the table when the statistics were gathered
You may find the following reference regarding statistics of use:
http://blogs.technet.com/rob/archive/2008/05/16/sql-server-statistics.aspx
Hope this helps but feel free to pose further queries.
Cheers, John
As #Chris noted, indexes are not rebuilt automatically in any SQL Server version. Proper index maintenance is often missing at sites with no dedicated DBA role, and often when databases are simply moved from development into production (along with Transaction Log maintenance).
SQL Server 2005+ has the ability to do online index reorganises and offline full rebuilds.
It has to be schedule and arranged yourself.
When sorting it out, it depends on the size of your tables and maintenance window.
Also, statistics are rebuilt automatically when the indexes are rebuilt but can be scheduled separately.
For a quick fix, assuming a nice long maintenance window and not too big (upto a few 100 GBs), just schedule this:
EXEC sp_msforeachtable 'SET QUOTED_IDENTIFIER ON ALTER INDEX ALL ON ? REBUILD WITH (FILLFACTOR = 90)'
Edit: only for > SQL 2005 this SQL