At first I thought I should go see my Dr. and have an MRI of my brain. But, the results I am seeing were confirmed by a second programmer. Running the same exact SQL is giving me different results at different times! It makes no sense. Here is what I am running (table names changed to protect the innocent)...
declare #customerid int;
declare #locationid int;
declare #reportdate datetime;
set #customerid=2063;
set #locationid=101;
set #reportdate=getdate();
select i.InvoiceDate as [Date], ... from Invoice i (nolock) ...
UNION ALL select ... from Remit e (nolock) ...
UNION ALL select ... from [Rec] r (nolock) join TrTypes t (nolock) on ...
The two select/union all's are identical but sometimes the second does not return records from the Remit table when the first shows them. And... Sometimes the first doesn't pull them either. I put these queries in SQL Server Management Studio and keep hitting F5... Sometime I get proper results, sometimes I don't.
Further, when I remove the UNION ALLs and just run the queries, nothing is returned from Remit.
Could this be a locking issue?
This query is from a C# program that produces a detail report and it never returns the Remit records. There is another program that creates summary records from the Remit table and it always returns the records from a similar query... That is why I had to look at the detail program/report.
Thoughts?
NOLOCK means you don't care about consistency, concurrency and locking/blocking. It allows you to read the same row 0 times, 1 time, or 2 times. It is not a magic "go faster" button for queries: you should only use it if you fully understand the potential consequences on a busy system (such as getting different results at different times!).
The solution is to either:
stop blindly throwing NOLOCK everywhere (accept your blocking to get accurate and consistent data, or use read committed snapshot isolation).
accept that NOLOCK sacrifices accuracy for less blocking, and sometimes it just won't give the right results. If you ask for this, you don't get to complain about it.
Related
Yesterday I had this strange (never seen before) situation when applying a patch to several databases from with SQL server management studio. I needed to update a large table (5 Million records+) over several databases. The statement I issued per database was something like:
set rowcount 100000
select 1
while ##rowcount > 1
update table
set column = newval
where column is null
Those statements ran for several seconds (anywhere between 2 and 30 seconds). On a second tab I ran this query:
select count(*)
from table
where column is null
Just to check how many rows where already processed. Impatient as I am, I pressed F5 for the count(*) statements every 20 seconds or so. And the - expected behavior - was that I had to wait for anything in the range from 0 to 30 seconds before the count(*) was calculated. I expected this behavior, since the update statement was running so the count(*) was next in line.
But then after a database or 10 this strange thing happened. I opened the next database (altered the database via USE) and after pressing F5 the count(*) responded immediately. And after pressing F5 again, direct result but without any progress, and again and again, the count(*) did not change. However the update was running.
And then after then n-th press, the count(*) dropped with exactly 100K records. I pressed F5 again, immediate result.. and so on and so on... WHY? wasn't the count(*) waiting... I did not allowed to read dirty or....
Only this database gave me this behavior. And I have really no clue what is causing this.
Between switching database I did not close or open any other tab, so the only thing I can imagine is that the connection parameters for that one database are different. However I'm overlooking those... but I can't find a difference.
as per what I understood from your description above is that you were not getting the result of count() correctly on some specific database. If that is the case then, Please use the below script Instead of your count()
SELECT
COUNT(1)
FROM Table WITH(NOLOCK)
WHERE Column IS NULL
Because count(*) takes more time to execute. and as your requirement is to just know the count Use Count(1) instead
Also there is a possibility of lock in the database if you are using transactions. So Use the WITH(NOLOCK) to avoid such blocking and get the result more faster.
There's no shortage of topics covering this function and error "An insufficient number of arguments were supplied for the procedure or function cdc.fn_cdc_get_all_changes". I've checked most of them, but can't figure out what's wrong.
The problem here specifically is that I can't even reproduce this. It just appears randomly a few times per day or two, and goes away in mere seconds, so usually I can only see it in the job history as a failed run.
All sources I've browsed say it typically comes from using the function to fetch data from a time range where the capture instance doesn't have data. But as the code below will show you, the values are checked for just these type of exceptions before running the function.
DECLARE #END_LSN BINARY(10), #MIN_TABLE_LSN BINARY(10);
SELECT #END_LSN = sys.fn_cdc_get_max_lsn();
SELECT #MIN_TABLE_LSN = MAX(__$start_lsn) FROM MY_AUDIT_TABLE
IF #MIN_TABLE_LSN IS NOT NULL
SELECT #MIN_TABLE_LSN = sys.fn_cdc_increment_lsn(#MIN_TABLE_LSN)
ELSE
SELECT #MIN_TABLE_LSN = sys.fn_cdc_get_min_lsn('dbo_MY_AUDIT_TABLE')
IF #MIN_TABLE_LSN IS NOT NULL
BEGIN
INSERT INTO MY_AUDIT_TABLE (...columns...)
SELECT ... columns...
FROM cdc.fn_cdc_get_all_changes_dbo_MY_SOURCE_TABLE(#MIN_TABLE_LSN, #END_LSN, 'all update old') C
JOIN cdc.lsn_time_mapping T WITH (NOLOCK) ON T.start_lsn = C.__$start_lsn
ORDER BY __$start_lsn ASC, __$seqval ASC
END
Now the only remaining alternatives which some people have even suggested, is that this code may sometimes pick the latest change from the AUDIT table and then increment that to an LSN that doesn't yet exist. But I've tested this manually tons of times, and it gives no errors. Also, using an #END_LSN value derived from another CDC table, where this particular MY_AUDIT_TABLE doesn't have records that far yet, works perfectly as well.
The only way I can produce this error manually, is by giving the function a newer #END_LSN value than what exist in lsn_time_mapping table. But such a scenario is only possible if SQL Server can actually create CDC-table records with start_lsn's that don't yet exist in lsn_time_mapping, and I hardly think that's possible. Or is it? That would mean that you can't reliably map an lsn to a datetime at the time the row has just become available.
Thanks for help and explanations again, as usual. :)
This
SELECT #MIN_TABLE_LSN = sys.fn_cdc_increment_lsn(#MIN_TABLE_LSN)
just adds 1 to #MIN_TABLE_LSN. It's not necessary that new #Min_Table_Lsn exists. So if you don't have any changes, #END_LSN may be lower than #Min_Table_Lsn and cdc.fn_cdc_get_all_changes_dbo_MY_SOURCE_TABLE() function may fail.
Note that "CDC Scanner job" adds a dummy LSN (with transaction id 0x0) every 5 min to the cdc.lsn_time_mapping table, that's why you may have scenarios when there are no changes and the script may succeed or not, depending on when you run it.
As mentioned above, you should change the condition that Min LSN is lower or equal to End LSN
Simple - if the BeginLsn/FromLsn that you are trying to supply to get_all_changes function, is not available in the change table, this error is thrown. By default set it to minLsn (get_min_lsn) and pass it to function. With the retention period in effect, after every x number of days, the change table data get's cleaned, so you should make sure you are setting FromLsn to an available value in CT table.
i was able to fix this by using the "full name"
Eg. table is dbo.lookup_test you need to pass dbo_lookup_test in sys.fn_cdc_get_min_lsn.
DECLARE #from_lsn binary(10), #to_lsn binary(10);
SET #from_lsn = sys.fn_cdc_get_min_lsn('dbo_lookup_test');
SET #to_lsn = sys.fn_cdc_get_max_lsn();
SELECT * FROM cdc.fn_cdc_get_all_changes_dbo_lookup_test
(#from_lsn, #to_lsn, N'all');
GO
Obviously the code above already took my answer as consideration, but the table naming requirement "{schema}_{tablename}" was not apparent and SQL Server was not helping by giving a misleading error message.
Please also note the lsn should be declared as binary(10). It can't be simply declared as binary. It will throw the same error.
I have a table MyTable with a field MyField, which is varbinary(max). I have the following query:
SELECT COUNT(*) FROM MyTable WHERE MyField IS NOT NULL
The SELECT clause can have anything, it does not matter. Because of the varbinary MyField in the Where clause, the execution never ends.
I tried even this:
SELECT Size
FROM
(
SELECT ISNULL(DATALENGTH(MyField), 0) AS Size FROM MyTable
) AS A
WHERE A.Size > 0
The inner query works fine, the whole query without the Where clause works fine, but with that Where clause it is stuck. Could anybody please explain this to me?
Thanks.
Don't think or assume that it couldn't possibly be blocking, just because a different query returns immediately. With a different where clause, and a different plan, and possibly different locking escalation, you could certainly have cases where one query is blocked and another isn't, even against the same table.
The query is obviously being blocked, if your definition of "stuck" is what I think it is. Now you just need to determine by who.
In one query window, do this:
SELECT ##SPID;
Make note of that number. Now in that same query window, run your query that gets "stuck" (in other words, don't select a spid in one window and expect it to have anything to do with your query that is already running in another window).
Then, in a different query window, do this:
SELECT blocking_session_id, status, command, wait_type, last_wait_type
FROM sys.dm_exec_requests
WHERE session_id = <spid from above>;
Here is a visualization that I suspect might help (note that my "stuck" query is different from yours):
If you get a non-zero number in the first column, then in that different query window, do this:
DBCC INPUTBUFFER(<that blocking session id>);
If you aren't blocked, I'd be curious to know what the other columns show.
As an aside, changing the WHERE clause to use slightly different predicates to identify the same rows isn't going to magically eliminate blocking. Also, there is no real benefit to doing this:
SELECT Size
FROM
(
SELECT ISNULL(DATALENGTH(MyField), 0) AS Size FROM MyTable
) AS A
WHERE A.Size > 0
When you can just do this:
SELECT ISNULL(DATALENGTH(MyField), 0) AS Size
FROM dbo.MyTable -- always use schema prefix
WHERE DATALENGTH(MyField) > 0; -- always use semi-colon
SQL Server is smart enough to not calculate the DATALENGTH twice, if that is your concern.
Just for fun and giggles I decided to toss this one out to you. I'm taking some things for granted here but you can look at tasks that had to wait. Look for wait types that begin with LCK_ as these will be your blocked tasks. Please note that on a busy server the ring buffer may have rolled over after a while. Also note that this is intended to supplement #AaronBertran's excellent answer and in no way meant to supplant it. His is more comprehensive and is the correct way to identify the issue while it's happening.
SELECT
td.r.value('#name','sysname') event_name,
td.r.value('#timestamp','datetime2(0)') event_timestamp,
td.r.value('(data[#name="wait_type"]/text)[1]','sysname') wait_type,
td.r.value('(data[#name="duration"]/value)[1]','bigint') wait_duration,
td.r.value('(action[#name="sql_text"]/value)[1]','nvarchar(max)') sql_text,
td.r.query('.') event_data
FROM (
SELECT
CAST(target_data AS XML) target_data
FROM sys.dm_xe_sessions s
JOIN sys.dm_xe_session_targets t
ON s.address = t.event_session_address
WHERE s.name = N'system_health'
and target_name = 'ring_buffer'
) base
CROSS APPLY target_data.nodes('/RingBufferTarget/*') td(r)
where td.r.value('#name','sysname') = 'wait_info';
Everywhere I look I see that in order to loop through results you have to use a cursor and in the same post someone saying cursors are bad don't use them (which has always been my philosophy) but now I am stuck. I need to loop through a result set!
Here's the situation. I need to come up with a list of ProductIDs that have 2 different statuses set to a specific value. I start the stored procedure, run the query that finds my products that meet the criteria.
So, now I have a list of ProductIDs that I need to run through my validation process:
16050
16052
41817
48255
Now I need for each of those products (there may be 1 there may be 1000, i don't know) to check a whole list of conditions:
Is a specific field = 'SIMPLE'? if so, perform a bunch of other queries and make sure everything is good
If it is not 'SIMPLE' then run a whole other set of queries and make sure that information is all good.
Is another field = 'YES'? if so, perform a bunch of other queries, if it is not, then do other queries.
Is a cursor what I need to use? Is there some other way to do what I need that I just am not seeing?
Thanks,
Leslie
I ended up using a WHILE loop that I can pass each ProductID into a series of checks!!
declare #counter int
declare #productKey varchar(20)
SET #counter = (select COUNT(*) from ##Magento)
while (1=1)
begin
SET #productKey = (select top 1 ProductKey from ##Magento)
print #productKey;
delete from ##Magento Where ProductKey = #productKey
SET #counter-=1;
IF (#counter=0) BREAK;
end
go
It's hard to say without knowing the specifics of your process, but one approach is to create a function that performs your logic and call that.
eg:
delete from yourtable
where productid in (select ProductID from FilteredProducts)
and dbo.ShouldBeDeletedFunction(ProductID) = 1
In general, cursors are bad, but there are always exceptions. Try to avoid them by thinking in terms of sets, rather than the attributes of an individual record.
The question from quite a long time boiling in my head, that out of the following two stored procedures which one would perform better.
Proc 1
CREATE PROCEDURE GetEmployeeDetails #EmployeeId uniqueidentifier,
#IncludeDepartmentInfo bit
AS
BEGIN
SELECT * FROM Employees
WHERE Employees.EmployeeId = #EmployeeId
IF (#IncludeDepartmentInfo = 1)
BEGIN
SELECT Departments.* FROM Departments, Employees
WHERE Departments.DepartmentId = Employees.DepartmentId
AND Employees.EmployeeId = #EmployeeId
END
END
Proc 2
CREATE PROCEDURE GetEmployeeDetails #EmployeeId uniqueidentifier,
#IncludeDepartmentInfo bit
AS
BEGIN
SELECT * FROM Employees
WHERE Employees.EmployeeId = #EmployeeId
SELECT Departments.* FROM Departments, Employees
WHERE Departments.DepartmentId = Employees.DepartmentId
AND Employees.EmployeeId = #EmployeeId
AND #IncludeDepartmentInfo = 1
END
the only difference between the two is use of 'if statment'.
if proc 1/proc 2 are called with alternating values of #IncludeDepartmentInfo then from my understanding proc 2 would perform better, because it will retain the same query plan irrespective of the value of #IncludeDepartmentInfo, whereas proc1 will change query plan in each call
answers are really appericated
PS: this is just a scenario, please don't go to the explicit query results but the essence of example. I am really particular about the query optimizer result (in both cases of 'if and where' and their difference), there are many aspects which I know could affect the performance which I want to avoid in this question.
SELECT Departments.* FROM Departments, Employees
WHERE Departments.DepartmentId = Employees.DepartmentId
AND Employees.EmployeeId = #EmployeeId
AND #IncludeDepartmentInfo = 1
When SQL compiles a query like this it must be compiled for any value of #IncludeDepartmentInfo. The resulted plan can well be one that scans the tables and performs the join and after that checks the variable, resulting in unnecessary I/O. The optimizer may be smart and move the check for the variable ahead of the actual I/O operations in the execution plan, but this is never guaranteed. This is why I always recommend to use explicit IFs in the T-SQL for queries that need to perform very differently based on a variable value (the typical example being OR conditions).
gbn's observation is also an important one: from an API design point of view is better to have a consistent return type (ie. always return the same shaped and number of result sets).
From a consistency perspective, number 2 will always return 2 datasets. Overloading aside, you wouldn't have a client code method that may be returns a result, maybe not.
If you reuse this code, the other calling client will have to know this flag too.
If the code does 2 different things, then why not 2 different stored procs?
Finally, it's far better practice to use modern JOIN syntax and separate joining from filtering. In this case, personally I'd use EXISTS too.
SELECT
D.*
FROM
Departments D
JOIN
Employees E ON D.DepartmentId = E.DepartmentId
WHERE
E.EmployeeId = #EmployeeId
AND
#IncludeDepartmentInfo = 1
When you use the 'if' statement, you may run only one query instead of two. I would think that one query would almost always be faster than two. Your point about query plans may be valid if the first query were complex and took a long time to run, and the second was trivial. However, the first query looks like it retrieves a single row based on a primary key - probably pretty fast every time. So, I would keep the 'if' - but I would test to verify.
The performance difference would be too small for anyone to notice.
Premature optimization is the root of all evil. Stop worrying about performance and start implementing features that make your customers smile.