SqlCommand.ExecuteReader became very slow - sql-server

I am using this code, to call an SQL function, which returns entries from an SQL Server database table
string cmd = String.Format("select * from dbo.GetData(#userId, #fileId, #created);");
using (SqlConnection conn = new SqlConnection(connectionString))
{
if (conn.State != ConnectionState.Open)
conn.Open();
SqlCommand command = new SqlCommand(cmd, conn);
if (String.IsNullOrEmpty(userId))
command.Parameters.AddWithValue("#userId", DBNull.Value);
else
command.Parameters.AddWithValue("#userId", userId);
if (String.IsNullOrEmpty(fileId))
command.Parameters.AddWithValue("#fileId", DBNull.Value);
else
command.Parameters.AddWithValue("#fileId", docId);
command.Parameters.AddWithValue("#created", created);
internalWatch.Reset();
internalWatch.Start();
IDataReader reader = command.ExecuteReader();
table = GetDataTableFromDataReader(reader);
reader.Close();
reader.Dispose();
conn.Close();
internalWatch.Stop();
The table I am working with contains a bit more than 1.5 million entries and should return a bit more than 250k entries.
If I am executing the SQL function within SSMS it needs 8 seconds to return the results and I already used the code above last week to get the results within my desktop application. At this time everything was fine. The code needed between 10-12 seconds to get the results.
The strange thing is, that today the code needs more than 40 seconds to return the same results, but I haven't change anything within the SQL function or the code itself. The only change I did in my programm is adding a few more classes, which have nothing to do with the code above.
If I am debugging the code, I can see, that the line
IDataReader reader = command.ExecuteReader();
needs the most of the time now.
Since I haven't change anything in the SQL function or the code itself, I can't understand why it is taking so long now...
And if needed, here is the SQL function, I am using:
ALTER FUNCTION [dbo].[GetData]
(#userId varchar(128) = NULL,
#fileId varchar(192) = NULL,
#created DateTimeOffset(7))
RETURNS TABLE
AS
RETURN (
WITH FindNewestVersion AS
(
SELECT
*,
ROW_NUMBER() OVER (PARTITINO BY FileId, UserId
ORDER BY created DESC) rn
FROM
table1
)
SELECT
q.Created, q.Updated, q.FileId, q.UserId,
F.column1, F.column2, F.column3
FROM
table2 AS F
INNER JOIN
table1 AS q ON F.column4 = q.PersonId AND F.created = q.created
INNER JOIN
(SELECT
created, PersonId, DocumentId
FROM
FindNewestVersion
WHERE
rn = 1) AS x ON q.created = x.created
AND q.PersonId = x.PersonId
AND q.FileId = x.FileId
WHERE
(F.column1 = 'Sample')
AND (q.Created <= #created)
AND (q.Updated >= #created)
AND Q.PersonId = ISNULL(#userId, Q.PersonId)
AND Q.FileId = ISNULL(#fileId, Q.FileId)
)
Thank you for any suggestions!

This seems like a case of parameter sniffing
One thing that you could do is rewrite your procedure as follows:
`
DROP FUNCTION [dbo].[GetData]
CREATE PROCEDURE[dbo].[GetData]
(
#userId varchar(128) = NULL,
#fileId varchar(192) = NULL,
#created DateTimeOffset(7)
)
RETURNS TABLE
AS
DECLARE #l_userId varchar(128) = NULL,
DECLARE #l_fileId varchar(192) = NULL,
DECLARE #l_created DateTimeOffset(7)
SET #l_userId = userId
SET #l_fileId = fileId
SET #l_created = #created
(
WITH FindNewestVersion as
(
Select *, ROW_NUMBER()
over (partition by FileId, UserId ORDER BY created DESC)rn from table1
)
SELECT q.Created, q.Updated, q.FileId, q.UserId,
F.column1, F.column2, F.column3
FROM table2 AS F INNER JOIN
table1 AS q
ON F.column4 = q.PersonId AND F.created = q.created
INNER JOIN
(
select created, PersonId, DocumentId from FindNewestVersion where rn = 1
) AS x ON q.created = x.created AND q.PersonId = x.PersonId AND q.FileId = x.FileId
WHERE (F.column1 = 'Sample') AND (q.Created <= #created) AND (q.Updated >= #created)
And Q.PersonId = ISNULL(#l_userId, Q.PersonId)
And Q.FileId = ISNULL(#l_fileId, Q.FileId)
)
`
You can then get you data as you would by calling a stored procedure.

This seems like a case of SET ARITHABORT ON
By default in SQL it's On But while we execute Through C# Code then its not understand the ARITHABORT ON.
One thing that you could do is rewrite your procedure as follows:
DROP FUNCTION [dbo].[GetData]
CREATE PROCEDURE[dbo].[GetData]
(
#userId varchar(128) = NULL,
#fileId varchar(192) = NULL,
#created DateTimeOffset(7)
)
RETURNS TABLE
AS
SET ARITHABORT ON
DECLARE #l_userId varchar(128) = NULL,
DECLARE #l_fileId varchar(192) = NULL,
DECLARE #l_created DateTimeOffset(7)
SET #l_userId = userId
SET #l_fileId = fileId
SET #l_created = #created
(
WITH FindNewestVersion as
(
Select *, ROW_NUMBER()
over (partition by FileId, UserId ORDER BY created DESC)rn from table1
)
SELECT q.Created, q.Updated, q.FileId, q.UserId,
F.column1, F.column2, F.column3
FROM table2 AS F INNER JOIN
table1 AS q
ON F.column4 = q.PersonId AND F.created = q.created
INNER JOIN
(
select created, PersonId, DocumentId from FindNewestVersion where rn = 1
) AS x ON q.created = x.created AND q.PersonId = x.PersonId AND q.FileId = x.FileId
WHERE (F.column1 = 'Sample') AND (q.Created <= #created) AND (q.Updated >= #created)
And Q.PersonId = ISNULL(#l_userId, Q.PersonId)
And Q.FileId = ISNULL(#l_fileId, Q.FileId)
)`enter code here`

Related

Stored procedure always returns same query

I tried to write a SQL query to get results with some parameters, but when I send parameters, it doesn't work. It always returns the last query in if else condition statement.
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
CREATE PROCEDURE [dbo].[sp_RealEstatesList]
#TransCode NVARCHAR(5),
#CatUrl NVARCHAR(255) = NULL,
#ForSaleItem BIT = NULL,
#NewItem BIT = NULL,
#ItemOfDay BIT = NULL,
#Random BIT = NULL,
#Top INT = NULL
AS
SET NOCOUNT ON
SET XACT_ABORT ON
SET FMTONLY OFF
BEGIN TRAN
IF (OBJECT_ID('tempdb..#TempTable') IS NOT NULL)
BEGIN
DROP TABLE #TempTable
END
SELECT
R.[ID],
RT.[Title],
R.[Price],
R.[NewItem],
R.[ForSaleItem],
R.[ItemOfDay],
R.[Url],
(SELECT TOP 1 CC.Url
FROM [dbo].[Category] CC
JOIN [dbo].[LinkTypes] LLT ON LLT.[MainID] = CC.ID
JOIN [dbo].[Links] LL ON LL.[LinkTypeID] = LLT.[ID]
JOIN [dbo].[RealEstates] RR ON RR.[ID] = LL.LinkID
WHERE LLT.[LinkedTypeID] = 17
AND LLT.[MainTypeID] = 1
AND RR.ID = R.[ID]) AS 'CatUrl',
(SELECT TOP 1
(SELECT TOP 1 XC.Url
FROM Category XC
JOIN [dbo].[LinkTypes] LLT ON LLT.[MainID] = XC.ID
JOIN [dbo].[Links] LL ON LL.[LinkTypeID] = LLT.[ID]
WHERE LLT.[LinkedTypeID] = 1
AND LLT.[MainTypeID] = 1
AND LL.LinkID = CC.ID)
FROM [dbo].[Category] CC
JOIN [dbo].[LinkTypes] LLT ON LLT.[MainID] = CC.ID
JOIN [dbo].[Links] LL ON LL.[LinkTypeID] = LLT.[ID]
JOIN [dbo].[RealEstates] RR ON RR.[ID] = LL.LinkID
WHERE LLT.[LinkedTypeID] = 17
AND LLT.[MainTypeID] = 1
AND RR.ID = R.[ID]) AS 'PCatUrl'
INTO
#TempTable
FROM
[dbo].[RealEstates] R
JOIN
[dbo].[RealEstatesT] RT ON R.[ID] = RT.[RealEsID]
JOIN
[dbo].[LinkTypes] LT ON LT.[MainTypeID] = 17
JOIN
[dbo].[Links] L ON L.[LinkTypeID] = LT.[ID]
JOIN
[dbo].[Translation] TR ON TR.[ID] = RT.[TransID]
WHERE
R.[Active] = 1
AND R.[Deleted] = 0
AND RT.[Deleted] = 0
AND LT.[LinkedTypeID] = 5
AND LT.[MainID] = R.[ID]
AND TR.[ShortName] = #TransCode
GROUP BY
R.[ID], RT.[Title], R.[Price], R.[NewItem], R.[ForSaleItem], R.[ItemOfDay], R.[Url]
if (#Top = null or #Top = 0)
begin
set #Top = 100000;
end
if(#NewItem != null)
begin
select TOP (#Top) [ID],[Title],[Price],[NewItem],[ForSaleItem],[CatUrl],[PCatUrl],[ItemOfDay],[Url]
from #TempTable
where NewItem = #NewItem
order by case when #Random = 1 then NEWID() end
end
else if(#ForSaleItem != null)
begin
select TOP (#Top) [ID],[Title],[Price],[NewItem],[ForSaleItem],[CatUrl],[PCatUrl],[ItemOfDay],[Url]
from #TempTable
where ForSaleItem = #ForSaleItem
order by case when #Random = 1 then NEWID() end
end
else if(#CatUrl != null)
begin
select TOP (#Top) [ID],[Title],[Price],[NewItem],[ForSaleItem],[CatUrl],[PCatUrl],[ItemOfDay],[Url]
from #TempTable
where CatUrl = #CatUrl or PCatUrl = #CatUrl
order by case when #Random = 1 then NEWID() end
end
else if(#ItemOfDay != null)
begin
select TOP (#Top) [ID],[Title],[Price],[NewItem],[ForSaleItem],[CatUrl],[PCatUrl],[ItemOfDay],[Url]
from #TempTable
where ItemOfDay = #ItemOfDay
order by case when #Random = 1 then NEWID() end
end
else
begin
select TOP (#Top) [ID],[Title],[Price],[NewItem],[ForSaleItem],[CatUrl],[PCatUrl],[ItemOfDay],[Url]
from #TempTable
order by case when #Random = 1 then NEWID() end
end
COMMIT
Transcode and Random parameters work well but CatUrl, ForSaleItem, NewItem, ItemOfDay parameters does not work. It always returns the last query in if else condition statement.
Hope you can help. I'm waiting for your answers.
edit :
I changed all my null checks like "NULLIF(#Top, '') IS NULL" and now if statements works well.

Optimize update and insert query

I have this code:
string query = #"UPDATE [dbo].[TrippingTariffTransaction]
SET [IsActive] = 0
WHERE Id in (SELECT Id
FROM [dbo].[TrippingTariffTransaction]
WHERE Trip = #Trip
AND TrippingTariffId = #TrippingTariffId);";
query += #"INSERT INTO [dbo].[TrippingTariffTransactionAuditTrail]
(LogDatetime, MasterlistId, ComputerName, TrippingTariffTransactionID, Activity)
SELECT
GETDATE(), #MasterlistId, #ComputerName, Id, #Activity
FROM
[dbo].[TrippingTariffTransaction]
WHERE
Trip = #Trip AND TrippingTariffId = #TrippingTariffId";
How can I optimize this code. I have redundant select. Thanks in advance :)
You can make use of the OUTPUT clause
update t
set IsActive = 0
OUTPUT getdate(), #MasterlistId, #ComputerName, INSERTED.Id, #Activity
INTO TrippingTariffTransactionAuditTrail
( LogDatetime, MasterlistId, ComputerName, TrippingTariffTransactionID, Activity)
FROM TrippingTariffTransaction t
WHERE t.Trip = #Trip
AND t.TrippingTariffId = #TrippingTariffId
You can use temp table:
SELECT Id INTO #T
FROM [dbo].[TrippingTariffTransaction]
WHERE Trip = #Trip
AND TrippingTariffId = #TrippingTariffId
UPDATE [dbo].[TrippingTariffTransaction]
SET [IsActive] = 0
WHERE Id in (SELECT Id FROM #T)
INSERT INTO [dbo].[TrippingTariffTransactionAuditTrail]
(LogDatetime, MasterlistId, ComputerName, TrippingTariffTransactionID, Activity)
SELECT
GETDATE(), #MasterlistId, #ComputerName, Id, #Activity
FROM #T
DROP TABLE #T

Data not displayed in SSRS report, but execting query in SQL shows data

I want to show some data in a report by using a stored procedure:
ALTER PROCEDURE [dbo].[usp_GetThirtyDaysSims]
(
#DateFrom DATETIME = NULL,
#DateTo DATETIME = NULL,
#O2QuarterStartDate INT = NULL,
#AreaId INT = NULL,
#StoreId INT = NULL,
#O2MonthStartDate INT = NULL,
#TransactionType NVARCHAR(1000)
)
AS
BEGIN
DECLARE #O2Quarter AS INT
DECLARE #O2Year AS INT
DECLARE #O2Month AS INT
IF (#AreaId = 0 OR #AreaId = 999999999)
BEGIN
SET #AreaId = NULL
END
IF (#StoreId = 0 OR #StoreId = 999999999)
BEGIN
SET #StoreId = NULL
END
IF (#O2QuarterStartDate = 0 OR #O2QuarterStartDate = 999999999)
BEGIN
SET #O2QuarterStartDate = NULL
END
IF (#O2MonthStartDate = 0 OR #O2MonthStartDate = 999999999)
BEGIN
SET #O2MonthStartDate = NULL
END
IF(#O2QuarterStartDate IS NOT NULL)
BEGIN
SELECT #O2Quarter = O2Quarter,#O2Year = O2Year
FROM DimDate
WHERE DateKey=#O2QuarterStartDate
SELECT #DateFrom = MIN([Date]),#DateTo = MAX([Date])
FROM DimDate
WHERE O2Quarter=#O2Quarter AND O2Year=#O2Year
END
IF(#O2MonthStartDate IS NOT NULL)
BEGIN
PRINT #O2MonthStartDate
SELECT #O2Month=O2Month, #O2Quarter = O2Quarter,#O2Year = O2Year
FROM DimDate
WHERE DateKey=#O2MonthStartDate
SELECT #DateFrom = MIN([Date]),#DateTo = MAX([Date])
FROM DimDate
WHERE O2Month=#O2Month AND O2Quarter=#O2Quarter AND O2Year=#O2Year
END
SELECT Area,StoreName,SUM(Tariff) SumOfTariff, COUNT(1) NoOfSimsSold
FROM [dbo].[ufn_GetThirtyDaysSimsData](#DateFrom,#DateTo) FT
INNER JOIN DimDate DD ON FT.TransactionDateId = DD.DateKey
WHERE (#DateFrom IS NOT NULL AND DD.[Date] >= #DateFrom)
AND (#DateTo IS NOT NULL AND DD.[Date] <= #DateTo)
AND (FT.AreaId = #AreaId OR #AreaId IS NULL)
AND (FT.StoreId = #StoreId OR #StoreId IS NULL)
AND (FT.TransactionType = #TransactionType OR #TransactionType IS NULL)
GROUP BY Area,StoreName
ORDER BY Area,StoreName
END
When I filter by using the 7 filters above, data is not showing on the report but there are no errors either - why might this be?
Make sure your function is table-based. You don't need to add the date parameters in the WHERE clause because you are already passing those in the function. Test your query by running it first in SSMS.
Here is a simplified version of the query:
SELECT Area, StoreName, SUM(Tariff) SumOfTariff, COUNT(1) NoOfSimsSold
FROM [dbo].[ufn_GetThirtyDaysSimsData](#DateFrom,#DateTo) FT
INNER JOIN DimDate DD ON FT.TransactionDateId = DD.DateKey
WHERE FT.AreaId = #AreaId
AND FT.StoreId = #StoreId
AND FT.TransactionType = #TransactionType
GROUP BY Area, StoreName
ORDER BY Area, StoreName

Select multiple columns but group by only one

Using SQL Server 2008. I have been researching this problem for days. Thought about CTE and trying to use a predicate on the problem tables.
Nothing I have tried or researched has worked so here I am. The problem is it's returning duplicate OrderID.
I've marked the problem joins.
Even tried OUTER APPLY but it caused certain searches not to work. Tried INNER APPLY then duplicates again.
The problem joins, the tables, have multiple references to OrderID.
So tblRun has multiple rows with the same OrderID showing which run it was in at and what date and so forth.
I really need suggestions from all the guru's out there.
Here is the SQL:
DECLARE #CompanyID INT = 22718,
#StartDate DATETIME = '',
#EndDate DATETIME = '',
#SalesRepID INT = NULL,
#AssignedTo INT = NULL,
#ServiceDefID INT = NULL,
#ProductName VARCHAR(512) = NULL,
#IsCCOrder BIT = NULL,
#OrderID INT = NULL,
#LocationID INT = NULL,
#SalesRepLocationID INT = NULL,
#PONum VARCHAR(150) = NULL,
#InvoiceID INT = NULL,
#IsSplitOrder BIT = NULL,
#ContactID INT = NULL,
#ContactName VARCHAR(150) = NULL,
#JobName VARCHAR(200) = NULL,
#Notes VARCHAR(MAX) = NULL,
#CompanyName VARCHAR(255) = NULL,
#DueDateFrom DATETIME = '',
#DueDateTo DATETIME = '',
#SubmitedDateFrom DATETIME = '',
#SubmitedDateTo DATETIME = '',
#IsRush BIT = NULL,
#Msg VARCHAR(1000) = NULL
--#Stages dbo.int_tbltype READONLY
DECLARE #Stages TABLE (ID INT)
--INSERT INTO #Stages (ID) VALUES (1)
DECLARE #DueDate DATETIME = NULL
SET NOCOUNT ON
DECLARE #OrderIDsTBL TABLE(OrderID INT)
IF #Msg IS NOT NULL
BEGIN
INSERT INTO #OrderIDsTBL (OrderID)
SELECT OrderID
FROM tblOrderLog
WHERE Msg LIKE '%' + #Msg + '%'
END
IF #OrderID IS NOT NULL
BEGIN
INSERT INTO #OrderIDsTBL (OrderID)
VALUES (#OrderID)
END
DECLARE #OderIDsCnt INT = (SELECT COUNT(OrderID) FROM #OrderIDsTBL)
DECLARE #StageCnt INT = (SELECT COUNT(ID) FROM #Stages)
SELECT
o.OrderID,
o.CompanyID,
co.Name AS CompanyName,
o.ContactID,
o.JobName,
p.FirstName + ' ' + p.LastName AS ContactName,
p2.FirstName + ' ' + p2.LastName AS SalesRep,
o.DueDate,
CASE WHEN MAX(oi.PriorityService) > 0 THEN 1 ELSE 0 END AS IsRush,
ISNULL(s.StageID, 0) AS StageID,
o.Notes, r.SubmittedComplete,
dbo.fOrderRunLocationCSVByOrderID(o.OrderID) AS LocationCSV,
(SELECT
STUFF((SELECT DISTINCT ' ' + st.Name + '<br />' FROM tblStage st
INNER JOIN tblOrderItem oi ON oi.OrderID = o.OrderID
INNER JOIN tblRun r ON r.OrderItemID = oi.OrderItemID
INNER JOIN tblStage s ON s.StageID = r.StageID
LEFT JOIN tblRunService rs ON rs.RunID = r.RunID
WHERE (s.StageID = st.StageID)
AND (rs.AssignedTo = #AssignedTo OR #AssignedTo IS NULL)
FOR XML PATH(''), TYPE).value('.','VARCHAR(max)'), 1, 1, ''))
AS Stages,
Row_Number() Over(Order By o.OrderID Desc) As RowNum
FROM
tblOrder o
INNER JOIN
tblCompany co ON co.CompanyID = o.CompanyID
INNER JOIN
tblParty p ON p.PartyID = o.ContactID
-------- PROBLEM JOINS ------------
LEFT JOIN
tblOrderItem oi ON oi.OrderID = o.OrderID
LEFT JOIN
tblRun r ON r.OrderItemID = oi.OrderItemID
LEFT JOIN
tblService srv ON srv.OrderItemID = oi.OrderItemID
-------- END PROBLEM JOINS ------------
LEFT JOIN
tblStage s ON s.StageID = r.StageID
LEFT JOIN
tblParty p2 ON p2.PartyID = o.SalesRepID
LEFT JOIN
tblEmployee e ON e.EmployeeID = o.SalesRepID
LEFT JOIN
tblShipTo st ON o.ShipToID = st.ShipToID
WHERE
(#CompanyID IS NULL OR (o.CompanyID = #CompanyID )) AND
(#IsCCOrder IS NULL OR (ISNULL(o.IsCreditCardOrder, 0) = #IsCCOrder )) AND
(#SalesRepID IS NULL OR o.SalesRepID = #SalesRepID) AND
(#ServiceDefID IS NULL OR (srv.ServiceDefID = #ServiceDefID)) AND
(#ProductName IS NULL OR (oi.Name LIKE '%' + #ProductName + '%')) AND
(#IsSplitOrder IS NULL OR (#IsSplitOrder = 1 AND oi.SplitOrderID IS NOT NULL)) AND
(
(#StartDate = '' OR #EndDate = '') OR
(#StartDate >= CreatedDate AND #EndDate <= COALESCE(CancelledDate, ClosedDate, GetDate())) OR
(#StartDate <= COALESCE(CancelledDate, ClosedDate, GETDATE()) AND #EndDate >= COALESCE(CancelledDate, ClosedDate, GetDate()) ) OR
(#StartDate <= CreatedDate AND #EndDate >= CreatedDate )
) AND
(#LocationID IS NULL OR (#LocationID = srv.LocationID OR srv.LocationID IS NULL)) AND
(#SalesRepLocationID IS NULL OR (#SalesRepLocationID = e.LocationID OR e.LocationID IS NULL))
AND (#InvoiceID IS NULL OR o.InvoiceID = #InvoiceID )
AND (#PONum IS NULL OR o.PONum LIKE '%' + #PONum + '%')
AND (COALESCE(s.StageID, 0) IN (SELECT ID FROM #Stages) OR #StageCnt = 0)
AND (o.ContactID = #ContactID OR #ContactID IS NULL)
AND (p.FirstName + ' ' + p.LastName LIKE '%' + #ContactName + '%' OR #ContactName IS NULL)
AND (o.JobName LIKE '%' + #JobName + '%' OR #JobName IS NULL)
AND (o.Notes LIKE '%' + #Notes + '%' OR #Notes IS NULL)
AND (co.Name LIKE '%' + #CompanyName + '%' OR #CompanyName IS NULL)
AND (o.DueDate >= #DueDateFrom OR #DueDateFrom = '')
AND (o.DueDate <= #DueDateTo OR #DueDateTo = '')
AND (r.SubmittedComplete >= #SubmitedDateFrom OR #SubmitedDateFrom = '')
AND (r.SubmittedComplete <= #SubmitedDateTo OR #SubmitedDateTo = '')
AND (#IsRush = (CASE WHEN oi.PriorityService > 0 THEN 1 ELSE 0 END)
OR #IsRush IS NULL)
AND (o.OrderID IN (SELECT OrderID FROM #OrderIDsTBL) OR #OderIDsCnt = 0)
GROUP BY
o.OrderID, o.CompanyID,
co.Name,
o.ContactID, o.JobName,
p.FirstName, p.LastName, p2.FirstName, p2.LastName,
o.DueDate, o.Notes,
r.SubmittedComplete,
s.StageID
Thanks for any suggestions. I've been working on this for some time now and just can't get it working right.
It looks like you're trying to do too much with a single SELECT statement. If you want one row per unique OrderID, then don't join with the tables that have multiple rows for the same OrderID. Remove the GROUP BY clause. Use one or more separate SELECT statements to get the details from the tables that have multiple rows per OrderID.
Thanks everyone for the suggestions but I found my own solution.
With the entire sql, I placed it into a temp table.
Then used this to sort out the duplicate OrderIds....
SELECT OrderID, CompanyID, ContactID, CompanyName, JobName, ContactName,
SalesRep, DueDate, IsRush , StageID, Notes, SubmittedComplete,
LocationCSV, Stages
FROM (SELECT *, ROW_NUMBER() OVER(PARTITION BY OrderID ORDER BY OrderID DESC) 'RowRank'
FROM #SearchTbl
)sub
WHERE RowRank = 1

Handling multiple records in a MS SQL trigger

I am having to use triggers in MSSQL for the first time, well triggers in general. Having read around and tested this myself I realise now that a trigger fires per command and not per row inserted, deleted or updated.
The entire thing is some statistics for an advertising system. Our main stat table is rather large and doesn't contain the data in a way that makes sense in most cases. It contains one row per advert clicked, viewed and etc. As a user one is more inclined to want to view this as day X has Y amount of clicks and Z amount of views and so forth. We have done this purely based on a SQL query so far, getting this sort of report from the main table, but as the table has grown so does the time for that query to execute. Because of this we have opted for using triggers to keep another table updated and hence making this a bit easier on the SQL server.
My issue is now to get this working with multiple records. What I have done is to create 2 stored procedures, one for handling the operation of an insert, and one for a delete. My insert trigger (written to work with a single record) then graps the data off the Inserted table, and sends it off to the stored procedure. The delete trigger works in the same way, and (obviously?) the update trigger does the same as a delete + an insert.
My issue is now how to best do this with multiple records. I have tried using a cursor, but as far as I have been able to read and see myself, this performs really badly. I have considered writing some "checks" as well - as in checking to see IF there are multiple records in the commands and then go with the cursor, and otherwise simply just avoid this. Anyhow, here's my solution with a cursor, and im wondering if there's a way of doing this better?
CREATE TRIGGER [dbo].[TR_STAT_INSERT]
ON [iqdev].[dbo].[Stat]
AFTER INSERT
AS
BEGIN
SET NOCOUNT ON;
DECLARE #Date DATE
DECLARE #CampaignId BIGINT
DECLARE #CampaignName varchar(500)
DECLARE #AdvertiserId BIGINT
DECLARE #PublisherId BIGINT
DECLARE #Unique BIT
DECLARE #Approved BIT
DECLARE #PublisherEarning money
DECLARE #AdvertiserCost money
DECLARE #Type smallint
DECLARE InsertCursor CURSOR FOR SELECT Id FROM Inserted
DECLARE #curId bigint
OPEN InsertCursor
FETCH NEXT FROM InsertCursor INTO #curId
WHILE ##FETCH_STATUS = 0
BEGIN
SELECT #Date = [Date], #PublisherId = [PublisherCustomerId], #Approved = [Approved], #Unique = [Unique], #Type = [Type], #AdvertiserCost = AdvertiserCost, #PublisherEarning = PublisherEarning
FROM Inserted
WHERE Id = #curId
SELECT #CampaignId = T1.CampaignId, #CampaignName = T2.Name, #AdvertiserId = T2.CustomerId
FROM Advert AS T1
INNER JOIN Campaign AS T2 on T1.CampaignId = T2.Id
WHERE T1.Id = (SELECT AdvertId FROM Inserted WHERE Id = #curId)
EXEC ProcStatInsertTrigger #Date, #CampaignId, #CampaignName, #AdvertiserId, #PublisherId, #Unique, #Approved, #PublisherEarning, #AdvertiserCost, #Type
FETCH NEXT FROM InsertCursor INTO #curId
END
CLOSE InsertCursor
DEALLOCATE InsertCursor
END
The stored procedure is rather big and intense and I do not think there's a way of having to avoid looping through the records of the Inserted table in one way or another (ok, maybe there is, but I'd like to be able to read the code too :p), so I'm not gonna bore you with that one (unless you like to think otherwise). So pretty much, is there a better way of doing this, and if so, how?
EDIT: Well after request, here's the sproc
CREATE PROCEDURE ProcStatInsertTrigger
#Date DATE,
#CampaignId BIGINT,
#CampaignName varchar(500),
#AdvertiserId BIGINT,
#PublisherId BIGINT,
#Unique BIT,
#Approved BIT,
#PublisherEarning money,
#AdvertiserCost money,
#Type smallint
AS
BEGIN
-- SET NOCOUNT ON added to prevent extra result sets from
-- interfering with SELECT statements.
SET NOCOUNT ON;
IF #Approved = 1
BEGIN
DECLARE #test bit
SELECT #test = 1 FROM CachedStats WHERE [Date] = #Date AND CampaignId = #CampaignId AND CustomerId = #PublisherId
IF #test IS NULL
BEGIN
INSERT INTO CachedStats ([Date], CustomerId, CampaignId, CampaignName) VALUES (#Date, #PublisherId, #CampaignId, #CampaignName)
END
SELECT #test = NULL
DECLARE #Clicks int
DECLARE #TotalAdvertiserCost money
DECLARE #TotalPublisherEarning money
DECLARE #PublisherCPC money
DECLARE #AdvertiserCPC money
SELECT #Clicks = Clicks, #TotalAdvertiserCost = AdvertiserCost + #AdvertiserCost, #TotalPublisherEarning = PublisherEarning + #PublisherEarning FROM CachedStats
WHERE [Date] = #Date AND CustomerId = #PublisherId AND CampaignId = #CampaignId
IF #Type = 0 -- If click add one to the calculation
BEGIN
SELECT #Clicks = #Clicks + 1
END
IF #Clicks > 0
BEGIN
SELECT #PublisherCPC = #TotalPublisherEarning / #Clicks, #AdvertiserCPC = #TotalAdvertiserCost / #Clicks
END
ELSE
BEGIN
SELECT #PublisherCPC = 0, #AdvertiserCPC = 0
END
IF #Type = 0
BEGIN
UPDATE CachedStats SET
Clicks = #Clicks,
UniqueClicks = UniqueClicks + #Unique,
PublisherEarning = #TotalPublisherEarning,
AdvertiserCost = #TotalAdvertiserCost,
PublisherCPC = #PublisherCPC,
AdvertiserCPC = #AdvertiserCPC
WHERE [Date] = #Date AND CustomerId = #PublisherId AND CampaignId = #CampaignId
END
ELSE IF #Type = 1 OR #Type = 4 -- lead or coreg
BEGIN
UPDATE CachedStats SET
Leads = Leads + 1,
PublisherEarning = #TotalPublisherEarning,
AdvertiserCost = #TotalAdvertiserCost,
AdvertiserCPC = #AdvertiserCPC,
PublisherCPC = #AdvertiserCPC
WHERE [Date] = #Date AND CustomerId = #PublisherId AND CampaignId = #CampaignId
END
ELSE IF #Type = 3 -- Isale
BEGIN
UPDATE CachedStats SET
Leads = Leads + 1,
PublisherEarning = #TotalPublisherEarning,
AdvertiserCost = #TotalAdvertiserCost,
AdvertiserCPC = #AdvertiserCPC,
PublisherCPC = #AdvertiserCPC,
AdvertiserOrderValue = #AdvertiserCost,
PublisherOrderValue = #PublisherEarning
WHERE [Date] = #Date AND CustomerId = #PublisherId AND CampaignId = #CampaignId
END
ELSE IF #Type = 2 -- View
BEGIN
UPDATE CachedStats SET
[Views] = [Views] + 1,
UniqueViews = UniqueViews + #Unique,
PublisherEarning = #TotalPublisherEarning,
AdvertiserCost = #TotalAdvertiserCost,
PublisherCPC = #PublisherCPC,
AdvertiserCPC = #AdvertiserCPC
WHERE [Date] = #Date AND CustomerId = #PublisherId AND CampaignId = #CampaignId
END
END
END
After help, here's my final result, posted in case others have a similiar issue
CREATE TRIGGER [dbo].[TR_STAT_INSERT]
ON [iqdev].[dbo].[Stat]
AFTER INSERT
AS
BEGIN
SET NOCOUNT ON
-- insert all missing "CachedStats" rows
INSERT INTO
CachedStats ([Date], AdvertId, CustomerId, CampaignId, CampaignName)
SELECT DISTINCT
CONVERT(Date, i.[Date]), i.AdvertId, i.[PublisherCustomerId], c.Id, c.Name
FROM
Inserted i
INNER JOIN Advert AS a ON a.Id = i.AdvertId
INNER JOIN Campaign AS c ON c.Id = a.CampaignId
WHERE
i.[Approved] = 1
AND NOT EXISTS (
SELECT 1
FROM CachedStats as t
WHERE
[Date] = CONVERT(Date, i.[Date])
AND CampaignId = c.Id
AND CustomerId = i.[PublisherCustomerId]
AND t.AdvertId = i.AdvertId
)
-- update all affected records at once
UPDATE
CachedStats
SET
Clicks =
Clicks + (
SELECT COUNT(*) FROM Inserted s
WHERE s.Approved = 1
AND s.PublisherCustomerId = i.PublisherCustomerId
AND CONVERT(Date, s.[Date]) = CONVERT(Date, i.[Date])
AND s.AdvertId = i.AdvertId
AND s.[Type] = 0
),
UniqueClicks =
UniqueClicks + (
SELECT COUNT(*) FROM Inserted s
WHERE s.Approved = 1
AND s.[Unique] = 1
AND s.PublisherCustomerId = i.PublisherCustomerId
AND CONVERT(Date, s.[Date]) = CONVERT(Date, i.[Date])
AND s.AdvertId = i.AdvertId
AND s.[Type] = 0
),
[Views] =
[Views] + (
SELECT COUNT(*) FROM Inserted s
WHERE s.Approved = 1
AND s.PublisherCustomerId = i.PublisherCustomerId
AND CONVERT(Date, s.[Date]) = CONVERT(Date, i.[Date])
AND s.AdvertId = i.AdvertId
AND s.[Type] = 2
),
UniqueViews =
UniqueViews + (
SELECT COUNT(*) FROM Inserted s
WHERE s.Approved = 1
AND s.[Unique] = 1
AND s.PublisherCustomerId = i.PublisherCustomerId
AND CONVERT(Date, s.[Date]) = CONVERT(Date, i.[Date])
AND s.AdvertId = i.AdvertId
AND s.[Type] = 2
),
Leads =
Leads + (
SELECT COUNT(*) FROM Inserted s
WHERE s.Approved = 1
AND s.[Unique] = 1
AND s.PublisherCustomerId = i.PublisherCustomerId
AND CONVERT(Date, s.[Date]) = CONVERT(Date, i.[Date])
AND s.AdvertId = i.AdvertId
AND s.[Type] IN (1,3,4)
),
PublisherEarning =
CachedStats.PublisherEarning + ISNULL((
SELECT SUM(PublisherEarning) FROM Inserted s
WHERE s.Approved = 1
AND s.PublisherCustomerId = i.PublisherCustomerId
AND CONVERT(Date, s.[Date]) = CONVERT(Date, i.[Date])
AND s.AdvertId = i.AdvertId
), 0),
AdvertiserCost =
CachedStats.AdvertiserCost + ISNULL((
SELECT SUM(AdvertiserCost) FROM Inserted s
WHERE s.Approved = 1
AND s.PublisherCustomerId = i.PublisherCustomerId
AND CONVERT(Date, s.[Date]) = CONVERT(Date, i.[Date])
AND s.AdvertId = i.AdvertId
), 0),
PublisherOrderValue =
PublisherOrderValue + ISNULL((
SELECT SUM(PublisherEarning) FROM Inserted s
WHERE s.Approved = 1
AND s.PublisherCustomerId = i.PublisherCustomerId
AND CONVERT(Date, s.[Date]) = CONVERT(Date, i.[Date])
AND s.AdvertId = i.AdvertId
AND s.[Type] = 3
), 0),
AdvertiserOrderValue =
AdvertiserOrderValue + ISNULL((
SELECT SUM(AdvertiserCost) FROM Inserted s
WHERE s.Approved = 1
AND s.PublisherCustomerId = i.PublisherCustomerId
AND CONVERT(Date, s.[Date]) = CONVERT(Date, i.[Date])
AND s.AdvertId = i.AdvertId
AND s.[Type] = 3
), 0),
PublisherCPC =
CASE WHEN (Clicks + (
SELECT COUNT(*) FROM Inserted s
WHERE s.Approved = 1
AND s.PublisherCustomerId = i.PublisherCustomerId
AND CONVERT(Date, s.[Date]) = CONVERT(Date, i.[Date])
AND s.AdvertId = i.AdvertId
AND s.[Type] = 0
)) > 0 THEN
(CachedStats.PublisherEarning + ISNULL((
SELECT SUM(PublisherEarning) FROM Inserted s
WHERE s.Approved = 1
AND s.PublisherCustomerId = i.PublisherCustomerId
AND CONVERT(Date, s.[Date]) = CONVERT(Date, i.[Date])
AND s.AdvertId = i.AdvertId
), 0)) -- COST ^
/ (
Clicks + (
SELECT COUNT(*) FROM Inserted s
WHERE s.Approved = 1
AND s.PublisherCustomerId = i.PublisherCustomerId
AND CONVERT(Date, s.[Date]) = CONVERT(Date, i.[Date])
AND s.AdvertId = i.AdvertId
AND s.[Type] = 0
)
) --- Clicks ^
ELSE
0
END,
AdvertiserCPC =
CASE WHEN (Clicks + (
SELECT COUNT(*) FROM Inserted s
WHERE s.Approved = 1
AND s.PublisherCustomerId = i.PublisherCustomerId
AND CONVERT(Date, s.[Date]) = CONVERT(Date, i.[Date])
AND s.AdvertId = i.AdvertId
AND s.[Type] = 0
)) > 0 THEN
(CachedStats.AdvertiserCost + ISNULL((
SELECT SUM(AdvertiserCost) FROM Inserted s
WHERE s.Approved = 1
AND s.PublisherCustomerId = i.PublisherCustomerId
AND CONVERT(Date, s.[Date]) = CONVERT(Date, i.[Date])
AND s.AdvertId = i.AdvertId
), 0)) -- COST ^
/ (
Clicks + (
SELECT COUNT(*) FROM Inserted s
WHERE s.Approved = 1
AND s.PublisherCustomerId = i.PublisherCustomerId
AND CONVERT(Date, s.[Date]) = CONVERT(Date, i.[Date])
AND s.AdvertId = i.AdvertId
AND s.[Type] = 0
)
) --- Clicks ^
ELSE
0
END
FROM
Inserted i
WHERE
i.Approved = 1 AND
CachedStats.Advertid = i.AdvertId AND
CachedStats.[Date] = Convert(Date, i.[Date]) AND
CachedStats.CustomerId = i.PublisherCustomerId
SET NOCOUNT OFF
END
It looks slightly different now because I had to index it per advertisement too - but thanks alot for the help - sped everything up from 30hour+ to 30 sec to generate the CachedStats from my own development Stat table :)
The trick with these kinds of situations is to turn the sequential operation (for each record do xyz) into a set-based operation (an UPDATE statement).
I have analyzed your stored procedure and merged your separate UPDATE statements into a single one. This single statement can then be transformed into a version that can be applied to all inserted records at once, eliminating the need for a stored procedure and thereby the need for a cursor.
EDIT: Below is the code that we finally got working. Execution time for the whole operation went down from "virtually forever" (for the original solution) to something under one second, according to the OP's feedback. Overall code size also decreased quite noticeably.
CREATE TRIGGER [dbo].[TR_STAT_INSERT]
ON [iqdev].[dbo].[Stat]
AFTER INSERT
AS
BEGIN
SET NOCOUNT ON
-- insert all missing "CachedStats" rows
INSERT INTO
CachedStats ([Date], AdvertId, CustomerId, CampaignId, CampaignName)
SELECT DISTINCT
CONVERT(Date, i.[Date]), i.AdvertId, i.PublisherCustomerId, c.Id, c.Name
FROM
Inserted i
INNER JOIN Advert a ON a.Id = i.AdvertId
INNER JOIN Campaign c ON c.Id = a.CampaignId
WHERE
i.Approved = 1
AND NOT EXISTS (
SELECT 1
FROM CachedStats
WHERE Advertid = i.AdvertId AND
CustomerId = i.PublisherCustomerId AND
[Date] = CONVERT(DATE, i.[Date])
)
-- update all affected records at once
UPDATE
CachedStats
SET
Clicks = Clicks + i.AddedClicks,
UniqueClicks = UniqueClicks + i.AddedUniqueClicks,
[Views] = [Views] + i.AddedViews,
UniqueViews = UniqueViews + i.AddedUniqueViews,
Leads = Leads + i.AddedLeads,
PublisherEarning = PublisherEarning + ISNULL(i.AddedPublisherEarning, 0),
AdvertiserCost = AdvertiserCost + ISNULL(i.AddedAdvertiserCost, 0),
PublisherOrderValue = PublisherOrderValue + ISNULL(i.AddedPublisherOrderValue, 0),
AdvertiserOrderValue = AdvertiserOrderValue + ISNULL(i.AddedAdvertiserOrderValue, 0)
FROM
(
SELECT
AdvertId,
CONVERT(DATE, [Date]) [Date],
PublisherCustomerId,
COUNT(*) NumRows,
SUM(CASE WHEN Type IN (0) THEN 1 ELSE 0 END) AddedClicks,
SUM(CASE WHEN Type IN (0) AND [Unique] = 1 THEN 1 ELSE 0 END) AddedUniqueClicks,
SUM(CASE WHEN Type IN (2) THEN 1 ELSE 0 END) AddedViews,
SUM(CASE WHEN Type IN (2) AND [Unique] = 1 THEN 1 ELSE 0 END) AddedUniqueViews,
SUM(CASE WHEN Type IN (1,3,4) AND [Unique] = 1 THEN 1 ELSE 0 END) AddedLeads,
SUM(PublisherEarning) AddedPublisherEarning,
SUM(AdvertiserCost) AddedAdvertiserCost,
SUM(CASE WHEN Type IN (3) THEN PublisherOrderValue ELSE 0 END) AddedPublisherOrderValue,
SUM(CASE WHEN Type IN (3) THEN AdvertiserOrderValue ELSE 0 END) AddedAdvertiserOrderValue
FROM
Inserted
WHERE
Approved = 1
GROUP BY
AdvertId,
CONVERT(DATE, [Date]),
PublisherCustomerId
) i
INNER JOIN CachedStats cs ON
cs.Advertid = i.AdvertId AND
cs.CustomerId = i.PublisherCustomerId AND
cs.[Date] = i.[Date]
SET NOCOUNT OFF
END
The operations involving the CachedStats table will greatly benefit from one multiple-column index over (Advertid, CustomerId, [Date]) (as confirmed by the OP).
Depending on what version of MSSQL you are running, you should also consider using Indexed Views for this as well. That could very well be a simpler approach than your triggers, depending on what the report query looks like. See here for more info.
Also, in your trigger, you should try to write your updates to the materialized results table as a set based operation, not a cursor. Writing a cursor based trigger could potentially just be moving your problem from the report query to your table inserts instead.
First thing I would do is use a FAST_FORWARD cursor instead. As you are only going from one record to the next and not doing any updates this will be much better for performance.
DECLARE CURSOR syntax
You can slightly optimize your cursor variation by doing FAST_FORWARD, READ_ONLY and LOCAL options on the cursor. Also, you're pulling the Id into your cursor, and then looping back to get the values. Either use CURRENT_OF or throw them all into variables. But, I wouldn't expect these changes to buy you much.
You really need to move to a set based approach. That stored proc is definitely doable in a set based model - though it may take 3 or 4 different update statements. But even 3 or 4 different triggers (1 for views, 1 for clicks, etc.) would be better than the cursor approach.
Your best bet is to move to a set based operation in the trigger. I'm not going write this for you 100% but let me get you started, and we can see where we go from there. Keep in mind I am writting this without tables / schemas and so I'm not going validate. Expect Typos:-)
Let's look at your update statements first, From what I can tell you are updating the same table with the same where clause the only difference is the columns. You can consolidate this to look like:
UPDATE CachedStats SET
/* Basically we are going to set the counts based on the type inline in the update clause*/
Leads= CASE WHEN (#Type = 1 OR #Type = 4 OR #Type=3 ) THEN Leads + 1 ELSE LEADS END,
Clicks=CASE WHEN (#Type=0) THEN Clicks+1 ELSE Clicks END,
Views=CASE WHEN (#Type=4) THEN Views+1 ELSE Views END,
PublisherEarning = #PublisherEarning + PublisherEarning,
AdvertiserCost = #AdvertiserCost +AdvertiserCost,
FROM CachedStats CS
INNER JOIN Inserted INS
ON CS.Date=Inserted.Date AND CS.CustomerId=Ins.PublisherId AND CS.CampaignId=Ins.CampaignId
I do aggree with you that this could get ugly but that's a decision you'll have to make.
As for your insert clause, I would handle that the same way you already are just insert into the table from the Inserted table whatever doesn't already exist.

Resources