I am working with some communications and returning data to a PLC, and executing a stored procedure often.
I was wondering if I could get an opinion on how I might be able to increase the efficiency of this stored procedure?
As of now, it runs roughly ~1600 or so times a day, which is fine. But moving forward, it's likely to run upwards of 5000 or some times a day. I was wondering what might be the best method to reduce the number of SELECT statements, is it possible I can use a CASE statement?
Below is some of the example code.
#machine nvarchar(25),
#suppliercode nvarchar(1),
#barcode nvarchar(16),
#sqlpass int OUTPUT,
#sqlfail int OUTPUT,
#output int OUTPUT,
#lotnomatch int OUTPUT,
#partnomatch int OUTPUT,
#foundlot nvarchar(16) OUTPUT
AS
BEGIN
SET #output = (SELECT COUNT(ID) FROM dbo.MCData WHERE Barcode = #barcode AND Machine = #machine)
IF (#output >= 1)
BEGIN
SET #sqlpass = 1
UPDATE dbo.MCBarcodeData SET Interfaced = 1, InterfacedDatetime = GetDate(), InterfacedMachine = #machine WHERE Barcode = #barcode AND Machine = #machine
END
IF (#output = 0)
BEGIN
SET #lotnomatch = (SELECT COUNT(ID) FROM dbo.MCData WHERE Barcode = #barcode AND Machine != #machine)
END
IF (#lotnomatch = 1)
BEGIN
SET #foundlot = (SELECT Machine FROM dbo.MCData WHERE Barcode = #Barcode)
END
IF (#output = 0 AND #lotnomatch = 0)
BEGIN
SET #partnomatch = 1
END
END
GO
Edit: Attached if the Query Execution Plan
Calculating statistics from a whole picture perspective on a regular basis always gets worse over time with more and more data. If it is something that isn't done very often then it isn't so bad, but if your doing it very often then it will hurt in the long run.
What I like to do is maintain the statistics as data is modified in a separate table.
For example, you might want a separate table that has a composite primary key of barcode and machine and then have a column for number of records in mcdata that match your key and number of records in mcdata that have the same barcode but not the same machine. You can also have a column for the machine name for the one that doesn't match if there is one.
CREATE TABLE MCDataStats
(
Barcode nvarchar(16),
Machine nvarchar(25),
NumMatchBarcodeMachineInMCData int,
NumMatchBarcodeNotMachineInMCData int,
LotMachineName nvarchar(25),
PRIMARY KEY (Barcode, Machine)
)
Then you could just run a single simple select on that statistics table.
CREATE PROCEDURE procMCDataStats_GET
#Barcode nvarchar(16),
#Machine nvarchar(25)
AS
BEGIN
SELECT *
FROM MCDataStats
WHERE Barcode = #Barcode AND
Machine = #Machine
END
To maintain the statistics you would update them as data is modified. Like in your stored procedure to add a record to mcdata. You would then insert a record in your statistics table for that barcode and machine you don't have it already. Then you would increment the number of records in mcdata column and then increment the number of records in mcdata that have same barcode but not the same machine for records in your statistics table with that barcode, but not the same machine. If you incremented any records in statistics table with that barcode, but not the same machine then you can verify the stats that are there to determine if you want to populate the lot column. And if you remove any records, make sure you go through and decrement.
CREATE PROCEDURE procMCData_ADD
#Barcode nvarchar(16),
#Machine nvarchar(25)
AS
BEGIN
INSERT INTO MCData (Barcode, Machine, ...)
VALUES (#Barcode, #Machine, ...)
--Insert into MCDataStats if barcode and machine don't exist
INSERT INTO MCDataStats (Barcode, Machine, NumMatchBarcodeMachineInMCData,
NumMatchBarcodeNotMachineInMCData)
SELECT #Barcode, #Machine, 0, 0
WHERE NOT EXISTS (SELECT TOP 1 * FROM MCDataStats WHERE Barcode = #Barcode AND Machine = #Machine)
UPDATE data
SET Interfaced = 1, InterfacedDatatime = GETDATE(), InterfacedMachine = #Machine
FROM MCDataStats AS stats
LEFT JOIN MCBarcodeData AS data ON data.Barcode = stats.Barcode AND data.Machine = stats.Machine AND stats.NumMatchBarcodeMachineInMCData = 1
--Update stats for matching barcode and machine
UPDATE stats
SET NumMatchBarcodeMachineInMCData = NumMatchBarcodeMachineInMCData + 1
FROM MCDataStats AS stats
WHERE Barcode = #Barcode AND Machine = #Machine
--Update stats for matching barcode but not machine
UPDATE stats
SET NumMatchBarcodeNotMachineInMCData = NumMatchBarcodeNotMachineInMCData + 1,
LotMachineName = (CASE WHEN NumMatchBarcodeMachineInMCData = 0 AND NumMatchBarcodeNotMachineInMCData = 0 THEN #Machine ELSE NULL END)
FROM MCDataStats AS stats
WHERE Barcode = #Barcode AND Machine <> #Machine
END
In my opinion you should consider to add non-clustered index in your tables.
For example:
CREATE NONCLUSTERED INDEX IX_MCData_Barcode_Machine
ON dbo.MCData (Barcode ASC, Machine ASC)
GO
CREATE NONCLUSTERED INDEX IX_MCBarcodeData_BarcodeMachine
ON dbo.MCBarcodeData (Barcode ASC, Machine ASC)
INCLUDE (Interfaced, InterfacedDatetime, InterfacedMachine)
GO
Related
I have inherited a system where there is a table X where points are allocated to users. Below is an example of how this looks like -
When a row is added the NewBalance is the actual total balance for the user.
How this is done is in a transaction (I am attaching values which are dynamically passed):
BEGIN
DECLARE #userId uniqueidentifier = 'DA04C99F-575A-434F-BD69-05F2C111360E'
DECLARE #oldBalance int
DECLARE #newPoints int = 25
SELECT TOP 1 #oldBalance = NewBalance FROM X WHERE UserId = ORDER BY [Date]
DESC
INSERT INTO X (UserId, Date, Value, NewBalance) VALUES (#userId,
SYSDATETIMEOFFSET(), #newPoints, #oldBalance + #newPoints)
END
The above piece of code can be called from multiple modules to add points each of which are running in different isolation levels. It is also possible that this gets called concurrently from 2 different modules - so we end up with something like this -
Obviously 2 different transactions read the same row while fetching the initial #oldBalance and then each added to it resulting in the problem.
We are thinking of making all modules that call this piece of code to run under Serializable isolation level. But we are having trouble replicating the problem in lower environments and so there is no realistic way to test.
Also, from what I understand irrespective of isolation level the SELECT TOP 1 will always place a shared lock on the row so it can be read by other transactions.
Any tips or reading material to solve the problem would be appreciated.
Depending on how much changeability you have over the table structure itself, you could do as others have suggested restrict it to a single row to show only the current row and then use a lock on the table.
You could also implement a counter type for each of the unique records so the current iteration of data inserted is always sequential because you're checking if it's been updated since you started your update.
I have added a simple way of doing the update using a procedure rather than just inserting. I am not aware of your application or database, but if you are able to read the output and store the balance to add you are able to implement a retry until the values match and it enables the insert.
CREATE TABLE X (UserId UniqueIdentifier, Date DATETIME, Value INT, NewBalance INT)
INSERT INTO X VALUES (NEWID(), GETUTCDATE(),10,10)
SELECT * FROM X
BEGIN
DECLARE #userId uniqueidentifier = '5396C445-8AC1-4B46-8E25-A416059D7976'
DECLARE #oldBalance int
DECLARE #newPoints int = 25
SELECT TOP 1 #oldBalance = NewBalance FROM X WHERE UserId = #userId ORDER BY [Date] DESC
EXEC dbo.usp_Update_X #userId, #oldBalance, #newPoints
END
CREATE PROCEDURE dbo.usp_Update_X
(
#UserID UniqueIdentifier
,#OldBalance INT
,#newPoints INT
)
AS
IF #OldBalance = (SELECT TOP 1 NewBalance FROM X WHERE UserId = #UserID ORDER BY Date DESC)
BEGIN
INSERT INTO X (UserId, Date, Value, NewBalance) VALUES (#userId,
SYSDATETIMEOFFSET(), #newPoints, #oldBalance + #newPoints)
Print 'Inserted Successfully'
END
ELSE
Print 'Changed - Get Balance Again'
GO
Once something like this has been implemented, you could then periodically check to ensure the values go in the correct order
SELECT UserId, Date, NewBalance, Value, SUM(Value) OVER (Partition By UserId ORDER BY Date ASC) AS NewBalanceCheck
FROM X
I would like to create a set of new rows in a DB where ID is = to 10,11,12,13,14,15 but all other values are null. This is assuming rows 1 through 9 already exist (in this example). My application will set the first and last row parameters.
Here's my query to create one row but I need a way to loop through rows 10 through 15 until all five rows are created:
#FirstRow int = 10 --will be set by application
,#LastRow int = 15 --will be set by application
,#FileName varchar(100) = NULL
,#CreatedDate date = NULL
,#CreatedBy varchar (50) = NULL
AS
BEGIN
INSERT INTO TABLE(TABLE_ID, FILENAME, CREATED_BY, CREATED_DATE)
VALUES (#FirstRow, #FileName, #CreatedBy, #CreatedDate)
END
The reason I need blank rows is because the application needs to update an existing row in a table. My application will be uploading thousands of documents to rows in a table based on file ID. The application requires that the rows already be inserted. The files are inserted after rows are added. The app then deletes all rows that are null.
Assuming the rows you're inserting are always consecutive, you can use a ghetto FOR loop like the one below to accomplish your goal:
--put all the other variable assignments above this line
DECLARE #i int = #FirstRow
WHILE (#i <= #LastRow)
BEGIN
INSERT INTO TABLE(TABLE_ID, FILENAME, CREATED_BY, CREATED_DATE)
VALUES (#i, #FileName, #CreatedBy, #CreatedDate)
SET #i = #i + 1;
END
Basically, we assigned #i to the lowest index, and then just iterate through, one by one, until we're at the max index.
If performance is a concern, the above approach will not be ideal.
If you don't have a numbers table as SMor mentioned, you can use an ad-hoc tally table
Example
Declare #FirstRow int = 10 --will be set by application
,#LastRow int = 15 --will be set by application
,#FileName varchar(100) = NULL
,#CreatedDate date = NULL
,#CreatedBy varchar (50) = NULL
INSERT INTO TABLE(TABLE_ID, FILENAME, CREATED_BY, CREATED_DATE)
Select Top (#LastRow-#FirstRow+1)
#FirstRow-1+Row_Number() Over (Order By (Select NULL))
,#FileName
,#CreatedBy
,#CreatedDate
From master..spt_values n1, master..spt_values n2
Data Generated
I have a table that stores an Id, a datetime and an int crescent value. This value increases until it "breaks" and returns to a 0-near value. Ex: ...1000, 1200, 1350, 8, 10, 25...
I need to count how many times this "overflow" happens, BUT... I'm talking about a table that stores 200k rows per day!
I had already solved it! But using a procedure with a cursor that iterates over it with a while-loop. But I KNOW it isn't the faster way to do it.
Can someone help me to find some another way?
Thanks!
->
Table structure:
Id Bigint Primary Key, CreatedAt DateTime, Value Not Null Int.
Problem:
If Delta-Value between two consecutive rows is < 0, increase a counter.
Table has 200k new rows every-day.
No trigger allowed.
[FIRST EDIT]
Table has the actual structure:
CREATE TABLE ValuesLog (
Id BIGINT PRIMARY KEY,
Machine BIGINT,
CreatedAt DATETIME,
Value INT
)
I need:
To check when the [Value] of some [Machine] suddenly decreases.
Some users said to used LEAD/LAG. But it has a problem... if I chose many machines, the LEAD/LAG fuctions doesn't care about "what machine it is". So, if I find for machine-1 and machine-2, if machine-1 increase but the machine-2 descrease, LEAD/LAG will give me a false positive.
So, how my table actually looks:
Many rows of the actual table
(The image above are selecting for 3 ou 4 machines. But, IN THIS EXAMPLE, the machines are not messed up. But can occurs! And in this case, LEAD/LAG doesn't care if the line above are machine-1 or machine-2)
What I want:
In that line 85, the [value] breaks and restart. Id like to count every occorrence when it happens, the selected machines.
So:
"Machine-1 restarted 6 times... Machine-9 restarted 10 times..."
I had done something LIKE this:
CREATE PROCEDURE CountProduction #Machine INT_ARRAY READONLY, #Start DATETIME, #End DATETIME AS
SET NOCOUNT ON
-- Declare counter and insert start values
DECLARE #Counter TABLE(
machine INT PRIMARY KEY,
lastValue BIGINT DEFAULT 0,
count BIGINT DEFAULT 0
)
INSERT INTO #Counter(machine) SELECT n FROM #Machine
-- Declare cursor to iteract over results of values log
DECLARE valueCursor CURSOR LOCAL FOR
SELECT
Value,
Aux.LastValue,
Aux.count
FROM
ValueLog,
#Machine AS Machine,
#Counter AS Counter
WHERE
ValueLog.Machine = Machine.n
AND Counter.machine = ValueLog.Machine
AND ValueLog.DateCreate BETWEEN #Start AND #End;
-- Start iteration
OPEN valueCursor
DECLARE #RowMachine INT
DECLARE #RowValue BIGINT
DECLARE #RowLastValue BIGINT
DECLARE #RowCount BIGINT
FETCH NEXT FROM valueCursor INTO #RowMachine, #RowValue, #RowLastValue, #RowCount
-- Iteration
DECLARE #increment INT
WHILE ##FETCH_STATUS = 0
BEGIN
IF #RowValue < #RowLastValue
SET #increment = 1
ELSE
SET #increment = 0
-- Update counters
UPDATE
#Counter
SET
lastValue = #RowValue,
count = count + #increment
WHERE
inj = #RowMachine
-- Proceed to iteration
FETCH NEXT FROM valueCursor INTO #RowMachine, #RowValue, #RowLastValue, #RowCount
END
-- Closing iteration
CLOSE valueCursor
DEALLOCATE valueCursor
SELECT machine, count FROM #Counter
Use LEAD(). If the next row < current row, count that occurrence.
Solved using #jeroen-mostert suggested
DECLARE #Start DATETIME
DECLARE #End DATETIME
SET #Start = '2019-01-01'
SET #End = GETDATE()
SELECT
Machine,
COUNT(DeltaValue) AS Prod
FROM
(SELECT
Log.Machine,
Log.Value - LAG(Log.Value) OVER (PARTITION BY Log.Machine ORDER BY Log.Id) AS DeltaValue
FROM
ValueLog AS Log,
(SELECT
Id,
Machine,
Value
FROM
ValueLog
) AS AuxLog
WHERE
AuxLog.Id = Log.Id
AND Proto.DateCreate BETWEEN #Start AND #End
AND Proto.Machine IN (1, 9, 10)
) as TB1
WHERE
DeltaValue < 0
GROUP BY
Machine
ORDER BY
Machine
In this case, the inner LAG/LEAD function didn't mess up the content (what happened for some reason when I created a view... I'll try to understand later).
Thanks everybody! I'm new at DB, and this question make me crazy for a whole day.
I have the following stored procedure running against an ExtractedMessages table which might contain up to 100M records (100,000,000).
For the purpose of my application, this stored procedure should run in less than one second
CREATE PROCEDURE GetNextMessages
#taskId bigint
AS
BEGIN
SET NOCOUNT ON;
DECLARE #ci INT
DECLARE #cr INT
SELECT
#ci = CurrentIndex, #cr = CurrentResources
FROM
ExtractedTasks
WHERE
Id = #taskId
UPDATE ExtractedTasks
SET CurrentIndex = #ci + #cr
WHERE Id = #taskId
SELECT *
FROM ExtractedMessages
WHERE TaskId = #taskId
ORDER BY Id
OFFSET #ci ROWS
FETCH NEXT #cr ROWS ONLY
END
NB: cr can not be more than 1500
Instead of querying your table twice you could do the following:
DECLARE #ci INT
DECLARE #cr INT
DECLARE #T TABLE (
ci INT
, cr INT
);
UPDATE ExtractedTasks
SET CurrentIndex = CurrentIndex + CurrentResources
OUTPUT DELETED.CurrentIndex, DELETED.CurrentResources
INTO #T (ci, cr)
WHERE Id = #taskId;
SELECT
#ci = ci, #cr = cr
FROM
#T
SELECT *
FROM ExtractedMessages
WHERE TaskId = #taskId
ORDER BY Id
OFFSET #ci ROWS
FETCH NEXT #cr ROWS ONLY
The key here is OUTPUT clause. It's going to insert deleted records (pre-update values) into a table variable. Which means you replace multiple selects from a table with one. Other than correct indexes I don't see anything for improvement.
Also, make sure you select just the columns you need, not everything. It's generally a good practice to list exact columns you need.
Thanks Everyone for your kind help
I was able to identify the problem, after all it was a memory management issue on the server that is running SQL server. I have increased the dedicated memory of the server. And the issue has been resolved. The SP works smoothly now
here just i am trying to know count of records in the temp table and passing to another variable, so that it was showing in the execution plan table scan about 100 %
please find the below query which i was doing
DECLARE #tmpANHdr TABLE (
hdrId INT IDENTITY,
CBFId NVARCHAR(32),
ACPT_REJ_FLAG NVARCHAR(8),
PROC_FILE_NAME NVARCHAR(50))
INSERT INTO #tmpANHdr
SELECT TOP 100 AHR_CB_FTS_FILE_ID,
AHR_ACCT_REJ_FLAG,
AHR_PROC_FILE_NAME
FROM FTS_ACK_NAK_HEADER WITH (NOLOCK)
WHERE AHR_FLAG IS NULL
OR AHR_FLAG = 0
DECLARE #varRecordCount INT
SELECT #varRecordCount = Count(1)
FROM #tmpANHdr
SET #varIndex = 1
IF( #varIndex <= #varRecordCount )
BEGIN
PRINT 'hi'
END
You could put a primary key (index) on the identity column and that might make it perform better:
DECLARE #tmpANHdr TABLE (
hdrId INT IDENTITY PRIMARY KEY,
CBFId NVARCHAR(32),
ACPT_REJ_FLAG NVARCHAR(8),
PROC_FILE_NAME NVARCHAR(50))
Also, do you end up using the table variable for anything else? Why not just do this?
DECLARE #varRecordCount INT
SELECT #varRecordCount = Count(TOP 100 *)
FROM FTS_ACK_NAK_HEADER WITH (NOLOCK)
WHERE AHR_FLAG IS NULL
OR AHR_FLAG = 0
A scan on a table variable with no more than 100 rows is likely to be pretty cheap anyway. The table scan operation might be costed at 100% of the plan for the statement but that is 100% of a pretty small number.
You can avoid this cost entirely though by just looking at ##rowcount after the insert as below.
DECLARE #varRecordCount INT;
INSERT INTO #tmpANHdr
SELECT TOP 100 AHR_CB_FTS_FILE_ID,
AHR_ACCT_REJ_FLAG,
AHR_PROC_FILE_NAME
FROM FTS_ACK_NAK_HEADER WITH (NOLOCK)
WHERE AHR_FLAG IS NULL
OR AHR_FLAG = 0
SET #varRecordCount = ##ROWCOUNT