Optimaze SQL Database table for multiple writes single read - sql-server

I am developing a feature that will be used as processes progress monitor.
I will span 40-50 threads that might take several minutes or even hours to finish and they will update it's status to a datatable.
From the web app i will create a polling mechanism that will read the processes status using one read every 0.5 sec.
I need to optimize the table for multiple writes at a sec and one read per 0.5 sec. I don't care if I read a dirty state since it's just for monitoring the process, it's not that critical.
This is the table I am using
CREATE TABLE [cmn].[ProcessProgress]
(
[id] [bigint] NOT NULL,
[status] [smallint] NOT NULL,
[step] [int] NOT NULL,
[max_step] [int] NOT NULL,
CONSTRAINT [PK_ProcessProgress] PRIMARY KEY CLUSTERED
(
[id] ASC
) WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
)
GO
And the select with (NOLOCK) I suppose I have to use
SELECT * FROM [cmn].[ProcessProgress] (NOLOCK)
Do I have to use a transaction with a special ISOLATION LEVEL or (nolock) is sufficient? Or (nolock) will make thinks worse?
Can you suggest what would be the most optimized solution for this problem?

Just set the READ_COMMITTED_SNAPSHOT option on your database, and the readers and writers will never conflict. Instead they will use Row Versioning:
alter database current set read_committed_snapshot on
In addition to increasing the concurrency and scalability of your application by eliminating blocking between readers and writers, it eliminates many deadlocks, and removes the incentive to perform dirty reads.

Here is an example (hope id did not make too many mistakes). test code at the bottom is what would be executed by each thread.
if object_id('ProcessProgress') is not null
drop table ProcessProgress
Go
CREATE TABLE [ProcessProgress]
(
[id] [bigint] NOT NULL IDENTITY(1,1), --added identity to shorten sample dev
[status] [smallint] NOT NULL, --1 -ready,2-inprogress, 3-complete
[step] [int] NOT NULL,
[max_step] [int] NOT NULL,
CONSTRAINT [PK_ProcessProgress] PRIMARY KEY CLUSTERED
(
[id] ASC
) WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
)
GO
SET NOCOUNT ON
INSERT INTO [ProcessProgress]
(status, step, max_step)
VALUES
(1, 1, 1)
GO 1000
Go
IF OBJECT_ID('StartWork') is not null drop proc StartWork
GO
CREATE PROC StartWork
AS
BEGIN
SET TRANSACTION ISOLATION LEVEL READ COMMITTED
BEGIN TRAN
;WITH TODO
AS (
SELECT TOP 1 Id, [status] from ProcessProgress WITH (ROWLOCK, READPAST) WHERE [status] = 1 --ready
)
UPDATE TODO
SET [status] = 2 --InProgress
OUTPUT inserted.id
COMMIT
END
GO
IF OBJECT_ID('FinishWork') is not null drop proc FinishWork
GO
CREATE PROC FinishWork
#id int
AS
BEGIN
SET TRANSACTION ISOLATION LEVEL READ COMMITTED
BEGIN TRAN
Update ProcessProgress
SET [Status] = 3 --finished
WHERE
id = #id
COMMIT
END
GO
/*tester*/
declare #idout table (id int)
insert into #idout exec StartWork
declare #idin int = (Select top 1 id from #idout)
exec FinishWork #idin

Related

Reset IDENTITY column whitout deleting all rows

I have the following table
CREATE TABLE [dbo].[MyTable](
[Name] NVARCHAR(200) NOT NULL,
[Surname] NVARCHAR(200) NOT NULL,
[Permanent] [bit] NULL,
[Idx] [bigint] IDENTITY(1,1) NOT NULL
CONSTRAINT [MyTable] PRIMARY KEY CLUSTERED
(
[Idx] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
that contains 1000 rows where [Permanent]=1 and 500.000 rows with [Permanent] NULL
In addition, I have a stored procedure (called MySP) that do the following steps:
DELETE FROM [MyTable] WHERE [Permanent] IS NULL
(lots of calculations)
INSERT results of point 2 into [MyTable]
MySP runs every day so the number of [Idx] increase very quickly as every day 500.000 rows are deleted and inserted.
My target is, during the execution of the SP, to reset the value of column [Idx] to mantain the max number of [Idx] lower possible.
1st try
I have tried to update with the following query at the end of MySP but the system gives me a (correct) error.
UPDATE A
SET A.[Idx]=B.[Idx_new]
FROM [dbo].[MyTable] A
INNER JOIN (
SELECT [Idx],ROW_NUMBER() OVER (ORDER BY [Idx]) AS [Idx_new]
FROM [dbo].[MyTable]
) B
ON A.[Idx]=B.[Idx]
2nd try
After reading the following two questions/answers
Reset identity seed after deleting records in SQL Server
How to update Identity Column in SQL Server?
I have add the following at the end of MySP
DBCC CHECKIDENT ('MyTable', RESEED, 1);
but also this doesn't not work as in [MyTable], differently from the situation of both quesions, remain some rows, so the is a concrete risk that [Idx] is not unique and that's not good as [Idx] is my primary key.
How could I reset the identity column value and also the rows that still remains into [MyTable]?
Using #Killer Queen suggest, I have solved using this snippet of code, which find the MAX([Idx]) of MyTable after the DELETE and the reeseed the identity before the new INSERT.
It works because the rows with [Permanent]=1 are the first rows inserted in the table so their [Idx] values start from 1 and are very low.
DELETE
FROM [MyTable]
WHERE [Permanent]!=1 OR [Permanent] IS NULL
DECLARE #MaxIdx As bigint
SET #MaxIdx = (SELECT ISNULL(MAX([Idx]),0) FROM [MyTable])
DBCC CHECKIDENT ('MyTable', RESEED, #MaxIdx);

How to find who deleted data from my table in sql server

I have some table in my table which is accessible to many. Some data is missing in my table now. How can I find who deleted those rows from that table.
You can use ApexSQL Log to fully investigate operations executed against your table. The database needs to be in the full recovery model, so the information on past operations is available inside the transaction log file for ApexSQL Log to read it. Once the tool analyzes your t-log, you will be able to see the time the operation began and ended, the operation type, the schema and object name of the object affected, the name of the user who executed the operation, and more. For UPDATEs, you’ll even be able to see the old and the new value of the updated fields.
There are several guides on this here https://solutioncenter.apexsql.com/apexsql-log-solutions-table-of-contents/
Furthermore, you can even use ApexSQL Log to rollback those transactions if you need to. It will simply 'Undo' them and rollback changes back to their original state.
You can find deleted data's UserName by following little snippet :
DECLARE #TableName sysname
SET #TableName = 'dbo.t1_new' --INPUT TABLE NAME
SELECT
u.[name] AS UserName
, l.[Begin Time] AS TransactionStartTime
FROM
fn_dblog(NULL, NULL) l
INNER JOIN
(
SELECT
[Transaction ID]
FROM
fn_dblog(NULL, NULL)
WHERE
AllocUnitName LIKE #TableName + '%'
AND
Operation = 'LOP_DELETE_ROWS'
) deletes
ON deletes.[Transaction ID] = l.[Transaction ID]
INNER JOIN
sysusers u
ON u.[sid] = l.[Transaction SID]
source : dba.stackexchange
(I don't recall who posted it)
Unfortunately, you can't see deleted records if you don't keep them somewhere yourself.
If you want to track this type of interventions, you should not really delete your records.
Instead, you should create a some more fields on your table.
Here is an exemple :
SET QUOTED_IDENTIFIER ON
GO
SET ANSI_PADDING ON
GO
CREATE TABLE [dbo].[Person](
[Pers_ID] [int] IDENTITY(1,1) NOT NULL,
[Pers_CompanyID] [int] NULL,
[Pers_FirstName] [nvarchar](50) NULL,
[Pers_LastName] [nvarchar](50) NULL,
[Pers_CreatedBy] [int] NULL,
[Pers_CreatedDate] [datetime] NULL,
[Pers_UpdatedBy] [int] NULL,
[Pers_UpdatedDate] [datetime] NULL,
[Pers_Deleted] [bit] NULL,
CONSTRAINT [PK_Person] PRIMARY KEY CLUSTERED
(
[Pers_ID] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
GO
SET ANSI_PADDING OFF
GO
When the user creates a record, you can set CreatedBy = UserID, CreatedDate = CurrentDate,
While updating a record, UpdatedBy = UserID, UpdatedDate = CurrentDate
And deleting, Deleted = True, UpdatedBy = UserID, UpdatedDate = CurrentDate.
And in your code, in all queries you should add the condition Deleted = null.
Thus, you can track who created, updated or deleted a record.

MS SQL Check for duplicate in two fields

I am trying create a trigger that will check if the Author already exist in a table based on a combination of their first and last name. From what Ive been reading this trigger should work, but when I try to insert any new author into the table it gives the "Author exists in table already!" error even though I am inserting an author that does not exist in the table.
Here is the trigger
USE [WebsiteDB]
GO
CREATE TRIGGER [dbo].[tr_AuthorExists] ON [dbo].[Authors]
AFTER INSERT
AS
if exists ( select * from Authors
inner join inserted i on i.author_fname=Authors.author_fname AND i.author_lname=Authors.author_lname)
begin
rollback
RAISERROR ('Author exists in table already!', 16, 1);
End
Here is the table
CREATE TABLE [dbo].[Authors](
[author_id] [int] IDENTITY(1,1) NOT NULL,
[author_fname] [nvarchar](50) NOT NULL,
[author_lname] [nvarchar](50) NOT NULL,
[author_middle] [nvarchar](50) NULL,
CONSTRAINT [PK_Authors] PRIMARY KEY CLUSTERED
(
[author_id] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
Any assistance would be appreciated!
You will need to do this as an INSTEAD of trigger. This also means you need to actually perform the insert inside the trigger. Something along these lines.
CREATE TRIGGER [dbo].[tr_AuthorExists] ON [dbo].[Authors]
instead of insert
AS
set nocount on;
if exists
(
select * from Authors a
inner join inserted i on i.author_fname = a.author_fname AND i.author_lname = a.author_lname
)
begin
rollback
RAISERROR ('Author exists in table already!', 16, 1);
End
else
insert Authors
select i.author_fname
, i.author_lname
, i.author_middle
from inserted i

How to speed up Xpath performance in SQL Server when searching for an element with specific text

I am supposed to remove whole rows and part of XML-documents from a table with an XML column based on a specific value in the XML column. However the table contains millions of rows and gets locked when I perform the operation. Currently it will take almost a week to clean it up, and the system is too critical to be taken offline for so long.
Are there any ways to optimize the xpath expressions in this script:
declare #slutdato datetime = '2012-03-01 00:00:00.000'
declare #startdato datetime = '2000-02-01 00:00:00.000'
declare #lev varchar(20) = 'suppliername'
declare #todelete varchar(10) = '~~~~~~~~~~'
CREATE TABLE #ids (selId int NOT NULL PRIMARY KEY)
INSERT into #ids
select id from dbo.proevesvar
WHERE leverandoer = #lev
and proevedato <= #slutdato
and proevedato >= #startdato
begin transaction /* delete whole rows */
delete from dbo.proevesvar
where id in (select selId from #ids)
and ProeveSvarXml.exist('/LaboratoryReport/LaboratoryResults/Result[Value=sql:variable(''#todelete'')]') = 1
and Proevesvarxml.exist('/LaboratoryReport/LaboratoryResults/Result[Value!=sql:variable(''#todelete'')]') = 0
commit
go
begin transaction /* delete single results */
UPDATE dbo.proevesvar SET ProeveSvarXml.modify('delete /LaboratoryReport/LaboratoryResults/Result[Value=sql:variable(''#todelete'')]')
where id in (select selId from #ids)
commit
go
The table definitions is:
CREATE TABLE [dbo].[ProeveSvar](
[ID] [int] IDENTITY(1,1) NOT NULL,
[CPRnr] [nchar](10) NOT NULL,
[ProeveDato] [datetime] NOT NULL,
[ProeveSvarXml] [xml] NOT NULL,
[Leverandoer] [nvarchar](50) NOT NULL,
[Proevenr] [nvarchar](50) NOT NULL,
[Lokationsnr] [nchar](13) NOT NULL,
[Modtaget] [datetime] NOT NULL,
CONSTRAINT [PK_ProeveSvar] PRIMARY KEY CLUSTERED
(
[ID] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY],
CONSTRAINT [IX_ProeveSvar_1] UNIQUE NONCLUSTERED
(
[CPRnr] ASC,
[Lokationsnr] ASC,
[Proevenr] ASC,
[ProeveDato] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
The first insert statement is very fast. I believe I can handle the locking by committing 50 rows at a time, so other requests can be handled in between my transactions.
The total number of rows for this supplier is about 5.5 million and the total rowcount in the table is around 13 million.
I've not really used xpath within SQL server before, but something which stands out is that you're doing lots of reads and writes in the same command (in the second statement). If possible, change your queries to..
CREATE TABLE #ids (selId int NOT NULL PRIMARY KEY)
INSERT into #ids
select id from dbo.proevesvar
WHERE leverandoer = #lev
and proevedato <= #slutdato
and proevedato >= #startdato
and ProeveSvarXml.exist('/LaboratoryReport/LaboratoryResults/Result[Value=sql:variable(''#todelete'')]') = 1
and Proevesvarxml.exist('/LaboratoryReport/LaboratoryResults/Result[Value!=sql:variable(''#todelete'')]') = 0
begin transaction /* delete whole rows */
delete from dbo.proevesvar
where id in (select selId from #ids)
This means that the first query will only create the new temporary table, and not write anything back, which will take slightly longer than your original, but the key thing is that your second query will ONLY be deleting records based on what's in your temporary table.
What you'll probably find is because it's deleting records, it's constantly re-building indices, and causing the reads to also be slower.
I'd also delete/disable any indices/constraints that don't actually help your query run.
Also, you're creating your clustered primary key on the ID, which isn't always the best thing to do. Especially if you're doing lots of date scans.
Can you also view the estimated execution plan for the top query, it would be interesting to see the order in which it checks the conditions. If it's doing the date first, then that's fine, but if it's doing the xpath before it checks the date, you might have to separte it into 3 queries, or add a new clustered index on 'proevedato,id'. This should force the query to only run the xpath for records which actually match the date.
Hope this helps.

Trigger Not Putting Data in History Table

I have the following trigger (along with others on similar tables) that sometimes fails to put data into the historic table. It should put data into a historic table exactly as it's inserted/updated and stamped with a date.
CREATE TRIGGER [dbo].[trig_UpdateHistoricProductCustomFields]
ON [dbo].[productCustomFields]
AFTER UPDATE,INSERT
AS
BEGIN
IF ((UPDATE(data)))
BEGIN
SET NOCOUNT ON;
DECLARE #date bigint
SET #date = datepart(yyyy,getdate())*10000000000+datepart(mm,getdate())*100000000+datepart(dd,getdate())*1000000+datepart(hh,getdate())*10000+datepart(mi,getdate())*100+datepart(ss,getdate())
INSERT INTO historicProductCustomFields (productId,customFieldNumber,data,effectiveDate) (SELECT productId,customFieldNumber,data,#date from inserted)
END
END
Schema:
CREATE TABLE [dbo].[productCustomFields](
[id] [int] IDENTITY(1,1) NOT NULL,
[productId] [int] NOT NULL,
[customFieldNumber] [int] NOT NULL,
[data] [varchar](50) NULL,
CONSTRAINT [PK_productCustomFields] PRIMARY KEY CLUSTERED
(
[id] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
CREATE TABLE [dbo].[historicProductCustomFields](
[id] [bigint] IDENTITY(1,1) NOT NULL,
[productId] [int] NOT NULL,
[customFieldNumber] [int] NOT NULL,
[data] [varchar](50) NULL,
[effectiveDate] [bigint] NOT NULL,
CONSTRAINT [PK_historicProductCustomFields] PRIMARY KEY CLUSTERED
(
[id] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
I insert and update only on one record at a time on the productCustomFields table. It seems to work 99% of the time and hard to test for failure. Can anyone shed some light on what I may be doing wrong or better practices for this type of trigger?
Environment is Sql Server Express 2005. I haven't rolled out the service pack yet for sql server either for this particular client.
I think the right way to solve this is keep a TRY CATCH block when inserting into the dbo.historicProductCustomFields table and write the errors into a custom errorlog table. From there it is easy to track this down.
I also see a PK on the historicProductCustomFields table but if you insert and update a given record in ProductCustomFields table then won't you get primary key violations on the historicProductCustomFields table?
You should schema qualify your table that you are inserting into.
You should check to ensure that there are not multiple triggers on the table, as if there are, only 1 trigger for that type of trigger will fire and if there are 2 defined, they are run in random order. In other words, 2 triggers of the same type (AFTER INSERT) then one would fire and the other would not, but you don't necessary have control as to which will fire.
try to use this trigger. i just give you example try to write trigger with this trigger.
create TRIGGER [dbo].[insert_Assets_Tran]
ON [dbo].[AssetMaster]
AFTER INSERT , UPDATE
AS BEGIN
DECLARE #isnum TINYINT;
SELECT #isnum = COUNT(*) FROM inserted;
IF (#isnum = 1)
INSERT INTO AssetTransaction
select [AssetId],[Brandname],[SrNo],[Modelno],[Processor],[Ram],[Hdd],[Display],[Os],[Office],[Purchasedt]
,[Expirydt],[Vendor],[VendorAMC],[Typename],[LocationName],[Empid],[CreatedBy],[CreatedOn],[ModifiedBy]
,[ModifiedOn],[Remark],[AssetStatus],[Category],[Oylstartdt],[Oylenddt],[Configuration]
,[AStatus],[Tassign]
FROM inserted;
ELSE
RAISERROR('some fields not supplied', 16, 1)
WITH SETERROR;
END

Resources