MSSQL Procedure - Performance considerations when updating a massive table - sql-server

Question

I have two tables
existing_bactria (may contain millions of rows)
new_bactria (may contain millions of rows)
sample tables:
CREATE TABLE [dbo].[existing_bacteria](
[bacteria_name] [nchar](10) NULL,
[bacteria_type] [nchar](10) NULL,
[bacteria_sub_type] [nchar](10) NULL,
[bacteria_size] [nchar](10) NULL,
[bacteria_family] [nchar](10) NULL,
[bacteria_discovery_year] [date] NOT NULL
)
CREATE TABLE [dbo].[new_bacteria](
[existing_bacteria_name] [nchar](10) NULL,
[bacteria_type] [nchar](10) NULL,
[bacteria_sub_type] [nchar](10) NULL,
[bacteria_size] [nchar](10) NULL,
[bacteria_family] [nchar](10) NULL,
[bacteria_discovery_year] [date] NOT NULL
)
I need to create a stored proc to update new_bactria table with a possible match from existing_bactria (update field new_bactria.existing_bacteria_name
By finding a match on the other fields from [existing_bacteria] (assuming only single record in existing_bacteria)
Since the tables are massive (millions of records each) I would like your opinion on how to go about the solution, here is what I got so far:
Solution 1:
the obvious solution is to fetch all into a cursor and iterate over the results and update existing_bacteria
But since there are million records - its not an optimal solution
-- pseudo code
db_cursor as select * from new_bacteria
OPEN db_cursor
FETCH NEXT FROM db_cursor INTO #row
WHILE ##FETCH_STATUS = 0
BEGIN
IF EXISTS (
SELECT
#bacteria_name = [bacteria_name]
,#bacteria_type = [bacteria_type]
,#bacteria_size = [bacteria_size]
FROM [dbo].[existing_bacteria]
where [bacteria_type] = #row.[bacteria_type] and #row.[bacteria_size] = [bacteria_size]
)
BEGIN
PRINT 'update new_bacteria.existing_bacteria_name with [bacteria_name] we found.';
END
-- go to next record
FETCH NEXT FROM db_cursor INTO #name
END
Solution 2:
solution2 is to Join both tables in the mssql procedure
and iterate on the results but this is also
-- pseudo code
select * from [new_bacteria]
inner join [existing_bacteria]
on [new_bacteria].bacteria_size = [existing_bacteria].bacteria_size
and [new_bacteria].bacteria_family = [existing_bacteria].bacteria_family
for each result update [existing_bacteria]
I am sure this is not an optimal because of the table size and the iteration
Solution 3:
solution3 is to let the db handle the data and update the tables directly using inner Join:
-- pseudo code
UPDATE R
SET R.existing_bacteria_name = p.[bacteria_name]
FROM [new_bacteria] AS R
inner join [existing_bacteria] P
on R.bacteria_size = P.bacteria_size
and R.bacteria_family = P.bacteria_family
I am not sure about this solution.

Answer 1

Based on your pseudo code, I'd go with solution 3 because it is a set based operation and should be much quicker than using a cursor or other loop.
If you are having issues with performance with solution 3...
and you don't have indexes on those tables, particularly those columns you are using to join the two tables, creating those would help.
create unique index uix_new_bacteria_bacteria_size_bacteria_family
on [new_bacteria] (bacteria_size,bacteria_family);
create unique index uix_existing_bacteria_bacteria_size_bacteria_family
on [existing_bacteria] (bacteria_size,bacteria_family) include (bacteria_name);
and then try:
update r
set r.existing_bacteria_name = p.[bacteria_name]
from [new_bacteria] AS R
inner join [existing_bacteria] P on R.bacteria_size = P.bacteria_size
and R.bacteria_family = P.bacteria_family;
Updating a few million rows should not be a problem with the right indexes.
This section is no longer relevant after an update to the question
Another issue possibly exists in that if bacteria_size and bacteria_family are not unique sets, you could have multiple matches.
(since they are nullable I would imagine they aren't unique unless you're using a filtered index)
In that case, before moving forward, I'd create a table to investigate multiple matches like this:
create table [dbo].[new_and_existing_bacteria_matches](
[existing_bacteria_name] [nchar](10) not null,
rn int not null,
[bacteria_type] [nchar](10) null,
[bacteria_sub_type] [nchar](10) null,
[bacteria_size] [nchar](10) null,
[bacteria_family] [nchar](10) null,
[bacteria_discovery_year] [date] not null,
constraint pk_new_and_existing primary key clustered ([existing_bacteria_name], rn)
);
insert into [new_and_existing_bacteria_matches]
([existing_bacteria_name],rn,[bacteria_type],[bacteria_sub_type],[bacteria_size],[bacteria_family],[bacteria_discovery_year])
select
e.[existing_bacteria_name]
, rn = row_number() over (partition by e.[existing_bacteria_name] order by n.[bacteria_type], n.[bacteria_sub_type])
, n.[bacteria_type]
, n.[bacteria_sub_type]
, n.[bacteria_size]
, n.[bacteria_family]
, n.[bacteria_discovery_year]
from [new_bacteria] as n
inner join [existing_bacteria] e on n.bacteria_size = e.bacteria_size
and n.bacteria_family = e.bacteria_family;
-- and query multiple matches with something like this:
select *
from [new_and_existing_bacteria_matches] n
where exists (
select 1
from [new_and_existing_bacteria_matches] i
where i.[existing_bacteria_name]=n.[existing_bacteria_name]
and rn>1
);

Answer 2

On the subject of performance I'd look at:
The "Recovery Model" of the database, if your DBA says you can have it in "simple mode" then do it, you want to have as little logging as possible.
Consider Disabling some Indexes on the TARGET table, and then rebuilding them when you've finished. On large scale operations the modifications to the index will lead to extra logging, and the manipulation of the index will take up space in your Buffer Pool.
Can you convert the NCHAR to CHAR, it will require less storage space consequently reducing IO, freeing up buffer space and reducing Logging.
If your target table has no Clustered index then try activating 'TraceFlag 610' (warning this is an Instance-wide setting so talk to your DBA)
If your environment allows it, the use of the TABLOCKX hint can remove locking overhead and also help meet the criteria for reduced logging.
For anyone who has to perform Bulk Inserts or Large scale updates, this white paper from Microsoft is a valuable read:
You can try a MERGE statement. It will perform the operation in a single pass of the data. (the problem with a merge is that it tries to do everything in one Transaction and you can end up with an unwanted Spool in the execution plan. I'd then move towards a Batch process looping through maybe 100,000 records at a time.)
(It will need some minor changes to suit your column matching/update requirements)
MERGE [dbo].[new_bacteria] T --TARGET TABLE
USING [dbo].[existing_bacteria] S --SOURCE TABLE
ON
S.[bacteria_name] = T.[existing_bacteria_name] --FILEDS TO MATCH ON
AND S.[bacteria_type] = T.[bacteria_type]
WHEN MATCHED
AND
ISNULL(T.[bacteria_sub_type],'') <> ISNULL(S.[bacteria_sub_type],'') --FIELDS WHERE YOURE LOOKING FOR A CHANGE
OR ISNULL(T.[bacteria_size],'') <> ISNULL(S.[bacteria_size],'')
THEN --UPDATE RECORDS THAT HAVE CHANGED
UPDATE
SET T.[bacteria_sub_type] = S.[bacteria_sub_type]
WHEN NOT MATCHED BY TARGET THEN --ANY NEW RECORDS IN THE SOURCE TABLE WILL BE INSERTED
INSERT(
[existing_bacteria_name],
[bacteria_type],
[bacteria_sub_type],
[bacteria_size],
[bacteria_family],
[bacteria_discovery_year]
)
VALUES(
s.[bacteria_name],
s.[bacteria_type],
s.[bacteria_sub_type],
s.[bacteria_size],
s.[bacteria_family],
s.[bacteria_discovery_year]
);
If the Single MERGE is too much for your system to handle, here's a method for embedding it in a loop that updates large batches. You can modify the batch size to match your Server's capabilities.
It works by using a couple of staging tables that ensure if anything goes wrong (i.e. server agent restart), the process can continue from where it left off. (If you have any question please ask).
--CAPTURE WHAT HAS CHANGED SINCE THE LAST TIME THE SP WAS RUN
--EXCEPT is a usefull command because it can compare NULLS, this removes the need for ISNULL or COALESCE
INSERT INTO [dbo].[existing_bacteria_changes]
SELECT
*
FROM
[dbo].[existing_bacteria]
EXCEPT
SELECT
*
FROM
[dbo].[new_bacteria]
--RUN FROM THIS POINT IN THE EVENT OF A FAILURE
DECLARE #R INT = 1
DECLARE #Batch INT = 100000
WHILE #R > 0
BEGIN
BEGIN TRAN --CARRY OUT A TRANSACTION WITH A SUBSET OF DATA
--USE DELETE WITH OUTPUT TO MOVE A BATCH OF RECORDS INTO A HOLDING AREA.
--The holding area will provide a rollback point so if the job fails at any point it will restart from where it last was.
DELETE TOP (#Batch)
FROM [dbo].[existing_bacteria_changes]
OUTPUT DELETED.* INTO [dbo].[existing_bacteria_Batch]
##ROWCOUNT
--LOG THE NUMBER OF RECORDS IN THE UPDATE SET, THIS WILL ENSURE THE NEXT ITTERATION
SET #R = ISNULL(##ROWCOUNT,0)
--RUN THE MERGE STATEMENT WITH THE SUBSET OF UPDATES
MERGE [dbo].[new_bacteria] T --TARGET TABLE
USING [dbo].[existing_bacteria_Batch] S --SOURCE TABLE
ON
S.[bacteria_name] = T.[existing_bacteria_name] --FILEDS TO MATCH ON
AND S.[bacteria_type] = T.[bacteria_type]
WHEN MATCHED
AND
ISNULL(T.[bacteria_sub_type],'') <> ISNULL(S.[bacteria_sub_type],'') --FIELDS WHERE YOURE LOOKING FOR A CHANGE
OR ISNULL(T.[bacteria_size],'') <> ISNULL(S.[bacteria_size],'')
THEN --UPDATE RECORDS THAT HAVE CHANGED
UPDATE
SET T.[bacteria_sub_type] = S.[bacteria_sub_type]
WHEN NOT MATCHED BY TARGET THEN --ANY NEW RECORDS IN THE SOURCE TABLE WILL BE INSERTED
INSERT(
[existing_bacteria_name],
[bacteria_type],
[bacteria_sub_type],
[bacteria_size],
[bacteria_family],
[bacteria_discovery_year]
)
VALUES(
s.[bacteria_name],
s.[bacteria_type],
s.[bacteria_sub_type],
s.[bacteria_size],
s.[bacteria_family],
s.[bacteria_discovery_year]
);
COMMIT;
--No point in logging this action
TRUNCATE [dbo].[existing_bacteria_Batch]
END

Answer 3

Definitely option 3. SET-based always wins from anything loopy.
That said, the biggest 'risk' might be that the amount of updated data 'overwhelms' your machine. More specific, it could happen that the transaction becomes so big that the system takes forever to finish it. To avoid this you could try splitting the one big UPDATE into multiple smaller UPDATEs and still work mostly set-based. Good indexing and knowing your data is key here.
For instance, starting from
UPDATE R
SET R.existing_bacteria_name = p.[bacteria_name]
FROM [new_bacteria] AS R
INNER JOIN [existing_bacteria] P
ON R.bacteria_size = P.bacteria_size
AND R.bacteria_family = P.bacteria_family
You might try 'chunk' the (target) table into smaller parts. E.g. by making a loop over the bacteria_discovery_year field, assuming that said column splits the table into e.g. 50 more or less equally sized parts. (BTW: I'm no biologist so I might be totally wrong there =)
You'd then get something along the lines of:
DECLARE #c_bacteria_discovery_year date
DECLARE year_loop CURSOR LOCAL STATIC
FOR SELECT DISTINCT bacteria_discovery_year
FROM [new_bacteria]
ORDER BY bacteria_discovery_year
OPEN year_loop
FETCH NEXT FROM year_loop INTO #c_bacteria_discovery_year
WHILE ##FETCH_STATUS = 0
BEGIN
UPDATE R
SET R.existing_bacteria_name = p.[bacteria_name]
FROM [new_bacteria] AS R
INNER JOIN [existing_bacteria] P
ON R.bacteria_size = P.bacteria_size
AND R.bacteria_family = P.bacteria_family
WHERE R.bacteria_discovery_year = #c_bacteria_discovery_year
FETCH NEXT FROM year_loop INTO #c_bacteria_discovery_year
END
CLOSE year_loop
DEALLOCATE year_loop
Some remarks:
Like I said, I don't know the distribution of the bacteria_discovery_year values, if 3 years make up 95% of the data it might not be such a great choice.
This will only work if there is an index on the bacteria_discovery_year column, preferably with bacteria_size and bacteria_family included.
You could add some PRINT inside the loop to see the progress and rows affected... it won't speed up anything, but it feels better if you know it's doing something =)
All in all, don't overdo it, if you split it into too many small chunks you'll end up with something that takes forever too.
PS: in any case you'll also need an index on the 'source' table that indexes the bacteria_size and bacteria_family column, preferably including the bacteria_name if the latter is not the (clustered) PK of the table.