Batches over groups - sql-server

Question

I need to process rows in a table in batches of not less than N rows. Each batch needs to contain an entire group of rows (group is just another column) i.e. when I select top N rows from the table for processing, I need to extend that N to cover the last group in the batch rather than splitting the last group between batches.
Sample data:
CREATE TABLE test01 (id INT PRIMARY KEY CLUSTERED IDENTITY(1, 1) NOT NULL
, person_name NVARCHAR(100)
, person_surname NVARCHAR(100)
, person_group_code CHAR(2) NOT NULL);
INSERT INTO
dbo.test01 (person_name
, person_surname
, person_group_code)
VALUES
('n1', 's1', 'g1')
, ('n2', 's2', 'g1')
, ('n3', 's3', 'g1')
, ('n4', 's4', 'g1')
, ('n5', 's5', 'g2')
, ('n6', 's6', 'g2')
, ('n7', 's7', 'g2')
, ('n8', 's8', 'g2')
, ('n9', 's9', 'g2')
, ('n10', 's10', 'g2')
, ('n11', 's11', 'g3')
, ('n12', 's12', 'g3')
, ('n13', 's13', 'g3')
, ('n14', 's14', 'g3');
My current attempt:
DECLARE #batch_start INT = 1
, #batch_size INT = 5;
DECLARE #max_id INT = (SELECT MAX(id) FROM dbo.test01);
WHILE #batch_start <= #max_id
BEGIN
SELECT *
FROM dbo.test01
WHERE id BETWEEN #batch_start AND #batch_start + #batch_size - 1;
SELECT #batch_start += #batch_size;
END;
DROP TABLE dbo.test01;
In the example above, I am splitting the 14 rows into 3 batches: 5 rows in batch #1, another 5 rows in batch #2 and then 4 rows in the final batch.
The first batch (id from 1 to 5) covers only fraction of the 'g2' group so I need to extend this batch to cover rows 1-10 (I need to process the entire g2 in a single batch).
(by the way, I don't mind batch upsizing - I need to make sure I cover at least one full group per batch).
The result would be that batch #1 would cover groups g1 and g2 (10 rows) then batch #2 would cover group g3 (4 rows) and there would be no batch #3 at all.
Now, the table is billions of rows and batch sizes are around 50K-100K each so I need a solution that performs well.
Any hints on how to approach this with minimal performance hit?

Answer 1

The first thing I've noticed is that your current code assumes no gaps in the identity column - However that is a mistake. An identity column may (and often do) have gaps in the numbers - so the first thing you want to do is use row_number() over(order by id) to get a continuous running number for all your records.
The second thing I've added as a column that gave a numeric id for each group ordered by the same order as the identity column - using a well-known technique for solving gaps and islands problems.
I've used a table variable to store this data for each id on the source table for the purpose if this demonstration, but you might want to use a temporary table and add indexes on the relevant columns to improve performance.
I've also renamed your #batch_size variable to #batch_min_size and added a few other variables.
So here is the table variable I've used:
DECLARE #Helper As Table (Id int, Rn int, GroupId int)
INSERT INTO #Helper (Id, Rn, GroupId)
SELECT Id,
ROW_NUMBER() OVER(ORDER BY ID) As Rn,
ROW_NUMBER() OVER(ORDER BY ID) -
ROW_NUMBER() OVER(PARTITION BY person_group_code ORDER BY ID) As GroupId
FROM dbo.test01
This is the content of this table:
Id Rn GroupId
1 1 0
2 2 0
3 3 0
4 4 0
5 5 4
6 6 4
7 7 4
8 8 4
9 9 4
10 10 4
11 11 10
12 12 10
13 13 10
14 14 10
I've used a while loop to do the batches.
In the loop, I've used this table to calculate the first and last id of each batch, as well as the last row number of the batch.
Then all I had to do was to use the first and last id in the where clause of the original table:
DECLARE #batch_min_size int = 10
, #batch_end int = 0
, #batch_start int
, #first_id_of_batch int
, #last_id_of_batch int
, #total_row_count int;
SELECT #total_row_count = COUNT(*) FROM #test01
WHILE #batch_end < #total_row_count
BEGIN
SELECT #batch_start = #batch_end + 1;
SELECT #batch_end = MAX(Rn)
, #first_id_of_batch = MIN(Id)
, #last_id_of_batch = MAX(Id)
FROM #Helper
WHERE Rn >= #batch_start
AND GroupId <=
(
SELECT MAX(GroupId)
FROM #Helper
WHERE Rn <= #batch_start + #batch_min_size - 1
)
SELECT id, person_name, person_surname, person_group_code
FROM dbo.test01
WHERE Id >= #first_id_of_batch
AND Id <= #last_id_of_batch
END
See a live demo on rextester.

Answer 2

See if below helps:
CREATE TABLE #Temp(g_record_count int, groupname varchar(50) )
insert into #Temp(g_record_count,groupname) SELECT MAX(id),person_group_code FROM dbo.test01 group by person_group_code
After this loop through this temporary table :
DECLARE #rec_per_batch INT = 1
WHILE #batch_start <= #max_id
BEGIN
select min(g_record_count) into #rec_per_batch from #temp where g_record_count>=#batch_size * #batch_start;
SELECT *
FROM dbo.test01
WHERE id BETWEEN #batch_start AND #rec_per_batch;
SELECT #batch_start += #batch_size;
END;