How to do this in SQL Server query instead of function? - sql-server

Question

I have a table that has a string in one of its columns.
My table look like this:
RowCnt
Lvl
TargetID
Codes
1000
1
0
1,1,0,1,0,1,...,1,0,0,0,0
1000
1
1
0,0,1,0,1,0,...,0,1,1,1,1
1000
1
2
1,0,0,0,1,1,...,0,0,0,0,0
1000
1
3
0,1,1,1,0,1,...,1,1,1,1,1
1000
1
4
1,1,0,0,1,0,...,0,0,1,0,0
1000
2
0
0,0,1,1,0,1,...,0,1,0,1,1
1000
2
1
0,1,0,1,1,1,...,1,1,1,1,0
1000
2
2
0,0,0,0,0,1,...,0,0,0,0,1
1500
1
0
1,1,1,1,1,0,...,1,1,1,1,0
1500
1
1
1,0,0,0,0,1,...,0,0,0,0,1
I have to compare each line with each line and see how many of digits differ in the Codes column.
So the first record 1,1,0,1,0,1,...,1,0,0,0,0 will be compared with the 2nd 0,0,1,0,1,0,...,0,1,1,1,1 and find that there are 14 out 328 digits different, then compare with the 3rd record 1,0,0,0,1,1,...,0,0,0,0,0 and find 29 / 328 till do all records
Then compare the 2nd record with the 3rd then the 4th and do on till do them all
My table has around 2000 records and I assume that would take around 4 million operations.
I have built a function to do the comparison
ALTER FUNCTION [dbo].[Sim]
(#x varchar(max),
#y varchar(max))
RETURNS decimal(18,10)
AS
BEGIN
DECLARE #Xt AS TABLE (id int identity, x int)
DECLARE #Yt AS TABLE (id int identity, y int)
DECLARE #Match int
DECLARE #All int
INSERT INTO #Xt (x)
SELECT value FROM STRING_SPLIT(#x, ',')
INSERT INTO #Yt (y)
SELECT value FROM STRING_SPLIT(#y, ',')
SELECT #Match = COUNT(*)
FROM #Xt xx
INNER JOIN #Yt yy ON xx.id = yy.id AND xx.x = yy.y
SELECT #All = COUNT(*)
FROM #Xt xx
INNER JOIN #Yt yy ON xx.id = yy.id
RETURN 1.0 * #Match / #All
END
and my query is like this
WITH Y AS (
select a.RowCnt, a.Lvl, a.TargetID a_TargetID, b.targetid b_TargetID, a.codes a_codes, b.codes b_codes, dbo.sim(a.codes, b.codes) sim
from TargetsComp A inner join TargetsComp B
on a.RowCnt = b.RowCnt
and a.TargetID < b.TargetID
)
insert into TargetFilled (RowCnt, Lvl, a_TargetID, b_TargetID, a_codes, b_codes, sim)
SELECT RowCnt, Lvl, a_TargetID, b_TargetID, a_codes, b_codes, sim FROM Y ORDER BY RowCnt,Lvl, sim desc
but my method fills C drive and does not finish!!
Any better way?

Answer 1

A faster version of your function would be an inline Table Valued Function.
CREATE OR ALTER FUNCTION dbo.Similar (#x varchar(max), #y varchar(max))
RETURNS TABLE AS RETURN
SELECT COUNT(CASE WHEN xJ.value <> yJ.value THEN 1 END) * 1.0 / COUNT(*) AS Pct
FROM (
SELECT *,
ROW_NUMBER() OVER (ORDER BY (SELECT 1)) rn
FROM STRING_SPLIT(#x, ',')
) xJ
JOIN (
SELECT *,
ROW_NUMBER() OVER (ORDER BY (SELECT 1)) rn
FROM STRING_SPLIT(#y, ',')
) yJ ON yJ.rn = xJ.rn;
However, STRING_SPLIT with a row-number is not guaranteed to always return results in the actual order of the string. It may do it once, it may do it a million times, but there is always a chance the compiler could rearrange things. So instead you could use OPENJSON
CREATE OR ALTER FUNCTION dbo.Similar (#x varchar(max), #y varchar(max))
RETURNS TABLE AS RETURN
SELECT COUNT(CASE WHEN xJ.value <> yJ.value THEN 1 END) * 1.0 / COUNT(*) AS Pct
FROM OPENJSON('[' + #x + ']') xJ
JOIN OPENJSON('[' + #y + ']') yJ ON yJ.[key] = xJ.[key];
You would use it like this
WITH Y AS (
select
a.RowCnt,
a.Lvl,
a.TargetID a_TargetID,
b.targetid b_TargetID,
a.codes a_codes,
b.codes b_codes,
sim.Pct sim
from TargetsComp A
inner join TargetsComp B
on a.RowCnt = b.RowCnt
and a.TargetID < b.TargetID
CROSS APPLY dbo.sim(a.codes, b.codes) sim
)
insert into TargetFilled
(RowCnt, Lvl, a_TargetID, b_TargetID, a_codes, b_codes, sim)
SELECT RowCnt, Lvl, a_TargetID, b_TargetID, a_codes, b_codes, sim
FROM Y;
-- you may want to add
-- WHERE sim.Pct < 100
I have removed the ORDER BY from the insert as I don't think it's necessary.
You should index your table as follows
CLUSTERED INDEX ON TargetsComp (RowCnt, TargetID)