determine the number of intersecting records - snowflake-cloud-data-platform

Question

I have a set of objects that have a variety of attributes:
id
title
owner_id
25
The Name of the object
5000
31
Another name of a different object
5000
71
Yet another
5000
19
example title
6000
21
another example
6000
23
more example
6000
and a list of attributes for each object (this list isn't completely normalized, but could be if necessary)
object_id
attribute
25
Blue
25
Green
25
Black
31
Blue
31
Purple
71
White
19
Grey
19
Blue
21
Yellow
23
Pink
23
Grey
The goal is to calculate how many objects each owner has that overlap any attribtues with the other owners. So in the example, owner 5000 has 3 objects ( 25, 31, 71 ), and owner 6000 has 3 ( 19, 21, 23 ).
5000
object 25 is (blue | green | black ) and overlaps with 19 because it also has a blue attribute.
object 31 is (blue | purple | white ) and overlaps with 19 because it also has a blue attribute
object 71 is (white) and doesn't overlap with any of owner 6000's objects, so it is unique.
6000
object 19 is (grey | blue) and overlaps 25
object 21 is (yellow) and is unique
object 23 is (pink & grey) is unique because none of 5000s objects are pink or grey
so since it doesn't really matter if 1 or all of the other owners objects match, i can create a summary table for each owner that lists out all of that companies unique attributes, which would look like:
5000: ( blue | green | black | purple | white )
6000: ( grey | blue | yellow | pink )
so the goal would be some output like:
owner_a
owner_b
count_a
count_b
similarity_a_to_b
similarity_b_to_a
unique_a
unique_b
5000
6000
3
3
2
1
1
2
the challenge is that I'm dealing with several thousand owner_ids, with several million objects, which have 10s of millions of attributes, so I'm trying to figure out how to summarize the data so I could generate these types of metrics.

Answer 1

Well the SQL can be done via:
WITH objects (id, title, owner_id) AS (
SELECT * FROM VALUES
(25, 'The Name of the object', 5000),
(31, 'Another name of a different object', 5000),
(71, 'Yet another', 5000),
(19, 'example title', 6000),
(21, 'another example', 6000),
(23, 'more example', 6000)
), attributes (object_id, attribute) AS (
SELECT * FROM VALUES
(25, 'Blue'),
(25, 'Green'),
(25, 'Black'),
(31, 'Blue'),
(31, 'Purple'),
(71, 'White'),
(19, 'Grey'),
(19, 'Blue'),
(21, 'Yellow'),
(23, 'Pink'),
(23, 'Grey')
), obj_att AS (
SELECT o.*, a.*
FROM objects AS o
JOIN attributes AS a
ON o.id = a.object_id
), owner_att AS (
SELECT DISTINCT OWNER_ID, ATTRIBUTE
FROM obj_att
), dist_owners AS (
SELECT DISTINCT OWNER_ID
FROM objects
), owner_pairs AS (
SELECT a.owner_id as a_id, b.owner_id as b_id
FROM dist_owners AS a
JOIN dist_owners AS b
ON a.owner_id <> b.owner_id
), object_mixed_attributes AS (
SELECT op.*, oba.*, owa.attribute as b_att
FROM owner_pairs AS op
JOIN obj_att AS oba ON oba.owner_id = op.a_id
LEFT JOIN owner_att AS owa ON owa.owner_id = op.b_id and oba.attribute = owa.attribute
)
SELECT op.a_id as owner_a
,op.b_id as owner_b
,count(distinct(om_a.id)) as count_a
,count(distinct(om_b.id)) as count_b
,count(distinct(iff(om_a.b_att is not null,om_a.id,null))) as similarity_a_to_b
,count(distinct(iff(om_b.b_att is not null,om_b.id,null))) as similarity_b_to_a
,count_a-similarity_a_to_b as unique_a
,count_b-similarity_b_to_a as unique_b
FROM owner_pairs AS op
JOIN object_mixed_attributes AS om_a ON om_a.a_id = op.a_id
JOIN object_mixed_attributes AS om_b ON om_b.a_id = op.b_id
WHERE op.a_id < op.b_id
GROUP BY 1,2;
which gives:
OWNER_A OWNER_B COUNT_A COUNT_B SIMILARITY_A_TO_B SIMILARITY_B_TO_A UNIQUE_A UNIQUE_B
5000 6000 3 3 2 1 1 2
Given it's all equijoin's you will get the best performance out merge joins. looking at the profiles for this tiny example there are a couple WithReference blocks which over large tables, can sometime be slow than just reading from the table twice, this comes form the CTE being reused. So playing there can help.
Also I have ben lazy and not fully expanded the selects, thus using x.* at time, and maybe those values are not needed lower down. The complier will solve that for us, and if your tables are that large, then it will be lost in the wash.