I have BI data stored in a table in snowflake. To simplify, let's say there are only 3 columns in the table:
user_id event_time event_key
I would like to create key clusters on top of the key events. For each user, I want to find groups of consecutive rows that their event_key is in <event_keys_array> and the time difference (event_time) from the previous event of the set is less than 30 seconds.
Meaning, if the event is created less than 30 seconds from the previous event and there are no event with event_key that is not included in <event_keys_array> between them, it will be considered as the same cluster.
How can I achieve this?
This can be done inline with a collection of nested window functions. I've taken some liberties with the "event_keys_array" requirement without some example data to go on? I tend to nest sub queries but this could just as easily be expressed in a chain of CTEs
The key thing is identifying each cluster start. With that the rest falls in to place.
CREATE OR REPLACE TEMPORARY TABLE event_stream
(
event_id NUMBER(38,0)
,user_id NUMBER(38,0)
,event_key NUMBER(38,0)
,event_time TIMESTAMP_NTZ(3)
);
INSERT INTO event_stream
(event_id,user_id,event_key,event_time)
VALUES
(1 ,1,1,'2023-01-25 16:25:01.123')--User 1 - Cluster 1
,(2 ,1,1,'2023-01-25 16:25:22.123')--User 1 - Cluster 1
,(3 ,1,1,'2023-01-25 16:25:46.123')--User 1 - Cluster 1
,(4 ,1,2,'2023-01-25 16:26:01.123')--User 1 - Cluster 2 (Not in array)
,(5 ,1,3,'2023-01-25 16:26:02.123')--User 1 - Cluster 3
,(6 ,2,1,'2023-01-25 16:25:01.123')--User 2 - Cluster 1
,(7 ,2,1,'2023-01-25 16:26:01.123')--User 2 - Cluster 2
,(8 ,2,1,'2023-01-25 16:27:01.123')--User 2 - Cluster 3 (in array)
,(9 ,2,3,'2023-01-25 16:27:04.123')--User 2 - Cluster 3 (in array)
,(10,2,2,'2023-01-25 16:27:07.123')--User 2 - Cluster 4
;
SELECT --Distinct to dedup final output down to window function outputs. remove to bring event level data through alongside cluster details.
DISTINCT
D.user_id AS user_id
,MAX(CASE WHEN D.event_position = 1 THEN D.event_time END) OVER(PARTITION BY D.user_id,D.grp) AS event_cluster_start_time
,MAX(CASE WHEN D.event_position_reverse = 1 THEN D.event_time END) OVER(PARTITION BY D.user_id,D.grp) AS event_cluster_end_time
,DATEDIFF(SECOND,event_cluster_start_time,event_cluster_end_time) AS event_cluster_duration_seconds
,COUNT(1) OVER(PARTITION BY D.user_id,D.grp) AS event_cluster_total_contained_events
,FIRST_VALUE(D.event_id) OVER(PARTITION BY D.user_id,D.grp ORDER BY D.event_time ASC) AS event_cluster_intitial_event_id
FROM (
SELECT *
,ROW_NUMBER() OVER(PARTITION BY A.user_id,A.grp ORDER BY A.event_time) AS event_position
,ROW_NUMBER() OVER(PARTITION BY A.user_id,A.grp ORDER BY A.event_time DESC) AS event_position_reverse
FROM (
SELECT *
--A rolling sum of cluster starts at the row level provides a value to partition the data on.
,SUM(A.is_start) OVER(PARTITION BY A.user_id ORDER BY A.event_time ROWS UNBOUNDED PRECEDING) AS grp
FROM (
SELECT A.event_id
,A.user_id
,A.event_key
,array_contains(A.event_key::variant, array_construct(1,3)) AS event_key_grouped
,A.event_time
,LAG(event_time,1) OVER(PARTITION BY A.user_id ORDER BY A.event_time) AS previous_event_time
,LAG(event_key_grouped,1) OVER(PARTITION BY A.user_id ORDER BY A.event_time) AS previous_event_key_grouped
,CASE
WHEN --Current event should be grouped with previous if within 30 seconds
DATEADD(SECOND,-30,A.event_time) <= previous_event_time
--add additional cluster inclusion criteria, e.g. same grouped key
AND event_key_grouped = previous_event_key_grouped
THEN NULL ELSE 1
END AS is_start
FROM event_stream A
) AS A
) AS A
) AS D
ORDER BY 1,2 ;
If you wanted to split clusters by another field value such as event_key you just need to add the field to all window function partitions.
Result Set:
Related
I have a table 'item_queue' containing, items, groups and a sequence number.
Each item is unique and is held against a group with a number indicating the sequence. The count is a total for that item e.g.
group_id|item_id|sequence_order_number|count
--------------------------------------------
A |123 |1 |20
A |124 |2 |30
B |125 |1 |10
Given this information I am trying to set up sequential start and end dates
The start datetime of the first item for a group is the current time, for example assume start of item 123 is '2019-04-04 12:00:00.000' then
end datetime would be start + (count * minutes) so '2019-04-04 12:20:00.000'
The start of item 124 would equal that end date as it is the next in the sequence for that group. the end is then calculated the same way to be '2019-04-04 12:50:00.000'
item 125 would start the time again at '2019-04-04 12:00:00.000' as it is in a different group
I have attempted a few ways to do this, and I think the answer is a recursive cte, but I can't wrap my head around it to make it work for one or multiple groups, my unsuccessful attempt for a single group:
;with cte as
(
select
group_id,
item_id,
count,
GETDATE() as start_datetime,
DATEADD(MINUTE, count, GETDATE()) as end_datetime,
iq.sequence_order_number
from item_queue iq
where iq.group_id = 'A'
union all
select
group_id,
item_id,
count,
cte.end_datetime,
DATEADD(MINUTE, count, cte2.end_datetime) as end_datetime,
iq.sequence_order_number
from item_queue iq
inner join cte
on cte.group_id = iq.group_id
and cte.sequence_order_number > iq.sequence_order_number
where iq.group_id = 'A'
)
select * from cte
I suspect the answer may involve a row number window something like
ROW_NUMBER() OVER (Partition By iq.group_id Order By iq.sequence_order_number ASC)
But I have had trouble using it recursively.
Using SQL server 2012, without the ability to upgrade this database.
The minutes you want to add are practically a cumulative sum. The sum() over() window function is available in 2012 and performs exactly that. Try:
select
*,
isnull(sum([count]) over
(
partition by group_id
order by item_id asc
rows between unbounded PRECEDING and 1 PRECEDING
)
,0) as cum_count_start,
sum([count]) over ( partition by group_id order by item_id asc ) as cum_count_end
from item_queue
You already know how to use dateadd after this point.
What the individual window function caluses do:
partition by group_id : Seperate (partition) the calculations for each group_id value subset
order by item_id asc : make a virtual sorting of the rows on which the window range will be applied
rows between.... : The actual window. For the start date, we want to consider all the lines from the start (thus unbounded preceding) to the previous one (thus 1 preceding), since you don't want the start date to include the current line's [count]. Note that ommitting this clause like we did on the cum_count_end is equivelant to rows between unbounded preceding and current row.
The isnull(...,0) is needed because for the first line of each group_id you want to add 0 to the start date, but the window function sees no rows and returns NULL, so we need to change this to 0.
I have DB which having 5 column as follows:
message_id
user_id_send
user_id_rec
message_date
message_details
Looking for a SQL Serve Query, I want to Filter Results from two columns (user_id_send,user_id_rec)for Given User ID based on following constrains:
Get the Latest Record (filtered on date or message_id)
Only Unique Records (1 - 2 , 2 - 1 are same so only one record will be returned which ever is the latest one)
Ordered by Descending based on message_id
SQL Query
The main purpose of this query is to get records of user_id to find out to whom he has sent messages and from whom he had received messages.
I have also attached the sheet for your reference.
Here is my try
WITH t
AS (SELECT *
FROM messages
WHERE user_id_sender = 1)
SELECT DISTINCT user_id_reciever,
*
FROM t;
WITH h
AS (SELECT *
FROM messages
WHERE user_id_reciever = 1)
SELECT DISTINCT user_id_sender,
*
FROM h;
;WITH tmpMsg AS (
SELECT M2.message_id
,M2.user_id_receiver
,M2.user_id_sender
,M2.message_date
,M2.message_details
,ROW_NUMBER() OVER (PARTITION BY user_id_receiver+user_id_sender ORDER BY message_date DESC) AS 'RowNum'
FROM messages M2
WHERE M2.user_id_receiver = 1
OR M2.user_id_sender = 1
)
SELECT T.message_id
,T.user_id_receiver
,T.user_id_sender
,T.message_date
,T.message_details
FROM tmpMsg T
WHERE RowNum <= 1
The above should fetch you the results you are looking for when you query for a particular user_id (replace the 1 with parameter e.g. #p_user_id). The user_id_receiver+user_id_sender in the PARTITION clause ensure that records with user id combinations such as 1 - 2, 2 - 1 are not selected twice.
Hope this helps.
select * from
(
select ROW_NUMBER() over (order by message_date DESC) as rowno,
* from messages
where user_id_receiver = 1
--order by message_date DESC
) T where T.rowno = 1
UNION ALL
select * from
(
select ROW_NUMBER() over (order by message_date DESC) as rowno,
* from messages
where user_id_sender = 1
-- order by message_date DESC
) T where T.rowno = 1
Explanation: For each group of user_id_sender, it orders internally by message_date desc, and then adds row numbers, and we only want the first one (chronologically last). Then do the same for user_id_receiver, and union the results together to get 1 result set with all the desired rows. You can then add your own order by clause and additional where conditions at the end as required.
Of course, this only works for any 1 user_id at a time (replace =1 with #user_id).
To get a result from all user_id's at once, is a totally different query, so I hope this helps?
I want to transform this data
account; docdate; docnum
17700; 9/11/2015; 1
17700; 9/12/2015; 2
70070; 9/1/2015; 4
70070; 9/2/2015; 6
70070; 9/3/2015; 9
into this
account; docdate; docnum
17700; 9/12/2015; 2
70070; 9/3/2015; 9
.. for each account I want to have one row with the most current (=max(docdate)) docdate. I already tried different approaches with cross apply and row_number but couldn't achieve the desired results
Use ROW_NUMBER:
SELCT account, docdate, docnum
FROM (
SELECT account, docdate, docnum,
ROW_NUMBER() OVER (PARTITION BY account
ORDER BY docdate DESC) AS rn
FROM mytable ) AS t
WHERE t.rn = 1
PARTITION BY account clause creates slices of rows sharing the same account value. ORDER BY docdate DESC places the record having the maximum docdate value at the top of its related slice. Hence rn = 1 points to the record with the maximum docdate value within each account partition.
I am trying to create a column in SQL SERVER that shows 1 OR 0(zero). I have a column of customer numbers that appear more than once. At the first hit on a unique non-repeated customer number it should show one and if it is repeated then 0(zero). How can I create this ?
CustNumber Unique
25122134 1
25122134 0
25122134 0
25122136 1
25122136 0
the solutions I am considering and trying out now are Rank() and Rank_DENSE().
declare #test table
(
CustNumber int
)
insert into #test values
(25122134),
(25122134),
(25122134),
(25122136),
(25122136)
select
* ,
// each CustNumber in partition has the same rank, but different row_number
case when (row_number() over (partition by CustNumber order by CustNumber)) = 1
then 1 else 0 end as [Unique]
// the 1st is unique, the rest (2..n) are not
from #test
order by CustNumber, [Unique] desc
// unique in each group should be displayed first
You don't want RANK because that, by definition, produces the same output for identical inputs.
ROW_NUMBER() and a simple CASE expression should do it:
;WITH Numbered as (
SELECT CustNumber,
ROW_NUMBER() OVER (PARTITION BY CustNumber
ORDER BY CustNumber) as rn --Unusual - pick a real column if you have a preference
FROM YourUnnamedTable
)
SELECT CustNumber,CASE WHEN rn = 1 THEN 1 ELSE 0 END as [Unique]
FROM Numbered
I have this requirement.
My table contains a series of rows with serialnos and several bit columns and date-time.
To Simplify I will focus on 1 bit column.In essence, I need to know the recent date that this bit was toggled.
Ex: The following table depicts the bit values for 7 serials for the latest 6 days (10 to 5).
SQl Fiddle schema + query
I have succesfully managed to get the result in a sample but is taking ages on the real table containing over 30 million records and approx 300K serial nos.
Pseudo -->
For each Serial:
Get (max Date) bit value as A (latest bit value ex 1)
Get (max Date) NOT A as B ( Find most recent date that was ex 0)
Get the (Min Date) > B
Group by SNO
I am sure an optimised approach exists.
For completeness the dataset contains rows that I need to filter out etc. However I can build and add these later when getting the basic executing more efficiently.
Tks for your time!
with cte as
(
select *, rn = ROW_NUMBER() OVER (ORDER BY sno)
from dbo.TestCape2
)
select MAX(y.Device_date) as MaxDate,
y.SNo
from cte x
inner join cte as y
on x.rn = y.rn + 1
and x.SNo = y.SNo
and x.Cape <> y.Cape
group by y.SNo
order by SNo;
And if you're using SQL-Server 2012 and up you can make use of LAG, which will take a look at the previous row.
select max(Device_date) as MaxDate,
SNo
from (
select SNo
,Device_date
,Cape
,LAG (Cape, 1, 0) OVER (PARTITION BY Sno ORDER BY Device_date) AS PrevCape
,LAG (Sno, 1, 0) OVER (PARTITION BY Sno ORDER BY Device_date) AS PrevSno
from dbo.TestCape2) t
where sno = PrevSno
and t.Cape <> t.PrevCape
group by sno
order by sno;