sql server data resampling by specific frequency - sql-server

Question

I have a dataset of stock price history, I want to organize to a specific frequency.
For example, the original dataset like as follow,
WITH CTE(ID ,[Datetime],[Open], [High], [Low], [Close]) AS (
select 'A','2015/11/30 23:51:00',11.0, 11.2, 11.0, 11.0 union all
select 'A','2015/11/30 23:53:00',11.0, 11.2, 10.8, 10.8 union all
select 'A','2015/11/30 23:54:00',10.8, 10.8, 10.4, 10.4 union all
select 'A','2015/11/30 23:55:00',10.4, 10.7, 10.4, 10.6 union all
select 'A','2015/11/30 23:57:00',10.7, 11.0, 10.7, 11.0 union all
select 'A','2015/11/30 23:58:00',11.0, 11.2, 10.8, 11.1 union all
select 'A','2015/11/30 23:59:00',11.1, 11.3, 11.0, 11.1 union ALL
select 'A','2015/12/01 00:00:00',11.1, 11.4, 11.1, 11.3
)
SELECT * FROM CTE
which is a non-completely continuous time-series of OHLC in minutes.
I want to create a SQL-SERVER procedure to resample the dataset at a specific frequency, like 3 mins as follow,
WITH CTE(ID ,[Datetime],[Open], [High], [Low], [Close]) AS (
select 'A','2015/11/30 23:51:00',11.0, 11.2, 10.8, 10.8 union all
select 'A','2015/11/30 23:54:00',10.8, 10.8, 10.4, 10.6 union all
select 'A','2015/11/30 23:57:00',10.7, 11.3, 10.7, 11.1 union all
select 'A','2015/12/01 00:00:00',11.1, 11.4, 11.1, 11.3
)
SELECT * FROM CTE
I don't have a good logic with great performance, any idea or reference is great!
I appreciate!
This is what I tried, bad logic, bad performance.
WITH CTE(ID ,[Datetime],[Open], [High], [Low], [Close]) AS (
select 'A','2015/11/30 23:51:00',11.0, 11.2, 11.0, 11.0 union all
select 'A','2015/11/30 23:53:00',11.0, 11.2, 10.8, 10.8 union all
select 'A','2015/11/30 23:54:00',10.8, 10.8, 10.4, 10.4 union all
select 'A','2015/11/30 23:55:00',10.4, 10.7, 10.4, 10.6 union all
select 'A','2015/11/30 23:57:00',10.7, 11.0, 10.7, 11.0 union all
select 'A','2015/11/30 23:58:00',11.0, 11.2, 10.8, 11.1 union all
select 'A','2015/11/30 23:59:00',11.1, 11.3, 11.0, 11.1 union ALL
select 'A','2015/12/01 00:00:00',11.1, 11.4, 11.1, 11.3
)
select
*
from(
select
*
, ROW_NUMBER() over (partition by ID, Datediff_group order by [Datetime]) as RowN
from(
select
ID, [Datetime], Datediff_group
, FIRST_VALUE([Open]) over (Partition by ID, Datediff_group order by [Datetime]) as [Open_grouped]
, MAX([High]) over (Partition by ID, Datediff_group) as [High_grouped]
, MIN([Low]) over (Partition by ID, Datediff_group) as [Low_grouped]
, FIRST_VALUE([Close]) over (Partition by ID, Datediff_group order by [Datetime] DESC) as [Close_grouped]
from(
select
*
, Datediff_MIN /3 as Datediff_group
from(
select
*
, DATEDIFF(minute, FIRST_VALUE([Datetime]) over (partition by ID order by [Datetime]), [Datetime]) as Datediff_MIN
from CTE
) as AAA
) as AA
) as A
) as B
where RowN = 1

Answer 1

You could use a recursive CTE to get the data out. Not sure how this would perform with a larger data set, and you would need to set the maximum recursion if there were many more rows.
WITH CTE(ID ,[time],[Open], [High], [Low], [Close]) AS (
select 'A','2015/11/30 23:51:00',11.0, 11.2, 11.0, 11.0 union all
select 'A','2015/11/30 23:53:00',11.0, 11.2, 10.8, 10.8 union all
select 'A','2015/11/30 23:54:00',10.8, 10.8, 10.4, 10.4 union all
select 'A','2015/11/30 23:55:00',10.4, 10.7, 10.4, 10.6 union all
select 'A','2015/11/30 23:57:00',10.7, 11.0, 10.7, 11.0 union all
select 'A','2015/11/30 23:58:00',11.0, 11.2, 10.8, 11.1 union all
select 'A','2015/11/30 23:59:00',11.1, 11.3, 11.0, 11.1 union ALL
select 'A','2015/12/01 00:00:00',11.1, 11.4, 11.1, 11.3),
ANCHOR AS (
SELECT MIN([time]) AS [time] FROM CTE),
RECURSION AS (
SELECT
c.*
FROM
CTE c
INNER JOIN ANCHOR a ON a.[time] =c.[time]
UNION ALL
SELECT
c.*
FROM
CTE c
INNER JOIN RECURSION r ON DATEDIFF(MINUTE, r.[time], c.[time]) = 3)
SELECT
*
FROM
RECURSION;

Answer 2

This would be a lot easier if your data had a primary key (data should always have a primary key!). Then you could use 1 query to identify the first (open) and last (close) key for every row, then it's a simple query to aggregate it all to 3 minute intervals.
WITH CTE(PK, ID ,[Datetime],[Open], [High], [Low], [Close]) AS (
select 1, 'A','2015/11/30 23:51:00',11.0, 11.2, 11.0, 11.0 union all
select 2, 'A','2015/11/30 23:53:00',11.0, 11.2, 10.8, 10.8 union all
select 3, 'A','2015/11/30 23:54:00',10.8, 10.8, 10.4, 10.4 union all
select 4, 'A','2015/11/30 23:55:00',10.4, 10.7, 10.4, 10.6 union all
select 5, 'A','2015/11/30 23:57:00',10.7, 11.0, 10.7, 11.0 union all
select 6, 'A','2015/11/30 23:58:00',11.0, 11.2, 10.8, 11.1 union all
select 7, 'A','2015/11/30 23:59:00',11.1, 11.3, 11.0, 11.1 union ALL
select 8, 'A','2015/12/01 00:00:00',11.1, 11.4, 11.1, 11.3
)
,CTE2 AS (
SELECT
CTE.*,
DATEDIFF(MINUTE, '2015/11/30 23:51:00', [Datetime]) / 3 AS [Sequence] -- Note the integer division,
MIN(PK) OVER (PARTITION BY DATEDIFF(MINUTE, '2015/11/30 23:51:00', [Datetime]) / 3) AS OpenPK,
MAX(PK) OVER (PARTITION BY DATEDIFF(MINUTE, '2015/11/30 23:51:00', [Datetime]) / 3) AS ClosePK
FROM
CTE
)
SELECT
CTE2.ID,
DATEADD(minute, CTE2.[Sequence] * 3, '2015/11/30 23:51:00') as [Datetime],
cteOpen.[Open],
MAX(CTE2.[High]) AS [High],
MIN(CTE2.[Low]) AS [Low],
cteClose.[Close]
FROM
CTE2
INNER JOIN CTE cteOpen ON cteOpen.PK = CTE2.OpenPK
INNER JOIN CTE cteClose ON cteClose.PK = CTE2.ClosePK
GROUP BY
CTE2.ID,
CTE2.[Sequence],
cteOpen.[Open],
cteClose.[Close]
ORDER BY
CTE2.[Sequence]
If your data includes more than one stock then you'll have to add ID into grouping/partitioning but this should be enough to get you started.