Select rows based on every nth interval of time - sql-server

Question

I have a table with a primary key (bigint), datetime, value, foreignKey to configuration tabel that consists of 100,000's of rows. I want to be able to obtain a row for a variable time interval. For example.
Select Timestamp, value from myTable where configID=3
AND{most recent for 15 min interval}
I have a CTE query that returns multiple rows for the interval interval
WITH Time_Interval(timestamp, value, minutes)
AS
(
Select timestamp, value, DatePart(Minute, Timestamp) from myTable
Where Timestamp >= '12/01/2012' and Timestamp <= 'Jan 10, 2013' and
ConfigID = 435 and (DatePart(Minute, Timestamp) % 15) = 0
)
Select Timestamp, value, minutes from Time_Interval
group by minutes, value, timestamp
order by Timestamp
such as:
2012-12-19 18:15:22.040 6.98 15
2012-12-19 18:15:29.887 6.98 15
2012-12-19 18:15:33.480 7.02 15
2012-12-19 18:15:49.370 7.01 15
2012-12-19 18:30:41.920 6.95 30
2012-12-19 18:30:52.437 6.93 30
2012-12-19 19:15:18.467 7.13 15
2012-12-19 19:15:34.250 7.11 15
2012-12-19 19:15:49.813 7.12 15
But as can be seen there are 4 for the 1st 15 minute interval, 2 for the next interval, etc... Worse,
If no data was obtain at an exact times stamp of 15 minutes, then there will be no value.
What I want is the most recent value for a fifteen minute interval... if if the only data for that intervall occurred at 1 second after the start of the interval.
I was thinking of Lead/over but again... the rows are not orgainzed that way. Primary Key is a bigInt and is a clustered Index. Both the timstamp column and ConfigID columns are Indexed. The above query returns 4583 rows in under a second.
Thanks for any help.

Answer 1

Try this on for size. It will even handle returning one row for instances when you have multiple timestamps for a given interval.
NOTE: This assumes your Bigint PK column is named: idx. Just substitute where you see "idx" if it is not.
;WITH Interval_Helper([minute],minute_group)
AS
(
SELECT 0, 1 UNION SELECT 1, 1 UNION SELECT 2, 1 UNION SELECT 3, 1 UNION SELECT 4, 1
UNION SELECT 5, 1 UNION SELECT 6, 1 UNION SELECT 7, 1 UNION SELECT 8, 1 UNION SELECT 9, 1
UNION SELECT 10, 1 UNION SELECT 11, 1 UNION SELECT 12, 1 UNION SELECT 13, 1 UNION SELECT 14, 1
UNION SELECT 15, 2 UNION SELECT 16, 2 UNION SELECT 17, 2 UNION SELECT 18, 2 UNION SELECT 19, 2
UNION SELECT 20, 2 UNION SELECT 21, 2 UNION SELECT 22, 2 UNION SELECT 23, 2 UNION SELECT 24, 2
UNION SELECT 25, 2 UNION SELECT 26, 2 UNION SELECT 27, 2 UNION SELECT 28, 2 UNION SELECT 29, 2
UNION SELECT 30, 3 UNION SELECT 31, 3 UNION SELECT 32, 3 UNION SELECT 33, 3 UNION SELECT 34, 3
UNION SELECT 35, 3 UNION SELECT 36, 3 UNION SELECT 37, 3 UNION SELECT 38, 3 UNION SELECT 39, 3
UNION SELECT 40, 3 UNION SELECT 41, 3 UNION SELECT 42, 3 UNION SELECT 43, 3 UNION SELECT 44, 3
UNION SELECT 45, 4 UNION SELECT 46, 4 UNION SELECT 47, 4 UNION SELECT 48, 4 UNION SELECT 49, 4
UNION SELECT 50, 4 UNION SELECT 51, 4 UNION SELECT 52, 4 UNION SELECT 53, 4 UNION SELECT 54, 4
UNION SELECT 55, 4 UNION SELECT 56, 4 UNION SELECT 57, 4 UNION SELECT 58, 4 UNION SELECT 59, 4
)
,Time_Interval([timestamp], value, [date], [hour], minute_group)
AS
(
SELECT A.[Timestamp]
,A.value
,CONVERT(smalldatetime, CONVERT(char(10), A.[Timestamp], 101))
,DATEPART(HOUR, A.[Timestamp])
,B.minute_group
FROM myTable A
JOIN Interval_Helper B
ON (DATEPART(minute, A.[Timestamp])) = B.[minute]
AND A.[Timestamp] >= '12/01/2012'
AND A.[Timestamp] <= '01/10/2013'
AND A.ConfigID = 435
)
,Time_Interval_TimeGroup([date], [hour], [minute], MaxTimestamp)
AS
(
SELECT [date]
,[hour]
,minute_group
,MAX([Timestamp]) as MaxTimestamp
FROM Time_Interval
GROUP BY [date]
,[hour]
,minute_group
)
,Time_Interval_TimeGroup_Latest(MaxTimestamp, MaxIdx)
AS
(
SELECT MaxTimestamp
,MAX(idx) as MaxIdx
FROM myTable A
JOIN Time_Interval_TimeGroup B
ON A.[Timestamp] = B.MaxTimestamp
GROUP BY MaxTimestamp
)
SELECT A.*
FROM myTable A
JOIN Time_Interval_TimeGroup_Latest B
ON A.idx = B.MaxIdx
ORDER BY A.[timestamp]
This is another take on the clever time group function from #MntManChris below:
CREATE FUNCTION dbo.fGetTimeGroup (#DatePart tinyint, #Date datetime)
RETURNS int
AS
BEGIN
RETURN CASE #DatePart
WHEN 1 THEN DATEPART(mi, #Date)
WHEN 2 THEN DATEPART(mi, #Date)/5 + 1 -- 5 min
WHEN 3 THEN DATEPART(mi, #Date)/15 + 1 -- 15 min
WHEN 4 THEN DATEPART(mi, #Date)/30 + 1 -- 30 min
WHEN 5 THEN DATEPART(hh, #Date) -- hr
WHEN 6 THEN DATEPART(hh, #Date)/6 + 1 -- 6 hours
WHEN 7 THEN DATEPART(hh, #Date)/12 + 1 -- 12 hours
WHEN 8 THEN DATEPART(d, #Date) -- day
ELSE -1
END
END

Answer 2

If you want to partition in 15 minute interval use datediff in minutes and divide by 15.
And use that partition to rank each interval.
WITH myTbl AS
(
SELECT
timestamp, value,
RANK() OVER (PARTITION BY (DATEDIFF(Mi,0, Timestamp)/15) ORDER BY Timestamp desc) RK
FROM myTable
--WHERE Timestamp BETWEEN '' AND ''
)
SELECT * FROM myTble
WHERE RK <= 1

Answer 3

As my comment above says I've used Rob's answer but implmented a user function to eliminate the Interval_Helper table and the first join. Here is the code for the user function.
BEGIN
DECLARE #Ans integer
if #DatePart = 1 -- min
return DATEPART(mi, #Date)
if #DatePart = 2 -- 5 min
return DatePart(mi,#Date)/5 + 1
if #DatePart = 3 -- 15 min
return DatePart(mi,#Date)/15 + 1
if #DatePart = 4 -- 30min
return DatePart(mi,#Date)/30 + 1
if #DatePart = 5 -- hr
return DATEPART(hh, #Date)
if #DatePart = 6 -- 6 hours
return DATEPART(hh, #Date)/6 + 1
if #DatePart = 7 -- 12 hours
return DATEPART(hh, #Date)/12 + 1
if #DatePart = 8 -- day
return DATEPART(d, #Date)
return -1
END
This then made the Time_Interval table look like
;WITH Time_Interval([timestamp], value, [date], [day], time_group)
AS
(
SELECT A.[Timestamp]
,A.value
,CONVERT(smalldatetime, CONVERT(char(10), A.[Timestamp], 101))
,DATEPART(dd, A.[Timestamp])
,dbo.fGetTimeGroup(#tInterval, A.[Timestamp]) as 'time_group'
FROM myTable A
where
A.[Timestamp] >= '12/01/2012'
AND A.[Timestamp] <= '01/10/2013'
AND A.ConfigID= 435
)
Since there is a switch from "hours" to "days" as the #TimeInterval goes from 1hr to 6hr, or 12hr or every day. I also had to have the Time_Interval_TimeGroup table switch from grouping by [hour] to grouping by [day] and of course having this in the select list.
Since this is part of a much larger abstract DB schema where both the table in question and the db are functions of the ConfigID and thus required dynamic SQL, implmenting this switch in grouping was not an issue, I simply implmented two different dynSql sections based on the value of #TimeInterval
Thanks