SQL Server 2014 Standard Edition Large Table performance - sql-server

Question

I have a question with regards to performance currently I have a table that is having trouble with query performance whenever the table rows in already millions of record.
This is the table:
CREATE TABLE [dbo].[HistorySampleValues]
(
[HistoryParameterID] [int] NOT NULL,
[SourceTimeStamp] [datetime2](7) NOT NULL,
[ArchiveTimestamp] [datetime2](7) NOT NULL CONSTRAINT [DF__HistorySa__Archi__2A164134] DEFAULT (getutcdate()),
[ValueStatus] [int] NOT NULL,
[ArchiveStatus] [int] NOT NULL,
[IntegerValue] [bigint] SPARSE NULL,
[DoubleValue] [float] SPARSE NULL,
[StringValue] [varchar](100) SPARSE NULL,
[EnumNamedSetName] [varchar](100) SPARSE NULL,
[EnumNumericValue] [int] SPARSE NULL,
[EnumTextualValue] [varchar](256) SPARSE NULL
) ON [PRIMARY]
CREATE CLUSTERED INDEX [Source_HistParameterID_Index] ON [dbo].[HistorySampleValues]
(
[HistoryParameterID] ASC,
[SourceTimeStamp] ASC
) WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON)
GO
It's fairly flat with a clustered index on HistoryParameterID and SourceTimeStamp.
This is the stored procedure that I'm using
SET NOCOUNT ON;
DECLARE #SqlCommand NVARCHAR(MAX)
SET #SqlCommand = 'SELECT HistoryParameterID,
SourceTimestamp, ArchiveTimestamp,ValueStatus,ArchiveStatus,
IntegerValue,DoubleValue,StringValue,EnumNumericValue,
EnumTextualValue,EnumNamedSetName
FROM [HistorySampleValues] WITH(NOLOCK)
WHERE ([HistoryParameterID] =' + #ParamIds + '
AND
[SourceTimeStamp] >= ''' + CONVERT(VARCHAR(30),#StartTime, 25) + '''
AND
[SourceTimeStamp] <= ''' + CONVERT(VARCHAR(30),#EndTime, 25) + ''')
AND ValueStatus = ' + #ValueStatus
EXECUTE( #SqlCommand )
As you can see the HistoryParameterID and SourceTimestamp are being used as the parameters for the first query. And retrieving 8hrs worth of records which is ~28k records, it returns with an erratic performance, 1.8seconds - 700ms
Will the design scale? whenever it reaches 77 billion records? or is there any strategy to be used? the version of SQL Server is Standard Edition so there is no partitioning, columnstore to be used. Or have I reached the maximum performance of SQL Server Standard Edition?
this is the updated stored proc
#ParamIds int,
#StartTime datetime,
#EndTime datetime,
#ValueStatus int
AS
BEGIN
SET NOCOUNT ON;
SELECT HistoryParameterID,
SourceTimestamp, ArchiveTimestamp,ValueStatus,ArchiveStatus,
IntegerValue,DoubleValue,StringValue,EnumNumericValue,
EnumTextualValue,EnumNamedSetName
FROM [HistorySampleValues] WITH(NOLOCK)
WHERE
HistoryParameterID = #ParamIds
AND (SourceTimeStamp >= #StartTime AND SourceTimeStamp <=#EndTime)
AND (#ValueStatus = -1 OR ValueStatus = #ValueStatus)
I got a 1.396 second client processing time in retrieving 41213 rows to a ~849600000 rows in the table.
is there a way to improve this?

Answer 1

Everytime you execute a new SQL command, it has to be compiled by the MS SQL Server. If you re-use the command, you save on compilation time. You need to directly execute the command in the stored procedure something like this, which should allow compilation and give you more consistent results.
SELECT ...
WHERE ([HistoryParameterID] = #ParamIds
AND [SourceTimeStamp] >= #StartTime
AND [SourceTimeStamp] <= #EndTime
AND ValueStatus = #ValueStatus
This will give you also an opportunity to monitor the performance of the command.