Exclude a column in History table - sql-server

I have a temporal table for my employee as shown below. But the problem is that the [data] column which is of type xml makes the history table grow so fast and users are running out of space.
To solve this I need a temporal table by excluding the data column which means I am not going to have data column in my temporal table; is that possible?
I do not care on what changes in the data xml at this point.
CREATE TABLE dbo.Employee
(
[EmployeeID] int NOT NULL PRIMARY KEY CLUSTERED
, [Name] nvarchar(100) NOT NULL
, [Position] varchar(100) NOT NULL
, [Department] varchar(100) NOT NULL
, [Data] [xml] NOT NULL
, [ValidFrom] datetime2 GENERATED ALWAYS AS ROW START
, [ValidTo] datetime2 GENERATED ALWAYS AS ROW END
, PERIOD FOR SYSTEM_TIME (ValidFrom, ValidTo)
)
WITH (SYSTEM_VERSIONING = ON (HISTORY_TABLE = dbo.EmployeeHistory));

Related

How do I create system versioned tables using SSDT in Visual Studio 2019?

I'm trying to create Table with system versioning using Database Project.
Following schema gives error:
SQL70633: System-versioned temporal table must have history table name explicitly provided.
CREATE TABLE [dbo].[Products]
(
[Id] INT NOT NULL PRIMARY KEY,
[Name] NVARCHAR(255) NOT NULL,
[ModifiedBy] NVARCHAR(127) NULL
)
WITH (SYSTEM_VERSIONING = ON)
GO
With explicit name:
SQL71501: Table: [dbo].[Products] has an unresolved reference to Table [history].[ProductsHistory].
SQL46010: Incorrect syntax near ].
CREATE TABLE [dbo].[Products]
(
[Id] INT NOT NULL PRIMARY KEY,
[Name] NVARCHAR(255) NOT NULL,
[ModifiedBy] NVARCHAR(127) NULL
)
WITH (SYSTEM_VERSIONING = ON (HISTORY_TABLE = [history].ProductsHistory))
GO
I've tried both, latest version of Visual Studio 2019 (16.7.5) and latest preview (16.8.0 Preview 3.2).
The syntax in both cases is invalid. Executing the first query in SSMS returns:
Cannot set SYSTEM_VERSIONING to ON when SYSTEM_TIME period is not defined.
The command needs a PERIOD FOR SYSTEM_TIME clause specifying the columns used to specify the validity period of a record.
The documentation examples show how to create a temporal table with a default, automatically named history table :
CREATE TABLE [dbo].[Products]
(
[Id] INT NOT NULL PRIMARY KEY,
[Name] NVARCHAR(255) NOT NULL,
[ModifiedBy] NVARCHAR(127) NULL,
SysStartTime DATETIME2 GENERATED ALWAYS AS ROW START NOT NULL,
SysEndTime DATETIME2 GENERATED ALWAYS AS ROW END NOT NULL,
PERIOD FOR SYSTEM_TIME (SysStartTime,SysEndTime)
)
WITH (SYSTEM_VERSIONING = ON)
In this case, the SysStartTime and SysEndTime are used to specify the validity period of a record.
Similar syntax is needed to create a temporal table with a user-specified table name
create TABLE [dbo].[Products]
(
[Id] INT NOT NULL PRIMARY KEY,
[Name] NVARCHAR(255) NOT NULL,
[ModifiedBy] NVARCHAR(127) NULL,
SysStartTime DATETIME2 GENERATED ALWAYS AS ROW START NOT NULL,
SysEndTime DATETIME2 GENERATED ALWAYS AS ROW END NOT NULL,
PERIOD FOR SYSTEM_TIME (SysStartTime,SysEndTime)
)
WITH (SYSTEM_VERSIONING = ON (HISTORY_TABLE = dbo.ProductHistory))
It's possible to create the history table on a different schema, eg history, as long as that schema exists, BUT it's probably not a good idea unless this solves some specific problem. The current and history table represent the same entity, depend on each other and have specific security restrictions so storing them in separate schemas can make life harder.
To create the table in a separate schema, first create the schema :
CREATE SCHEMA history
Then use the schema in the table definition:
WITH (SYSTEM_VERSIONING = ON (HISTORY_TABLE = history.ProductHistory))

How to prevent SQL71609 with SSDT on a new temporal table (system-versioned)

I try to use SQL Server Temporal Tables within Visual Studio 2017 and SQL Server Data Tools (SSDT).
But I get immediately following error:
SQL71609: System-versioned current and history tables do not have
matching schemas. Mismatched column: '[dbo].[MyTable].[ValidFrom]'
I don't see any mistake. Do I miss something?
I created a small repository on GIT HUB for reproduction
The current table is defined as:
CREATE TABLE [dbo].[MyTable]
(
[TenantId] UNIQUEIDENTIFIER NOT NULL CONSTRAINT [DF_MyTable_TenantId] DEFAULT
CAST(SESSION_CONTEXT(N'TenantId') AS UNIQUEIDENTIFIER),
[Rn] BIGINT IDENTITY(1,1) NOT NULL,
[Id] UNIQUEIDENTIFIER NOT NULL,
[PropA] INT NOT NULL,
[PropB] NVARCHAR(100) NOT NULL,
[ValidFrom] DATETIME2 GENERATED ALWAYS AS ROW START HIDDEN NOT NULL CONSTRAINT [DF_ValidFrom] DEFAULT CONVERT(DATETIME2, '0001-01-01'),
[ValidTo] DATETIME2 GENERATED ALWAYS AS ROW END HIDDEN NOT NULL CONSTRAINT [DF_ValidTo] DEFAULT CONVERT(DATETIME2, '9999-12-31 23:59:59.9999999'),
PERIOD FOR SYSTEM_TIME ([ValidFrom], [ValidTo]),
CONSTRAINT [PK_MyTable] PRIMARY KEY NONCLUSTERED ([Id]),
)
WITH (SYSTEM_VERSIONING = ON (HISTORY_TABLE = [dbo].[MyTableHistory]))
GO
CREATE UNIQUE CLUSTERED INDEX [CIX_MyTable] ON [dbo].[MyTable]([Rn])
GO
And the history table :
CREATE TABLE [dbo].[MyTableHistory]
(
[TenantId] UNIQUEIDENTIFIER NOT NULL,
[Rn] BIGINT IDENTITY(1,1) NOT NULL,
[Id] UNIQUEIDENTIFIER NOT NULL,
[PropA] INT NOT NULL,
[PropB] NVARCHAR(100) NOT NULL,
[ValidFrom] DATETIME2,
[ValidTo] DATETIME2,
);
GO
CREATE CLUSTERED COLUMNSTORE INDEX [COLIX_MyTableHistory]
ON [dbo].[MyTableHistory];
GO
CREATE NONCLUSTERED INDEX [IX_ImpactHistory_ValidFrom_ValidTo_Id]
ON [dbo].[MyTableHistory] ([ValidFrom], [ValidTo], [Id]);
GO
Not really sure why you are getting this particular error message.
I've tested your code on db fiddle and got different errors.
BTW, please note that you don't have to write the history table yourself - if you only set it's name using the SYSTEM_VERSIONING = ON (HISTORY_TABLE = [dbo].[MyTableHistory]) and not create it, SQL Server will generate it automatically for you - as can be seen in this fiddle.
For the first attempt I've got this error:
Msg 13518 Level 16 State 1 Line 20
Setting SYSTEM_VERSIONING to ON failed because history table 'fiddle_e3d361da65804a39b041c8149132b443.dbo.MyTableHistory' has IDENTITY column specification. Consider dropping all IDENTITY column specifications and trying again.
So I've removed the identity from [Rn] column in the history table and tried again.
Then I've got this error:
Msg 13530 Level 16 State 1 Line 20
Setting SYSTEM_VERSIONING to ON failed because system column 'ValidFrom' in history table 'fiddle_d6660ab11cdc448dba35790867169a14.dbo.MyTableHistory' corresponds to a period column in table 'fiddle_d6660ab11cdc448dba35790867169a14.dbo.MyTable' and cannot be nullable.
So I've changed both the ValidFrom and ValidTo columns to NOT NULL and finally got it working.
The working version is copied to here:
CREATE TABLE [dbo].[MyTableHistory]
(
[TenantId] UNIQUEIDENTIFIER NOT NULL,
[Rn] BIGINT NOT NULL,
[Id] UNIQUEIDENTIFIER NOT NULL,
[PropA] INT NOT NULL,
[PropB] NVARCHAR(100) NOT NULL,
[ValidFrom] DATETIME2 NOT NULL,
[ValidTo] DATETIME2 NOT NULL,
);
CREATE CLUSTERED COLUMNSTORE INDEX [COLIX_MyTableHistory]
ON [dbo].[MyTableHistory];
CREATE NONCLUSTERED INDEX [IX_ImpactHistory_ValidFrom_ValidTo_Id]
ON [dbo].[MyTableHistory] ([ValidFrom], [ValidTo], [Id]);
CREATE TABLE [dbo].[MyTable]
(
[TenantId] UNIQUEIDENTIFIER NOT NULL CONSTRAINT [DF_MyTable_TenantId] DEFAULT CAST(SESSION_CONTEXT(N'TenantId') AS UNIQUEIDENTIFIER),
[Rn] BIGINT IDENTITY(1,1) NOT NULL,
[Id] UNIQUEIDENTIFIER NOT NULL,
[PropA] INT NOT NULL,
[PropB] NVARCHAR(100) NOT NULL,
[ValidFrom] DATETIME2 GENERATED ALWAYS AS ROW START HIDDEN NOT NULL CONSTRAINT [DF_ValidFrom] DEFAULT CONVERT(DATETIME2, '0001-01-01'),
[ValidTo] DATETIME2 GENERATED ALWAYS AS ROW END HIDDEN NOT NULL CONSTRAINT [DF_ValidTo] DEFAULT CONVERT(DATETIME2, '9999-12-31 23:59:59.9999999'),
PERIOD FOR SYSTEM_TIME ([ValidFrom], [ValidTo]),
CONSTRAINT [PK_MyTable] PRIMARY KEY NONCLUSTERED ([Id]),
)
WITH (SYSTEM_VERSIONING = ON (HISTORY_TABLE = [dbo].[MyTableHistory]))
CREATE UNIQUE CLUSTERED INDEX [CIX_MyTable] ON [dbo].[MyTable]([Rn])

Altering an existing column to 'Primary Key' and 'NOT NULL' without dropping the table

I have a table named dbo.ReferenceDetails which contains several millions of records. Here is the create and select script of my table :
CREATE TABLE [dbo].[RefDetails](
[REQ_XREF_TYPE] [varchar](12) NULL
, [REQUEST_ID] [varchar](24) NULL
, [CROSS_REFERENCE] [varchar](32) NULL
, [RUN_BY] [varchar](100) NULL
, [RUN_DATE] [datetime] NULL
, [ISCURRENTRECORD] [int] NULL
, [RECORDSTARTDATE] [datetime2](7) NULL
, [RECORDENDDATE] [datetime2](7) NULL
, [UPDATE_FLAG] [varchar](50) NULL
, [SECUREFLAG] [int] NULL
, [EVENT_TIMESTAMP] [datetime2](7) NULL
) ON [PRIMARY]
SELECT TOP (10)
[REQ_XREF_TYPE]
,[REQUEST_ID]
,[CROSS_REFERENCE]
,[RUN_BY]
,[RUN_DATE]
,[ISCURRENTRECORD]
,[RECORDSTARTDATE]
,[RECORDENDDATE]
,[UPDATE_FLAG]
,[SECUREFLAG]
,[EVENT_TIMESTAMP]
FROM [dbo].[ReferenceDetails]
Can I alter the table so that REQ_XREF_TYPE, ISCURRENTRECORD and EVENT_TIMESTAMP is Primary Key and NOT NULL without dropping the table?
Your response will be appreciated. :)
See below. You first need to convert the columns to NOT NULL then you create the Primary Key. If you already have data in the table then the creation of the primary key may take some time.
ALTER TABLE [dbo].[RefDetails] ALTER COLUMN [REQ_XREF_TYPE] VARCHAR(12) NOT NULL
ALTER TABLE [dbo].[RefDetails] ALTER COLUMN [ISCURRENTRECORD] INT NOT NULL
ALTER TABLE [dbo].[RefDetails] ALTER COLUMN [EVENT_TIMESTAMP] DATETIME2(7) NOT NULL
ALTER TABLE [dbo].[RefDetails] ADD CONSTRAINT PK_RefDetails PRIMARY KEY ([REQ_XREF_TYPE],[ISCURRENTRECORD],[EVENT_TIMESTAMP]);
Note: Your creation script says the table name is RefDetails but your OP says ReferenceDetails. I went with the creation script name.
Update:
The Primary Key requires that any column(s) selected contain a unique combination - duplicates are not allowed. If duplicates exist, the creation of the primary key will fail. To check for duplicates before creating a primary key, run the following:
SELECT [REQ_XREF_TYPE], [ISCURRENTRECORD], [EVENT_TIMESTAMP], CountDupes = COUNT(1)
FROM [dbo].[RefDetails]
GROUP BY [REQ_XREF_TYPE], [ISCURRENTRECORD], [EVENT_TIMESTAMP]
HAVING COUNT(1) > 1
ORDER BY [REQ_XREF_TYPE], [ISCURRENTRECORD], [EVENT_TIMESTAMP]
You are expecting 0 results, which means there are no duplicates. Any result will identify the unique set of duplicate records along with the number of times they are duplicates (see the CountDupes column result).
If you get 0 results, then you are clear to create the primary key.
If you get any results, then you will need to address this (i.e., remove the duplicates or include additional columns that create a unique combination).

Explicitly Set ValidFrom in Temporal Table

I'm trying to set the ValidFrom range for the current record in a temporal table. I'm doing this because I'm rebuilding history from another system (non SQL) into a data warehouse so the current version of records may be "as of" a date that's in the past. If I can't get this to work, my fall back is to add a row in the history table that fills in the gap but I'm thinking there's a way to get this to work. Maybe there are some ways with alter columns?
/******** CURRENT TIME=2018-03-10 15:32:26 *****/
CREATE TABLE TestHist(
ID int NOT NULL,
Name varchar(max),
--Temporal Stuff
ValidFrom datetime2(7) NOT NULL,
ValidTo datetime2(7) NOT NULL
)
GO
CREATE TABLE Test(
ID int IDENTITY(1,1) NOT NULL,
Name varchar(max),
--Temporal Stuff
ValidFrom datetime2(7) GENERATED ALWAYS AS ROW START NOT NULL,
ValidTo datetime2(7) GENERATED ALWAYS AS ROW END NOT NULL,
PRIMARY KEY CLUSTERED (ID ASC) ,
PERIOD FOR SYSTEM_TIME (ValidFrom, ValidTo)
)
WITH( SYSTEM_VERSIONING = ON ( HISTORY_TABLE = dbo.TestHist ) )
GO
ALTER TABLE Test SET (SYSTEM_VERSIONING = OFF)
go
--THIS WORKS BUT SETS THE VALIDFROM TO CURRENT TIME
insert into Test(name) values ('fred')
--AND BTW, THIS IS HOW I LOAD THE HISTORY (THIS WORKS TOO)
insert into TestHist(ID,Name,ValidFrom,ValidTo) values (1,'joe',null,'1/1/18','1/15/18')
insert into TestHist(ID,Name,ValidFrom,ValidTo) values (1,'steve','fred','2/1/18','3/1/18')
But the problem is that it sets the current ValidFrom time arbitrarily to when you do your insert statement:
select * from test
ID Name ParentName ValidFrom ValidTo
1 fred NULL 2018-03-10 15:32:26.4403041 9999-12-31 23:59:59.9999999
And here's what I wish I could do:
--THIS DOESN'T WORK
insert into Test(name,ValidFrom,ValidTo) values ('fred','2/1/18','9999-12-31 23:59:59.997')
I get this error:
Msg 13536, Level 16, State 1, Line 38
Cannot insert an explicit value into a GENERATED ALWAYS column in table 'CodeAnalytics.dbo.Test'. Use INSERT with a column list to exclude the GENERATED ALWAYS column, or insert a DEFAULT into GENERATED ALWAYS column.
You cannot update ValidFrom on the temporal table. However, you can update ValidFrom on the history table that keeps track of changes. You create a record there by changing any value in the temporal table.
So you can do following steps:
Change anything in rows of your temporal table where you want to change the value of the ValidFrom column. This step creates a record in the history table for every changed record in the original table.
Set system versioning off for your temporal table.
Update ValidFrom in your history table.
Set system versioning back on for your temporal table.
Edit: oops. it's 2019 now. anyway, i needed to do this, so leaving here in case anyone else finds useful.
maybe something like this is what you're looking for?
CREATE TABLE TestHist(
ID int NOT NULL,
Name varchar(max),
--Temporal Stuff
ValidFrom datetime2(7) NOT NULL,
ValidTo datetime2(7) NOT NULL
)
GO
CREATE TABLE Test(
ID int IDENTITY(1,1) NOT NULL,
Name varchar(max),
--Temporal Stuff
ValidFrom datetime2(7) GENERATED ALWAYS AS ROW START NOT NULL,
ValidTo datetime2(7) GENERATED ALWAYS AS ROW END NOT NULL,
PRIMARY KEY CLUSTERED (ID ASC) ,
PERIOD FOR SYSTEM_TIME (ValidFrom, ValidTo)
)
WITH( SYSTEM_VERSIONING = ON ( HISTORY_TABLE = dbo.TestHist ) )
GO
ALTER TABLE Test SET (SYSTEM_VERSIONING = OFF);
insert into TestHist(ID,Name,ValidFrom,ValidTo) values (1,'steve','2/1/18','3/1/18')
insert into TestHist(ID,Name,ValidFrom,ValidTo) values (1,'joe','1/1/18','1/15/18')
SET IDENTITY_INSERT Test ON;
insert into Test(id, name) values (1, 'fred')
--after dropping period one can update validfrom on temporal table from max history
alter table Test DROP PERIOD FOR SYSTEM_TIME;
GO
update test set validfrom =(select max(validto) from TestHist where id=test.ID);
--now add period and turn system versioning back on
alter table test ADD PERIOD FOR SYSTEM_TIME (ValidFrom, ValidTo);
GO
alter table test set (system_versioning = on (HISTORY_TABLE=dbo.TestHist));
GO
--think this gives what you want
select * from test for system_time all

How to improve my query performance by indexing

i just want to know how will i index the this table for optimal performance? This will potentially hold around 20M rows.
CREATE TABLE [dbo].[Table1](
[ID] [bigint] NOT NULL,
[Col1] [varchar](100) NULL,
[Col2] [varchar](100) NULL,
[Description] [varchar](100) NULL
) ON [PRIMARY]
Basically, this table will be queried ONLY in this manner.
SELECT ID FROM Table1
WHERE Col1 = 'exactVal1' AND Col2 = 'exactVal2' AND [Description] = 'exactDesc'
This is what i did:
CREATE NONCLUSTERED INDEX IX_ID
ON Table1(ID)
GO
CREATE NONCLUSTERED INDEX IX_Col1
ON Table1(Col1)
GO
CREATE NONCLUSTERED INDEX IX_Col2
ON Table1(Col2)
GO
CREATE NONCLUSTERED INDEX IX_ValueDescription
ON Table1(ValueDescription)
GO
Am i right to index all these columns? Not really that confident yet. Just new to SQL stuff, please let me know if im on the right track.
Again, a lot of data will be put on this table. Unfortunately, i cannot test the performance yet since there are no available data. But I will soon be generating some dummy data to test the performance. But it would be great if there is already another option(suggestion) available that i can compare the results with.
Thanks,
jack
I would combine these indexes into one index, instead of having three separate indexes. For example:
CREATE INDEX ix_cols ON dbo.Table1 (Col1, Col2, Description)
If this combination of columns is unique within the table, then you should add the UNIQUE keyword to make the index unique. This is for performance reasons, but, also, more importantly, to enforce uniqueness. It may also be created as a primary key if that is appropriate.
Placing all of the columns into one index will give better performance because it will not be necessary for SQL Server to use multiple passes to find the row you are seeking.
Try this -
CREATE TABLE dbo.Table1
(
ID BIGINT NOT NULL
, Col1 VARCHAR(100) NULL
, Col2 VARCHAR(100) NULL
, [Description] VARCHAR(100) NULL
)
GO
CREATE CLUSTERED INDEX IX_Table1 ON dbo.Table1
(
Col1
, Col2
, [Description]
)
Or this -
CREATE TABLE dbo.Table1
(
ID BIGINT PRIMARY KEY NOT NULL
, Col1 VARCHAR(100) NULL
, Col2 VARCHAR(100) NULL
, [Description] VARCHAR(100) NULL
)
GO
CREATE UNIQUE NONCLUSTERED INDEX IX_Table1 ON dbo.Table1
(
Col1
, Col2
, [Description]
)

Resources