I was wondering whether it is possible to have a always encrypted Memory Optimized Table and also have its Primary Key automatically seeded? For example, I want to create the following:
CREATE TABLE Foo
(
[Id] [int] Identity(1,1) NOT NULL,
[Bar] NVARCHAR(MAX) NOT NULL,
CONSTRAINT [PK_Foo] PRIMARY KEY NONCLUSTERED ([Id] ASC)
)
WITH (MEMORY_OPTIMIZED = ON, DURABILITY = SCHEMA_ONLY)
GO
Fairly simple table and is initially created fine however, when I attempt to encrypt the column Bar I get the following exception.
Apr 21 2017 09:23:00 [Error] WorkitemExecution: Message:Inner
exception: System.Data.SqlClient.SqlException Details: Cannot insert
explicit value for identity column in table 'Foo' when IDENTITY_INSERT
is set to OFF..
I also tried setting SET IDENTITY_INSERT Foo ON after the create table statement but still no luck.
Reading the documentation as well doesn't seem to suggest this isn't possible but perhaps I have missed something? If this isn't possible I have another approach which I know will work.
I guess my question in summary is.. "do Memory Optimized Tables support always encrypted with an identity column".
You cannot encrypt identity columns using Always Encrypted, regardless if they are stored in memory optimized tables or in regular tables. In the normal Always Encrypted query processing flow, values inserted into encrypted columns are created and encrypted on the client-side. Note that only the client has encryption keys and can encrypt/decrypt data. SQL Server, by design, does not have access to encryption keys and cannot perform cryptographic operations.Identity column values are generated on the server side, so they do not fit the above workflow.
It would be useful to understand why you are trying to encrypt your identity column. Always Encrypted is designed to be used for protecting sensitive data. Auto-generated identifiers are typically not considered to be sensitive information.
Thanks,
Jakub
Even after your encryption through wizard failed, you should have a CMK and CEK created for you. You can check the name of the CEK that was created by browsing to YourDb -> Security -> Always Encrypted Keys -> Column Encryption Keys in SSMS object explorer. In my case the name of CEK was CEK_Auto1
If you find that there are no Column Encryption Keys in the above location, you can create new one by following the steps outlined in Provisioning Column Master Keys (New Column Master Key) and Provisioning Column Encryption Keys (New Column Encryption Key) in this article
You should be able to create the table as follows
CREATE TABLE Foo
(
[Id] [int] Identity(1,1) NOT NULL,
[Bar] NVARCHAR(MAX) COLLATE Latin1_General_BIN2 ENCRYPTED WITH (COLUMN_ENCRYPTION_KEY = [CEK_Auto1], ENCRYPTION_TYPE = Deterministic, ALGORITHM = 'AEAD_AES_256_CBC_HMAC_SHA_256') NULL,
CONSTRAINT [PK_Foo] PRIMARY KEY NONCLUSTERED ([Id] ASC)
)
WITH (MEMORY_OPTIMIZED = ON, DURABILITY = SCHEMA_ONLY)
GO
Related
Hope for help because of the following problem. Assume we have a table
CREATE TABLE [dbo].[dummy](
[id] [char](36) NOT NULL,
[name] [varchar](50) NOT NULL
) ON [PRIMARY]
If I create a primary key like this (version 1)
ALTER TABLE dummy ADD CONSTRAINT PK_dummy PRIMARY KEY (ID);
I get a unique name. In this case PK_dummy.
But if I create a primary key like this (version 2)
ALTER TABLE dummy ADD PRIMARY KEY Clustered (ID);
The name changes with every recreation of this primary key.
The format is always PK__dummy__"a dynamic number"
What is the meaning of this number?
And how can I identify primary keys created with version 2 in a hugh database?
Thanks for hints.
What is the meaning of this number?
This depends on product version - it is either based on a unique id or generated randomly.
how can I identify primary keys created with version 2 in a huge database?
SELECT *
FROM sys.key_constraints
WHERE is_system_named = 1
If you don't define the name of a constraint, index, key, etc, SQL Server will give it a name. To ensure uniqueness across the database, it therefore will add "random" characters at the end.
If having a consistent name is important then define the name in your statement, as you did in the first statement.
I have a scenario where files will be uploaded into a database table (dbo.FileImport) with each line of the file in a new row. Each row will contain the line data and the name of the file it came from. The file names are unique but may contain a few million lines. Multiple file's data may exist in the table at one time.
Each file is processed and the results are stored in a separate table. After processing the data related to the file, the data is deleted from the import table to keep the table from growing indefinitely.
The table structure is as follows:
CREATE TABLE [dbo].[FileImport] (
[Id] BIGINT IDENTITY (1, 1) NOT NULL,
[FileName] VARCHAR (100) NOT NULL,
[LineData] NVARCHAR (300) NOT NULL
);
During the processing the data for the relevant file is loaded with the following query:
SELECT [LineData] FROM [dbo].[FileImport] WHERE [FileName] = #FileName
And then deleted with the following statement:
DELETE FROM [dbo].[FileImport] WHERE [FileName] = #FileName
My question is pertaining to the table design with regard to performance and longevity...
Is it necessary to have the [Id] column if I never use it (I am concerned about running out of numbers in the Identity eventually too)?
Should I add a PRIMARY KEY Constraint to the [Id] column?
Should I have a CLUSTERED or NONCLUSTERED index for the [FileName] column?
Should I be making use of NOLOCK whenever I query this table (it is updated very regularly)?
Would there be concern of fragmentation with the continual adding and deleting of data to/from this table? If so, how should I handle this?
Any advice or thoughts would be much appreciated. Opinionated designs are welcome ;-)
Update 2017-12-10
I failed to mention that the lines of a file may not be unique. So please take this into account if this affects the recommendation.
An example script in the answer would be an added bonus! ;-)
Is it necessary to have the [Id] column if I never use it (I am
concerned about running out of numbers in the Identity eventually
too)?
It is not necessary to have an unused column. This is not a relational table and will not be referenced by a foreign key so one could make the argument a primary key is unnecessary.
I would not be concerned about running out of 64-bit integer values. bigint can hold a positive value of up to 36,028,797,018,963,967. It would take centuries to run out of values if you load 1 billion rows a second.
Should I add a PRIMARY KEY Constraint to the [Id] column?
I would create a composite clustered primary key on FileName and ID. That would provide an incremental value to facilitate retrieving rows in the order of insertion and the FileName leftmost key column would benefit your queries greatly.
Should I have a CLUSTERED or NONCLUSTERED index for the [FileName]
column?
See above.
Should I be making use of NOLOCK whenever I query this table (it is
updated very regularly)?
No. Assuming you query by FileName, only the rows requested will be touched with the suggested primary key.
Would there be concern of fragmentation with the continual adding and
deleting of data to/from this table? If so, how should I handle this?
Incremental keys avoid fragmentation.
EDIT:
Here's the suggested DDL for the table:
CREATE TABLE dbo.FileImport (
FileName VARCHAR (100) NOT NULL
, RecordNumber BIGINT NOT NULL IDENTITY
, LineData NVARCHAR (300) NOT NULL
CONSTRAINT PK_FileImport PRIMARY KEY CLUSTERED(FileName, RecordNumber)
);
Here is a rough sketch how I would do it
CREATE TABLE [FileImport].[FileName] (
[FileId] BIGINT IDENTITY (1, 1) NOT NULL,
[FileName] VARCHAR (100) NOT NULL
);
go
alter table [FileImport].[FileName]
add constraint pk_FileName primary key nonclustered (FileId)
go
create clustered index cix_FileName on [FileImport].[FileName]([FileName])
go
CREATE TABLE [FileImport].[LineData] (
[FileId] VARCHAR (100) NOT NULL,
[LineDataId] BIGINT IDENTITY (1, 1) NOT NULL,
[LineData] NVARCHAR (300) NOT NULLL.
constraint fk_LineData_FileName foreign key (FileId) references [FileImport].[FileName](FIleId)
);
alter table [FileImport].[LineData]
add constraint pk_FileName primary key clustered (FileId, LineDataId)
go
This is with some normalization so you don't have to reference your full file name every time - you probably don't have to do (in case you prefer not to and just move FileName to second table instead of the FileId and cluster your index on (FileName, LeneDataId)) it but since we are using relational database ...
No need for any additional indexes - tables are sorted by the right keys
Should I be making use of NOLOCK whenever I query this table (it is
updated very regularly)?
If your data means anything to you, don't use it, It's a matter in fact, if you have to use it - something really wrong with your DB architecture. The way it is indexed SQL Server will use Seek operation which is very fast.
Would there be concern of fragmentation with the continual adding and
deleting of data to/from this table? If so, how should I handle this?
You can set up a maintenance job that rebuilds your indexes and run it nightly with Agent (or what ever)
I want to enforce a unique constraint on a column which must be encrypted in MSSQL 2005 using Cell Level Encyption (CLE).
CREATE MASTER KEY ENCRYPTION BY PASSWORD = 'itsaSECRET!!!3£3£3£!!!'
CREATE CERTIFICATE ERCERT WITH SUBJECT = 'A cert for use by procs'
CREATE SYMMETRIC KEY ERKEY
WITH ALGORITHM = AES_256
ENCRYPTION BY CERTIFICATE ERCERT
-- This illustrates the point.
-- The results differ
DECLARE #TempTable TABLE
(
[Email] [varbinary](254) UNIQUE NOT NULL
)
OPEN SYMMETRIC KEY ERKEY DECRYPTION BY CERTIFICATE ERCERT
insert into #TempTable(Email) VALUES(EncryptByKey(Key_GUID('ERKEY'), N'duplicate'))
insert into #TempTable(Email) VALUES(EncryptByKey(Key_GUID('ERKEY'), N'duplicate'))
insert into #TempTable(Email) VALUES(EncryptByKey(Key_GUID('ERKEY'), N'duplicate'))
CLOSE SYMMETRIC KEY ERKEY
select * from #TempTable
The output makes it obvious why the constraint has 'not' been enforced. (Please excuse the awesomeness of my ascii-art.)
Email
-------
1 | 0x00703529AF46D24BA863A3534260374E01000000328909B51BA44A49510F24DF31C46F2E30977626D96617E2BD13D9115EB578852EEBAE326B8F3E2D422230478A29767C
2 | 0x00703529AF46D24BA863A3534260374E01000000773E06E1B53F2C57F97C54370FECBB45BC8A154FEA5CEEB9B6BB1133305282328AAFAD65B9BDC595F0006474190F6482
3 | 0x00703529AF46D24BA863A3534260374E01000000C9EDB1C83B52E60598038D832D34D75867AB0ABB23F9044B7EBC76832F22C432A867078D10974DC3717D6086D3031BDB
But, how do I work around this?
Normally, you could use "deterministic encryption", for example, AES ECB mode, or AES-SIV. But since you are within the limitations of SQL Server encryption, you'll have to find another way. Here is an old post that discusses the issue: http://blogs.msdn.com/b/raulga/archive/2006/03/11/549754.aspx. Here is a newer post that mentions that SQL Server 2016 will support deterministic encryption: http://www.brentozar.com/archive/2015/05/sql-server-2016-security-roadmap-session-notes-msignite/, which is what you are looking for.
I am having a primary key column in a table, which was designed as an unencrypted column in earlier stage. Now we are planning to make this as an encrypted column by using the SQL servers Inbuilt granular encryption.
I use the following sample code for testing.
I am having the following requirements on the same column.
The column is used very extensively as a foreign key in many other tables.
In the application same column is Indexed in multiple tables as a part of composite primary key/ Non clustered Indexes .
Need to search the column based on wild card search
Sort the column in both directions
The problem which I am facing is whenver I am encrypting the value it is getting different values each time.
I am not able to do a group by on the same column
when need to perform wild card based search or sorting it needed to unencrypt all data stored in the column.
Is there any better way to tackle these issues? I dont prefer using any user defined function for the same.
Please help me out.
CREATE DATABASE Bank
GO
USE Bank
GO
CREATE TABLE Account (
AccountId int NOT NULL IDENTITY(1,1) PRIMARY KEY,
AccountNumber varchar(50),
EncryptedAccountNumber varbinary(128) )
GO
-----------------
CREATE MASTER KEY ENCRYPTION BY PASSWORD = 'Pa$$w0rd'
-----------------
CREATE CERTIFICATE BankCert
WITH SUBJECT = 'Account Numbers';
GO
-----------------
CREATE SYMMETRIC KEY BankAccountKey
WITH ALGORITHM = AES_256
ENCRYPTION BY CERTIFICATE BankCert;
GO
--Encryption-----------------
OPEN SYMMETRIC KEY BankAccountKey
DECRYPTION BY CERTIFICATE BankCert; --insert original and encrypted values
INSERT INTO Account
VALUES ('123456789', ENCRYPTBYKEY(KEY_GUID('BankAccountKey'), '123456789')),
('987654321', ENCRYPTBYKEY(KEY_GUID('BankAccountKey'), '987654321'))
SELECT * FROM Account
--Decryption-----------------------------------------------------------
OPEN SYMMETRIC KEY BankAccountKey
DECRYPTION BY CERTIFICATE BankCert;
GO
--list original and decrypted values
SELECT
AccountNumber,
EncryptedAccountNumber,
LEN(EncryptedAccountNumber) AS Size,
CONVERT(varchar, DECRYPTBYKEY(EncryptedAccountNumber)) AS DecryptedAccountNumber
FROM Account
All column level encryption inside of the databases is non-deterministic and random. You could use SQL Server 2016 Always Encrypted and configure it for deterministic encryption, but with low cardinality you would be running the risk of someone using the driver to create a hash table of your account numbers and defeat the encryption. It would be best to remove it from the primary key and remove the column on all tables it references as a foreign key, that way it would only be on one table and encrypted. As far as searching, you could create a hash on the last four digits to narrow the results before decrypting. That would reduce the workload.
I'm rather new to SQL Server, (did learn SQL back in late 1980's, DB2 if I recall) Today I'm integrating my database layer into SQL, to begin with, SQL Server.
To begin with. As I do today, I will generate in runtime every databases objects, tables objects and indexes programmatically as I do with almost every visual and data object in my projects. That is, I use the visual designing tools very limited.
Every column in my project has a external description file's (every user has profile which contains these files), just as I do with database key's and for visual objects as for effect's as positioning, length, picture-mask, font size, etc. etc. i.e. dynamic forms. Almost every window, grids, filters is created in runtime just as far most of my database connections.
I did build a small test "machine" to create tables in this environment and did well, very easy to create tables within program (I use delphi and ADO)
The problem I encounter is when I flag a column as "autoincrement" or as Identity in SQL Server or if I describe a column as primary key, then SQL Server Management Studio creates automatically a index or key.
That would be ok if I could manage the name it gives this index or key.
Example of this situations:
AdoCommand.CommandText := Str_SQL;
TRY
AdoCommand.Execute;
FINALLY
NotiFy_TimeOut ('Table was created', wait_for_one_sec);
END;
My database engine creates this SQL script which I pass into the string Str_SQL above:
CREATE TABLE GENGITFL
(
NR INT NOT NULL IDENTITY,
GJM CHAR(3) NOT NULL,
HEITI VARCHAR(25) NOT NULL,
KAUPG REAL NULL,
SOLUG REAL NULL,
TOLLG REAL NULL,
DUMMY VARCHAR(20) NULL,
UNIQUE (GJM),
PRIMARY KEY (GJM)
)
SQL Server creates these two indexes automatically :
PK__GENGITFL__C51F17260A463F49
UQ__GENGITFL__C51F17277FA3E6E6
I don't want to use these names for these files, I would prefer names as:
IDX_GENGITFL_GJM
IDX_GENGITFL_NR
The reason should be obvious in light of my intro, the runtime engine can't create these names automatically and I consider it not a option to look for what index files are available within system database. If my external description say there should be index, I would like just to create names for the index automatically by using the prefix, IDX_ next the table name and last the field name or name's with underscore between, as IDX_GENGITFL_GJM etc.
Hope someone understand my poor english and presentation.. I'm rather rusty in english.
Thanks
Edit: After help from marc_s my SQL "script" is like this:
CREATE TABLE GENGITFL
(
NR INT NOT NULL IDENTITY,
GJM CHAR(3) NOT NULL,
HEITI VARCHAR(25) NOT NULL,
KAUPG REAL NULL,
SOLUG REAL NULL,
TOLLG REAL NULL,
DUMMY VARCHAR(20) NULL,
CONSTRAINT IDX_GENGITFL_NR UNIQUE (NR),
CONSTRAINT IDX_GENGITFL_GJM PRIMARY KEY (GJM),
)
CREATE INDEX IDX_GENGITFL_HEITI ON GENGITFL (HEITI)
Thanks again.
If you don't want the system default names - then just specify your own! :
CREATE TABLE GENGITFL
(
NR INT NOT NULL IDENTITY,
GJM CHAR(3) NOT NULL,
HEITI VARCHAR(25) NOT NULL,
KAUPG REAL NULL,
SOLUG REAL NULL,
TOLLG REAL NULL,
DUMMY VARCHAR(20) NULL,
CONSTRAINT IDX_GENGITFL_NR UNIQUE (GJM),
CONSTRAINT IDX_GENGITFL_GJM PRIMARY KEY (GJM)
)
See those CONSTRAINT (yourownnamehere) before the UNIQUE and the PRIMARY KEY ?
Now, your constraints are named as you defined.