We need to configure Always Encrypted feature for all databases. We wanted to create common Column Master Key in Master database so that we can link CMK to all databases on server. But we are not able to link CMK created in master DB to all databases. We need to create individually create master key in each database. Is there any way to create a CMK in master database and refer to all databases on the server.
Column Master Key is per-database object, i.e. you can't share it between databases. However, it is nothing more than metadata, i.e. a pointer where the actual key (certificate) is stored. With Always Encrypted the database do not have access to the actual encryption keys. They are stored in external key store (Windows Certificate Store, Azure Key Vault). So what you need to do is to replicate (copy) the existing certificate by registering it in the rest of the databases. Just script the existing CMK as CREATE COLUMN MASTER KEY statement and execute it in each of the databases:
CREATE COLUMN MASTER KEY [CMK_Auto1]
WITH
(
KEY_STORE_PROVIDER_NAME = N'MSSQL_CERTIFICATE_STORE',
KEY_PATH = N'CurrentUser/my/2CC027B4FCA85D4244B528E8CA5F73D1EBB18C69'
)
GO
This will create a new CMK in each database, but all of them will use the same certificate to encrypt the data. Then encrypt the columns you want using the existing CMK, which you created with the script above.
Related
I am having difficulties with Azure Data Factory migrating data from an encrypted table. I was wondering what I have missed as I can't find a way to attach a certificate in the pipeline to extract the data.
I have a Link Service connection that connects successfully, but I am guessing it is possibly here that the certificate needs to be added for the decrypting on the data.
In 'Copy Data' the Source dataset, the Linked service connects to the database fine. But the Table, preview data Fails.
I get a 22301 error. The detail is as follows:
A database operation failed with the following error:
Failed to decrypt column ...
Failed to decrypt a column encryption key using key store provider: 'MSSQL_CERTIFICATE_STORE'. The last 10 bytes of the encrypted column encryption key are: '...'.
Certificate with thumbprint '...' not found in certificate store 'My' in certificate location 'CurrentUser'. Verify the certificate path in the column master key definition in the database is correct, and the certificate has been imported correctly into the certificate location/store.
Parameter name: masterKeyPath
Failed to decrypt column ...
Failed to decrypt a column encryption key using key store provider: 'MSSQL_CERTIFICATE_STORE'. The last 10 bytes of the encrypted column encryption key are: '...'.
Certificate with thumbprint '...' not found in certificate store 'My' in certificate location 'CurrentUser'. Verify the certificate path in the column master key definition in the database is correct, and the certificate has been imported correctly into the certificate location/store.
Parameter name: masterKeyPath, SqlErrorNumber=0,Class=11,State=0,
Certificate with thumbprint '...' not found in certificate store 'My' in certificate location 'CurrentUser'. Verify the certificate path in the column master key definition in the database is correct, and the certificate has been imported correctly into the certificate location/store.
Parameter name: masterKeyPath
Activity ID: ...
I have no idea what this means, is there someone who can explain what the problem is and how I can rectify this I would be extremely grateful.
What I am trying to achieve, with very little success is to migrate the data from the encrypted data table in one database unencrypted to another. I want to use an Azure Data factory pipeline and make redundant SSIS.
Many thanks
I have tried using the solution, however I am now getting this error.
Error 2200.
User configuration issue
Failure happened on 'Source' side.
ErrorCode=SqlOperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=A database operation failed with the following error: 'Failed to decrypt column 'Firstname'.
Failed to decrypt a column encryption key using key store provider: 'AZURE_KEY_VAULT'. Verify the properties of the column encryption key and its column master key in your database. The last 10 bytes of the encrypted column encryption key are: '##-##-##-##-##-##-##-##-##-##'.
Could not find any resources appropriate for the specified culture or the neutral culture.
Make sure "Microsoft.SqlServer.Management.AlwaysEncrypted.AzureKeyVaultProvider.Strings.resources" was correctly embedded or linked into assembly "Microsoft.SqlServer.Management.AlwaysEncrypted.AzureKeyVaultProvider" at compile time, or that all the satellite assemblies required are loadable and fully signed.',Source=,''Type=System.Data.SqlClient.SqlException,Message=Failed to decrypt column 'Firstname'.
Failed to decrypt a column encryption key using key store provider: 'AZURE_KEY_VAULT'. Verify the properties of the column encryption key and its column master key in your database. The last 10 bytes of the encrypted column encryption key are: '##-##-##-##-##-##-##-##-##-##'.
Could not find any resources appropriate for the specified culture or the neutral culture.
Make sure "Microsoft.SqlServer.Management.AlwaysEncrypted.AzureKeyVaultProvider.Strings.resources" was correctly embedded or linked into assembly "Microsoft.SqlServer.Management.AlwaysEncrypted.AzureKeyVaultProvider" at compile time, or that all the satellite assemblies required are loadable and fully signed.,Source=.Net SqlClient Data Provider,SqlErrorNumber=0,Class=11,ErrorCode=-2146232060,State=0,Errors=[{Class=11,Number=0,State=0,Message=Failed to decrypt column 'Firstname'.,},{Class=11,Number=0,State=0,Message=Failed to decrypt a column encryption key using key store provider: 'AZURE_KEY_VAULT'. Verify the properties of the column encryption key and its column master key in your database.
The last 10 bytes of the encrypted column encryption key are: 'AE-1D-E5-C1-60-F0-2F-42-3C-C1'.,},{Class=11,Number=0,State=0,Message=Could not find any resources appropriate for the specified culture or the neutral culture.
Make sure "Microsoft.SqlServer.Management.AlwaysEncrypted.AzureKeyVaultProvider.Strings.resources" was correctly embedded or linked into assembly "Microsoft.SqlServer.Management.AlwaysEncrypted.AzureKeyVaultProvider" at compile time, or that all the satellite assemblies required are loadable and fully signed.,},],''Type=System.Resources.MissingManifestResourceException,Message=Could not find any resources appropriate for the specified culture or the neutral culture.
Make sure "Microsoft.SqlServer.Management.AlwaysEncrypted.AzureKeyVaultProvider.Strings.resources" was correctly embedded or linked into assembly "Microsoft.SqlServer.Management.AlwaysEncrypted.AzureKeyVaultProvider" at compile time, or that all the satellite assemblies required are loadable and fully signed.,Source=mscorlib,'
I'm really not sure what that all means, and from constant searching I have no solution to the problem. Why is this made so difficult, surely its just a handshake. It is very frustrating, I'm going to have to write a blog after I find the solution, I can't find anyone that has come across this issue before that wants to write about it.
Here is the setting up of ADF...
The Source SetUp
Fails to read the table data, Fails with Error 22301
SSMS Keys for Encryption
Master Key
Column Key
Key in Azure Key Vault
Link Service for Azure SQL Database
Link Service for SQL Server Database the encrypted data is here. It is supplied by a third party, i have no control over this.
The encrypted data is in the SQL Server Database, I want to decrypt this and save it to a table in my Azure SQL Database.
The columns are encrypted like so:
[Firstname] nvarchar COLLATE Danish_Norwegian_BIN2 ENCRYPTED WITH (COLUMN_ENCRYPTION_KEY = [ColumnEncryptionKey], ENCRYPTION_TYPE = Randomized, ALGORITHM = 'AEAD_AES_256_CBC_HMAC_SHA_256') NULL,
To transfer Encrypted data from data factory you have to use the Azure key vault:
Open the SQL Server Management Studio and connect with the server.
Expand Database >> security >> Always Encrypted keys >> New Column master key.
Give the master key a name, Choose the Azure Key Vault as your key storage location, login into your account, choose your Azure subscription, choose the Azure key vault, generate the key, and then click "ok."
Expand Database >> security >> Always Encrypted keys >> New Column Encryption key.
Name your column encryption key, choose the master key and click on ok.
Encrypt columns
To Encrypt a column or columns, right-click on the Table, click on "Encrypt Columns" and select single or multiple columns as per your requirement to encrypt.
Access policies
To add an access policy, open your Azure portal, navigate to the Azure key vault, select the settings tab's access policies link, and then click + Add access policy. Select the principal, which is your Azure data factory name, before choosing the important rights. Click on Add after selecting Secret Permissions, followed by Certificate Permissions.
linked service configuration
Create linked service, Name linked service, select Azure subscription, select the server name, select the database name, select the authentication type, provide the user name and password, select always encrypted select keystore authentication type as System Assigned Managed Identity and then click ok create.
Create pipeline select source and sink and run the pipeline
Reference: Using Always Encrypted
I have a SQL Server 2019 Enterprise HA system that has a encrypted master key, and every time that I have to fail over to the secondary database (or vice n versa) I have to run this script in order to have the Secondary working after the failover.
DROP SYMMETRIC KEY DataProtectionKey
DROP CERTIFICATE DataProtection
DROP MASTER KEY
CREATE MASTER KEY ENCRYPTION BY PASSWORD = 'My_encryption_key'
CREATE CERTIFICATE DataProtection WITH SUBJECT = 'Data Protection'
CREATE SYMMETRIC KEY DataProtectionKey WITH ALGORITHM = AES_256 ENCRYPTION BY CERTIFICATE DataProtection
USE [MYDATABASE]
GO
ALTER MASTER KEY FORCE REGENERATE
WITH ENCRYPTION BY PASSWORD = 'My_encryption_key'
GO
I also tried to use this other script after failing over to see if I could finally insert the Master Key and solve for all the issue.
USE [MYDATABASE]
GO
OPEN MASTER KEY DECRYPTION BY PASSWORD = 'My_encryption_key';
ALTER MASTER KEY DROP ENCRYPTION BY SERVICE MASTER KEY
ALTER MASTER KEY ADD ENCRYPTION BY SERVICE MASTER KEY
CLOSE MASTER KEY;
Unfortunately, every time that I fail over the DB, I have to run one of the two scripts.
There is any way that I could do this once for all ?
Thank you
You can try to create the certificate on primary, backup, and restore on secondary.
This code is from my repo created for a demo for AlwaysOn with docker.
Primary
-- PRIMARY
USE [master]
GO
-- Create masterkey
CREATE MASTER KEY ENCRYPTION BY PASSWORD = 'P#ssw0rd!';
GO
-- Create certificate
CREATE CERTIFICATE ao_certificate WITH SUBJECT = 'ao_certificate';
GO
BACKUP CERTIFICATE ao_certificate
TO FILE = '/var/opt/mssql/shared/ao_certificate.cert'
WITH PRIVATE KEY (
FILE = '/var/opt/mssql/shared/ao_certificate.key',
ENCRYPTION BY PASSWORD = 'P#ssw0rd!'
);
GO
Secondary
-- SECONDARY
USE [master]
GO
-- Create master key
CREATE MASTER KEY ENCRYPTION BY PASSWORD = 'P#ssw0rd!';
GO
-- create certificate with private key created on primary server and saved to shared location
CREATE CERTIFICATE ao_certificate
FROM FILE = '/var/opt/mssql/shared/ao_certificate.cert'
WITH PRIVATE KEY (
FILE = '/var/opt/mssql/shared/ao_certificate.key',
DECRYPTION BY PASSWORD = 'P#ssw0rd!'
)
GO
Yes, and the fix is relatively simple. Take a backup of the service master key (SMK) from the secondary replica and restore it to the primary replica. This will re-encrypt anything already encrypted with the existing SMK with the restored SMK. And since the SMK is now the same between both replicas, the failover will be seamless.
The reason that the direction is from secondary to primary is that the re-encryption performs writes (albeit not many) into your user database which is allowed only on the primary replica.
A colleague from my team found a better solution for this issue.
On the primary DB, I will call DBS1 backup the master key using the encryption password that you used to create the DB:
BACKUP SERVICE MASTER KEY TO FILE = 'D:\Backup\dbs1_service_master_key.key' ENCRYPTION BY PASSWORD = 'Wh#t3v3r_P#$$w0rd';
Promote the secondary DB to primary on the HA (failover), so you can run the script below:
USE [YOURDATABASE]
GO
open master key decryption by password = 'Wh#t3v3r_P#$$w0rd'
Alter Master Key Add encryption by Service Master Key
GO
Copy the service master key that you backed up on the primary DBS1 and copy in a location on DBS2.
Still on the "secondary" DBS that now is primary on the HA group, restore the certificate:
RESTORE SERVICE MASTER KEY FROM FILE = 'D:\Backup\dbs1_service_master_key.key' DECRYPTION BY PASSWORD = 'Wh#t3v3r_P#$$w0rd';
I have an SQL instance with multiple data bases. Each of them are TDE encrypted.
I know that all data bases use the same service master key, SMK, to encrypt tempdb. I have back up the SMK using:
BACKUP SERVICE MASTER KEY
TO FILE = 'tmp-path.key'
ENCRYPTION BY PASSWORD = 'temp-password';
Now, how do I back up the distinct database master key, DMK? Each data base use a different one, but the SQL statement doesn't allow to specify which one to back up. Next, is the command I'm running.
BACKUP MASTER KEY
TO FILE = 'tmep-path.key'
ENCRYPTION BY PASSWORD = 'temp-passowrd'
I though that by:
use [specific-db];
GO
It will back up an specific one, but the command won't run. You need to be on master, to run the command.
Thanks
The way TDE is configured in SQL Server is as follow:
Create Master key in master database
Create Certificate using the master key
Create Database Encryption key using the certificate
There are only two things you need to/can backup here:
Master Encryption Key - The script you have shown in your question will do that.
BACKUP MASTER KEY
TO FILE = 'tmep-path.key'
ENCRYPTION BY PASSWORD = 'temp-passowrd'
Certificate - You can either create a separate Certificate for each database to be encrypted or a single certificate to encrypt all database, You will create and backup the certificate as follow:
-- Create Certificate
USE Master;
GO
CREATE CERTIFICATE TDE_MyDB_Cert
WITH SUBJECT = 'My TDE Certificate for MyDB user database'
GO
This certificate is protected by the service master key.
-- Encrypt Database using the Certificate
USE [MyDB]
GO
CREATE DATABASE ENCRYPTION KEY
WITH ALGORITHM = AES_128
ENCRYPTION BY SERVER CERTIFICATE TDE_MyDB_Cert
GO
Now you need to backup the certificate with the private key for the database which you would do, doing:
USE master;
GO
BACKUP CERTIFICATE TDE_MyDB_Cert
TO FILE = 'C:\TDE_MyDB_Cert_File.cer'
WITH PRIVATE KEY (FILE = 'C:\TDE_MyDB_Cert_PKey.pvk' ,
ENCRYPTION BY PASSWORD = 'S0M34tR0nGP#$$w)rd' )
GO
TDE_MyDB_Cert_File.cer Is the certificate backup
TDE_MyDB_Cert_PKey.pvk is the backup for Private key for the database
Where is the SQL Server Database master key (DMK) password stored? Is it stored in master database? in registry?. If the hacker gets hold of the mdf files will he be able to get the password that was used to create DMK?
In the SQL Server there is Encryption Hierarchy - everything is encrypted by something. You can create one Database Mater Key in each database in order to protect the certificates, assymetric and symetric keys in it - the DMK is used to encrypted them.
Database master keys are protected by the Service Master Key. The
Service Master Key is created by SQL Server setup and is encrypted
with the Windows Data Protection API (DPAPI).
Let's say we have a database backup and restore it to our local instance. If I try to read some of the ecnrypted data, without knowing the DMK password in order to encrypt the certificate I am getting the following error:
OPEN MASTER KEY DECRYPTION BY PASSWORD = 'I_DO_NOT_KNOW_THE_PASS';
SELECT *, CAST(DecryptByAsymKey(AsymKey_ID(N'smGK_АSymmetricKey_01'), BufferEncryptedEmail) AS NVARCHAR(444))
FROM SecurityUsersAssimetricKey
CLOSE MASTER KEY
Msg 15313, Level 16, State 1, Line 1 The key is not encrypted using
the specified decryptor.
So, in my case (when the DMK is encrtpyed by password), I can only read the ecnrypted data knowing the password:
OPEN MASTER KEY DECRYPTION BY PASSWORD = 'smGK_MasterKeyPassword';
SELECT *, CAST(DecryptByAsymKey(AsymKey_ID(N'smGK_АSymmetricKey_01'), BufferEncryptedEmail) AS NVARCHAR(444))
FROM SecurityUsersAssimetricKey
CLOSE MASTER KEY
In order to protect the whole data, you can check howbackup encryption is made.
If backup encryption is applied, the backup is encrypted using certificate. Without this certificated, the backup cannot be restored. You can store this certificate anywhere you like. Generally, you are right for the brute force - if the database is restored, you can try to brute force the DMK, when you have it, you can read all encrypted columns. But when you cannot restore the database using its backup, you can do nothing.
I recently implemented database encryption using Symmetric/Asymmetric keys and have the Database Master Key (DMK) encrypted by password. Now if I'm understand the encryption hierarchy correctly, the DMK password will then be stored in the master database and is encrypted by the Service Master Key (SMK). My goal is to copy the database to another server to serve as a "test environment". In order to do so, I'll need to restore a copy of the Service Master Key on the destination server in order to properly encrypt/decrypt the data. I just want make sure that I'm reading the documentation correctly regarding the RESTORE MASTER KEY command. When I restore the SMK, any encrypted data on the destination server will first be decrypted by the current SMK and then re-encrypted using the new SMK. Is it safe to assume that no other database should be adversely affected if they have encryption?
Looking at the syntax for CREATE DATABASE ENCRYPTION KEY, the database master key (DMK) is encrypted by either a server-level certificate or a server-level asymmetric key. In order to restore that database on another server, the certificate or asymmetric key that protects the DMK needs to be present in the master database at the destination. Once you have that, you should be good to go.
If this is a cross-environment restore (e.g. prod → dev), I like to re-encrypt the key with an encryptor that doesn't exist at the source. It's a little added protection that ensures that the restore happens one way (i.e. you can't overwrite prod with dev).