Using SQL Server 2017 I followed this article to apply Always Encrypt tech:
https://www.codeproject.com/Articles/1110564/Always-Encrypted-feature-in-SQL-Server
Which is simply says:
Create Column Master key.
Create Column Encryption key.
Encrypt the
columns using the latter.
The certificate that being used for step 1. was generated using the wizard and saved in:
Key Store: Windows Certificate Store - Local Machine
I created very simple console app that is using EF to try to get data from the table that I encrypted the column in.
I added to connection string this: column encryption setting=Enabled.
When I try to get the data:
using (MyEntities en = new MyEntities())
{
var x = en.TableHasColEnc.ToList();
}
I get this error:
'Failed to decrypt column 'X'.
Failed to decrypt a column encryption key using key store provider: 'MSSQL_CERTIFICATE_STORE'. The last 10 bytes of the encrypted column encryption key are: 'XX-XX-XX-XX-XX-XX-XX-XX-XX-XX'.
Certificate with thumbprint '...' not found in certificate store 'My' in certificate location 'LocalMachine'. Verify the certificate path in the column master key definition in the database is correct, and the certificate has been imported correctly into the certificate location/store.
Parameter name: masterKeyPath'
What I got that the user that is running this app is not able to access that location to get the certificate, but why ?
If I ran the app as Admin it'll work normally.
I opened mmc and checked Certificates (Local Computer) and found the certificate that has been used in step 1. and it looks like it's correctly in local machine not current user.
Why I'm getting this error and how to solve it ?
This answer helped me to solve the problem:
How to give ASP.NET access to a private key in a certificate in the certificate store?
The idea is to grant the user who's gonna run the app (or in case of web app the IIS application pool user) the permission to be able to read private keys.
In the correct answer explains how to grant IIS user the permission, you can do the same for who ever is running your app.
UPDATE
If your app is on one server and the DB is on another (so your Encryption Certificate most probably on DB's server) you need to export and then import that certificate from DB's server to your app server and give the user who is running your app the permission to use the private key of this certificate.
Related
I am having difficulties with Azure Data Factory migrating data from an encrypted table. I was wondering what I have missed as I can't find a way to attach a certificate in the pipeline to extract the data.
I have a Link Service connection that connects successfully, but I am guessing it is possibly here that the certificate needs to be added for the decrypting on the data.
In 'Copy Data' the Source dataset, the Linked service connects to the database fine. But the Table, preview data Fails.
I get a 22301 error. The detail is as follows:
A database operation failed with the following error:
Failed to decrypt column ...
Failed to decrypt a column encryption key using key store provider: 'MSSQL_CERTIFICATE_STORE'. The last 10 bytes of the encrypted column encryption key are: '...'.
Certificate with thumbprint '...' not found in certificate store 'My' in certificate location 'CurrentUser'. Verify the certificate path in the column master key definition in the database is correct, and the certificate has been imported correctly into the certificate location/store.
Parameter name: masterKeyPath
Failed to decrypt column ...
Failed to decrypt a column encryption key using key store provider: 'MSSQL_CERTIFICATE_STORE'. The last 10 bytes of the encrypted column encryption key are: '...'.
Certificate with thumbprint '...' not found in certificate store 'My' in certificate location 'CurrentUser'. Verify the certificate path in the column master key definition in the database is correct, and the certificate has been imported correctly into the certificate location/store.
Parameter name: masterKeyPath, SqlErrorNumber=0,Class=11,State=0,
Certificate with thumbprint '...' not found in certificate store 'My' in certificate location 'CurrentUser'. Verify the certificate path in the column master key definition in the database is correct, and the certificate has been imported correctly into the certificate location/store.
Parameter name: masterKeyPath
Activity ID: ...
I have no idea what this means, is there someone who can explain what the problem is and how I can rectify this I would be extremely grateful.
What I am trying to achieve, with very little success is to migrate the data from the encrypted data table in one database unencrypted to another. I want to use an Azure Data factory pipeline and make redundant SSIS.
Many thanks
I have tried using the solution, however I am now getting this error.
Error 2200.
User configuration issue
Failure happened on 'Source' side.
ErrorCode=SqlOperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=A database operation failed with the following error: 'Failed to decrypt column 'Firstname'.
Failed to decrypt a column encryption key using key store provider: 'AZURE_KEY_VAULT'. Verify the properties of the column encryption key and its column master key in your database. The last 10 bytes of the encrypted column encryption key are: '##-##-##-##-##-##-##-##-##-##'.
Could not find any resources appropriate for the specified culture or the neutral culture.
Make sure "Microsoft.SqlServer.Management.AlwaysEncrypted.AzureKeyVaultProvider.Strings.resources" was correctly embedded or linked into assembly "Microsoft.SqlServer.Management.AlwaysEncrypted.AzureKeyVaultProvider" at compile time, or that all the satellite assemblies required are loadable and fully signed.',Source=,''Type=System.Data.SqlClient.SqlException,Message=Failed to decrypt column 'Firstname'.
Failed to decrypt a column encryption key using key store provider: 'AZURE_KEY_VAULT'. Verify the properties of the column encryption key and its column master key in your database. The last 10 bytes of the encrypted column encryption key are: '##-##-##-##-##-##-##-##-##-##'.
Could not find any resources appropriate for the specified culture or the neutral culture.
Make sure "Microsoft.SqlServer.Management.AlwaysEncrypted.AzureKeyVaultProvider.Strings.resources" was correctly embedded or linked into assembly "Microsoft.SqlServer.Management.AlwaysEncrypted.AzureKeyVaultProvider" at compile time, or that all the satellite assemblies required are loadable and fully signed.,Source=.Net SqlClient Data Provider,SqlErrorNumber=0,Class=11,ErrorCode=-2146232060,State=0,Errors=[{Class=11,Number=0,State=0,Message=Failed to decrypt column 'Firstname'.,},{Class=11,Number=0,State=0,Message=Failed to decrypt a column encryption key using key store provider: 'AZURE_KEY_VAULT'. Verify the properties of the column encryption key and its column master key in your database.
The last 10 bytes of the encrypted column encryption key are: 'AE-1D-E5-C1-60-F0-2F-42-3C-C1'.,},{Class=11,Number=0,State=0,Message=Could not find any resources appropriate for the specified culture or the neutral culture.
Make sure "Microsoft.SqlServer.Management.AlwaysEncrypted.AzureKeyVaultProvider.Strings.resources" was correctly embedded or linked into assembly "Microsoft.SqlServer.Management.AlwaysEncrypted.AzureKeyVaultProvider" at compile time, or that all the satellite assemblies required are loadable and fully signed.,},],''Type=System.Resources.MissingManifestResourceException,Message=Could not find any resources appropriate for the specified culture or the neutral culture.
Make sure "Microsoft.SqlServer.Management.AlwaysEncrypted.AzureKeyVaultProvider.Strings.resources" was correctly embedded or linked into assembly "Microsoft.SqlServer.Management.AlwaysEncrypted.AzureKeyVaultProvider" at compile time, or that all the satellite assemblies required are loadable and fully signed.,Source=mscorlib,'
I'm really not sure what that all means, and from constant searching I have no solution to the problem. Why is this made so difficult, surely its just a handshake. It is very frustrating, I'm going to have to write a blog after I find the solution, I can't find anyone that has come across this issue before that wants to write about it.
Here is the setting up of ADF...
The Source SetUp
Fails to read the table data, Fails with Error 22301
SSMS Keys for Encryption
Master Key
Column Key
Key in Azure Key Vault
Link Service for Azure SQL Database
Link Service for SQL Server Database the encrypted data is here. It is supplied by a third party, i have no control over this.
The encrypted data is in the SQL Server Database, I want to decrypt this and save it to a table in my Azure SQL Database.
The columns are encrypted like so:
[Firstname] nvarchar COLLATE Danish_Norwegian_BIN2 ENCRYPTED WITH (COLUMN_ENCRYPTION_KEY = [ColumnEncryptionKey], ENCRYPTION_TYPE = Randomized, ALGORITHM = 'AEAD_AES_256_CBC_HMAC_SHA_256') NULL,
To transfer Encrypted data from data factory you have to use the Azure key vault:
Open the SQL Server Management Studio and connect with the server.
Expand Database >> security >> Always Encrypted keys >> New Column master key.
Give the master key a name, Choose the Azure Key Vault as your key storage location, login into your account, choose your Azure subscription, choose the Azure key vault, generate the key, and then click "ok."
Expand Database >> security >> Always Encrypted keys >> New Column Encryption key.
Name your column encryption key, choose the master key and click on ok.
Encrypt columns
To Encrypt a column or columns, right-click on the Table, click on "Encrypt Columns" and select single or multiple columns as per your requirement to encrypt.
Access policies
To add an access policy, open your Azure portal, navigate to the Azure key vault, select the settings tab's access policies link, and then click + Add access policy. Select the principal, which is your Azure data factory name, before choosing the important rights. Click on Add after selecting Secret Permissions, followed by Certificate Permissions.
linked service configuration
Create linked service, Name linked service, select Azure subscription, select the server name, select the database name, select the authentication type, provide the user name and password, select always encrypted select keystore authentication type as System Assigned Managed Identity and then click ok create.
Create pipeline select source and sink and run the pipeline
Reference: Using Always Encrypted
My SNOWFLAKE database is SSO login enabled and the SSO connectivity works perfectly fine when I connect through my chrome browser.
When I try to connect to SNOWFLAKE database using DBeaver (external browser) I get the below error .
NOTE : I can confirm that I am able to see the identity verification (through explorer browser) page and the identity has also been verified. I feel the issue happens when the explorer browser confirms the identity verification back to DBeaver.
Can anyone please help ?
The above error is a generic message and could be seen due to misconfigurations either at the identity provider end or at the service provider end.
It is recommended to verify the configurations for your identity provider and make sure all the steps are performed correctly.
Below could be the common reasons for this error, However, there might be other improper configs as well which could lead to a similar error message.
a. Mismatch in user configuration details at Idp(Identity provider) and Snowflake.
b. SSO certificates are incorrect.
Solution
a. Username configured at the Identity provider end should match with the login_name at snowflake end for that user. For instance, If the SAML response shows NameID as abc#xyz.com. Then login_name configured at Snowflake end should be same as abc#xyz.com
SAML response snippet:
abc#xyz.com
Set the login_name same as the NameID configured at the identity provider side.
alter user set login_name='abc#xyz.com';
b. SSO certificate configured at Snowflake end should match with the certificate configured at the identity provider end.
Note:
The certificate value contains a number of new lines. Remove the new lines, forming a certificate value with a single line.
If the above suggestions did not help then please check the error codes for the failed login attempt in Snowflake Information Schema using the below query. And check the reason for that error code here. The below query retrieve up to 100 login events of every user your current role is allowed to monitor in the last hour and you can modify it appropriately.
select * from table(information_schema.login_history(dateadd('hours',-1,current_timestamp()),current_timestamp())) order by event_time
I created an Azure Function to load data from Eudonet CRM to my Azure SQL Database. I have two databases :
named Datawarehouse
named Datawarehouse-Dev
Both databases are identical and are in the same server.
When I load data directly to "Datawarehouse" the copy works fine, but when I change the database name to "Datawarehouse-Dev", I receive the following errors :
Index #0 Message: Login failed for user 'AzureFunction'. LineNumber: 65536 Source: Core .Net SqlClient Data Provider Procedure: Error Code:18456
-- Sql server error. If error code <17: => check sql transac code (user error). Else: => software or hardware errors (check availability of database)
Login failed for user 'AzureFunction'.
If anyone has an idea on where the problem could come from I would be very grateful and I also don't understand why there is an authentification error since they're both in the same server and are accessed with the same user/password.
Thanks in advance
Though #adnane already resolved the issue after using the connection string directly into Function application setting instead of storing it into Vault. This approach might compromise the application security because using connection string directly might expose it to unauthorized person.
Azure Key Vault is a good place to keep our application credentials in a secured and centralized manner. Moving the secrets to Key Vault becomes even more important while our Azure solution is growing.
In case, if anyone still looking for the solution by storing the connection string in Azure Key Vault and then using it in the Function, please follow the below steps.
Firstly, open the Azure Key Vault service and from the Settings menu select Access policies. Then select + Add new access policy.
Then choose Select principal and search for the name of the Function App as shown in below example.
Once your principal is selected choose the Secret permissions menu. In this case, we’ll only need to get the secret from the Key Vault (concretely read our connection string). Therefore, check Get permission only and then select OK.
At the end, select Save to store the new functionapp-demo-mw access policy.
ADDING SECRET TO AZURE KEY VAULT
Adding a secret to Azure Key Vault is straightforward. From the Key Vault, Settings menu select Secrets and then select + Generate/Import secret.
For Key Vault secret two values are required – name and the value. In this case we’ve called our secret OrderManagementDbConnectionString and as a value we put our SQL Database connection string. Select Create to save the secret.
By default, the secret is Enabled so it’s ready to use. Once the secret is created, we’ll need to get its URI (a unique location identifying the secret). Go to the Settings menu and select Secrets. We’ll find here our recently added secret (OrderManagementDbConnectionString). Select the secret and we’ll see it’s the only version in the list.
Select the current version of secret and copy its secret identifier. The identifier is an URI with pattern : https://<url_of_the_key_value>/<secret_name>/<secret_version>.
GETTING SECRET FROM KEY VAULT IN AZURE FUNCTION APP
Go back to the Azure Function App (functionapp-demo-mw) and on the Overview tab, select Configuration in Configured featured section.
Well, Select + New application settings. Put a name that describes the new setting (we’ve put OrderManagementConnectionString). At the end of the last year, Microsoft has added an option of sourcing Key Vault secrets directly from App Settings. This simplifies a lot the way how the secrets were used before. So, set the value of the setting to a secret reference in the following format:
#Microsoft.KeyVault(SecretUri=**secret_uri_with_version**)
We just need to replace secret_uri_with_version with the value we’ve previously copied from the secret in Azure Key Vault.
In Azure Function, you’ll just retrieve the value from the application settings and work with it the same way as it was directly a connection string stored in application settings.
// get value from appliction settings
var connectionString = Environment.GetEnvironmentVariable("OrderManagementConnectionString");
// create connection
SqlConnection connection = new SqlConnection(connectionString);
I've used Always encrypted for a database table column on a DB on Azure. I've gone through this article - https://learn.microsoft.com/en-us/azure/sql-database/sql-database-always-encrypted-azure-key-vault - followed it very carefully. I have encrypted the columns, and that was done successfully. I can see that the column is encrypted.
When I try to follow the same article to now try to view the data from an application, I get error stating:
Failed to decrypt a column encryption key using key store provider:
'AZURE_KEY_VAULT'. Verify the properties of the column encryption key
and its column master key in your database. The last 10 bytes of the
encrypted column encryption key are: '54-36-01-E0-5C-A1-82-80-B1-B4'.
This is the error in the code, but then after I let that error go, I get:
Access denied. Caller was not found on any access policy. Caller:
appid=afd26169-bbac-4a45-ad3c-2b4492d19c6e;oid=dabbc750-5601-442b-9809-3a17f74d5aa2;numgroups=0;iss=https://sts.windows.net/bd8eb048-c497-4576-80eb-99e763b83ffd/
Vault: AWKeyVault2;location=eastus2
It's worth noting that when I use SSMS, and I set the connection string options to Column Encryption Setting=Enabled, I can see the decrypted data in SSMS.
For the most part, I think it's a permissions problem, but I'm at a loss. I've given the App Registration full access to the Azure Subscription. I've generated a 'Secret' for the App Registration also, and using that in the application. If I can see the data from SSMS, I think it means the encryption was successful, so it must be something to do with the fact that the application just cannot get to the key in the azure key vault. What am I missing?
OK, so figured this out. It definitely was a permissions thing. In order to completely verify that you have the correct permissions, go to Key Vault Blade:
Select Access Policies from the Key Vault resource blade menu on theleft
Click the "add new" link/button at the top
Select Principal to select the application that you are using (i.e. the app registration from which you got the client ID from)
From the Key permissions drop down, make sure you give it "Decrypt", "Sign", "Get", "UnwrapKey" permissions
Make sure to save changes
This is one of the last things - you still have to make sure your app registration has the correct permissions for your subscription.
Maybe a little late to the party, but I recently publihed an article about the topic and it might help better: https://www.codeproject.com/Articles/5355073/Full-Tutorial-on-using-Always-Encrypted-with-Azure
Basically, there is a mix of settings and coding that will allow you to use Always Encrypted with Azure Key Vault and it is not only related Azure settings and permissions.
You need to register your Web App in Azure Active Directory, take the according Application ID, then create a new Secret for it, take the Secret value - then write some code to authenticate to AKV using these two.
But that is also with a bit of skepsis - please read the article.
I have a web application that read and write from/to database and I use with always encrypted.
When I tried my web application in my local device(where I developed the site) it's work perfect but when I created web site in the server it's can't read from the database because my application doesn't found the CEK(the public key cert).
In the server I created a new database and do the always encrypted, and add the applicationpoolidentity to my database with the following steps:
Add login account IIS APPPOOL\name applicationpoolidentity.
Add the login as user in the relevant database.
I search a lot for solution but nothing work.
For example:
Create new windows user connect to database with SQL Server in
Single-User Mode : I saw that solution will be problematic.
Add name application pool identity as login windows auth in the
SQL-Server but not working. Who can i connect with application pool
identity as login to SQL-Server?
I gave up, I need help :)
OK, after many searches for a soulotion I found one.
Untill now I created always with Current User Certificate Stores and because of that I can't use with the cert in all users.
So I generate new certificate in Local Machine Certificate Stores
and add the application pool identity to the cert permissions with the following steps:
run mmc.
Click Certificates -> right click on the specific cert -> all task -> manage private key -> click add -> enter the name application pool identity to the textbox -> click ok -> check v on read permission and click ok.
Now the always encrypted work perfect.