I'm having an issue with SQL Server truncating encrypted values. It seems very strange as the length of the string it's encrypting isn't long at all.
If I execute the following SQL in a query window:
OPEN SYMMETRIC KEY MySymmetricKey DECRYPTION BY CERTIFICATE MyCertificate;
SELECT
CONVERT(nvarchar, DecryptByKey(
ENCRYPTBYKEY(KEY_GUID('MySymmetricKey'), convert(varbinary(8000), cast('hellothisisastringwhichisntverylong' AS nvarchar(max))))
))
the output is: "hellothisisastringwhichisntver"
I've looked through lots of examples, however I can't see what I'm doing wrong?
I should point out also that I have tried variations of not using nvarchar(max) and varbinary(8000) and they don't change the outcome.
Cheers!
Your CONVERT statement is not specifying the NVARCHAR length to use so it's using the default NVARCHAR(30)
https://technet.microsoft.com/en-us/library/ms187928(v=sql.100).aspx
Try this instead (or whatever other length you want):
OPEN SYMMETRIC KEY MySymmetricKey DECRYPTION BY CERTIFICATE MyCertificate;
SELECT
CONVERT(nvarchar(100), DecryptByKey(
ENCRYPTBYKEY(KEY_GUID('MySymmetricKey'), convert(varbinary(8000), cast('hellothisisastringwhichisntverylong' AS nvarchar(max))))
))
Related
In theory varchar(max) and varbinary(max) columns should be capable of storing up to 2GB of data but I cannot store a unicode string 5000 characters long.
I've looked through other questions on this topic and they all suggest checking column sizes. I've done this and see that all related columns are declared with max size.
The key difference from similar questions is that, when storing I'm encrypting data using EncryptByKey and I think that it's the bottleneck I'm looking for. From MSDN I know that return type of EncryptByKey has max size of 8000 bytes, and it is not clear what is max size of #cleartext argument, but I suspect it's the same.
The following code gives me error :
OPEN SYMMETRIC KEY SK1 DECRYPTION BY CERTIFICATE Cert1;
DECLARE #tmp5k AS NVARCHAR(max);
SET #tmp5k = N'...5000 characters...';
SELECT EncryptByKey(Key_GUID('SK1'), #tmp5k);
GO
[22001][8152] String or binary data would be truncated.
How to encrypt and store big strings (around 5k unicode characters)?
So I ran into this issue when using C# and trying to encrypt and inserts a long JSON string into SQL. What ended up working was converting the plain-text string to binary and then using the same SQL EncryptByKey function to insert that instead.
If you're doing this is just SQL, I think you can use this function:
CONVERT(VARBINARY(MAX), #tmp5k) AS ToBinary
So using our example:
OPEN SYMMETRIC KEY SK1 DECRYPTION BY CERTIFICATE Cert1;
DECLARE #tmp5k AS NVARCHAR(max);
SET #tmp5k = N'...5000 characters...';
SELECT EncryptByKey(Key_GUID('SK1'), CONVERT(VARBINARY(MAX), #tmp5k));
GO
And here's an example of using SQL to convert the binary back to a string:
CONVERT(VARCHAR(100), CONVERT(VARBINARY(100), #TestString)) AS StringFromBinaryFromString ;
I am inserting data into a table like this through an SSIS OLEDB command
OPEN SYMMETRIC KEY crypto_key
DECRYPTION BY CERTIFICATE crypto_cert;
insert into tmp_ssis_auth_load_detail values(EncryptByKey(Key_GUID('crypto_key'), ?))
CLOSE SYMMETRIC KEY crypto_key
When I try to decrypt the data with the following query(in SQL Management Studio)
OPEN SYMMETRIC KEY crypto_key
DECRYPTION BY CERTIFICATE crypto_cert;
select DecryptByKey(account_number) from tmp_ssis_auth_load_detail
CLOSE SYMMETRIC KEY crypto_key
I get the result as 0x4556808712810676 and when I convert it to varchar I get some garbage value like "EV€‡v". The original value I gave the input query was "4556808712810676" which is returned with "0x" prefix by the decrypt function.
If I run the insert in SQL Management Studio I am able to decrypt it successfully.
What am I doing wrong here?
The datatype of the column being sent from the previous step was initially "DT_STR". Changing it to "DT_TEXT" fixed the issue.
I am trying to Encrypt and then Decrypt text in SQL Server 2012. I was expecting the third print to give me back customer_abc:
DECLARE
#var_customer VARCHAR(25),
#var_password VARBINARY(8000)
SET #var_customer = 'customer_abc'
SET #var_password = EncryptByPassPhrase('secret', #var_customer )
print #var_customer
print #var_password
print DecryptByPassPhrase('secret', #var_password )
Result:
customer_abc
0x01000000398F9A0D3FE98D29E8F56D6B1908EA87C08706786319DD1BBB3F150FFC5B7F3C
0x637573746F6D65725F616263
Your code is fine, except that you have to explicitly convert to get the actual varchar value. The output of Decrypt functions are still in binary. Change the last line to:
PRINT CONVERT(VARCHAR(25),DecryptByPassPhrase('secret', #var_password));
As an aside, I strongly suggest you support Unicode (so NVARCHAR instead of VARCHAR). Unless you want to look like you don't take your users' password security seriously.
Got a request to change comment field max size in application. Before had it set to varchar(500), so after reading documentation i have decided to change data type of the field from varchar(500) to varchar(max). Database accepted changes without any problems (using Microsoft SQL Server Management Studio 2005 and Microsoft SQL Server Management Studio 2008 for database management).
Then i went on changing the software. Software is written in Delphi with RemObjects to communication with database. So I changed the TDASchema for the server, it mapped my new varchar(max) field as String(65536) data type (got me a little worried there about such an explicit static size, but I went on). Then I Retrieved DataTable Schema for my TDAMemDataTable object, which updated all the fields.
I started the application and decided to see whether my database will accept changes on this specific changed field. I have edited one of the records and clicked the button to synchronize the DataSet with server and got such a fail message:
The data types varchar(max) and text are incompatible in the equal to operator
I interpret it as that my server object (the one that maps database fields with RemObjects objects) have mapped field data types to wrong data types in RemObjects.
How can this be resolved? What are the alternatives?
P.S. In this release Build .1267 logs from RemObjects it clearly states that:
fixed: DataSnap: fails to post updates to MSSQL 2005 VARCHAR(MAX)
I am using build version .1067. Wonder if update will fix the problem
P.P.S. After update to the latest version of RemObjects, the problem persists.
This error message usually happens when trying to compare a varchar(n) and text using an equality operator (usually in a where clause in sql but possible elsewhere). there was an article on MSDN which covered a few points which might relate to this.
when you store data to a VARCHAR(N) column, the values are physically stored in the same way. But when you store it to a VARCHAR(MAX) column, behind the screen the data is handled as a TEXT value. So there is some additional processing needed when dealing with a VARCHAR(MAX) value. (only if the size exceeds 8000)
You mentioned that the TDASchema had mapped your new field as String(65536) which, although never having used RemObjects before, i would assume somewhere in it's own code (or yours) is trying to do a comparison of some kind hence the error message.
Try using VARCHAR(8000) instead of MAX and see if that fixes the issue.
The other option if you can find where in the code it is doing this equality check, is to try doing a cast()
As you suspected, I think the root of your problems is that the fields haven't come into the TDASchema as the correct types. I've just tried it here and varchar(max) and nvarchar(max) fields come through to my schema as Memo and WideMemo respectively, not String(65536).
I'm using Delphi XE6 and SQL Server 2008 R2 via FireDAC.
This suggests an issue retrieving the metadata from the database. What database driver are you using? Can you try FireDAC (if available) or another driver to see if the problem persists?
Resolution for Delphi 7 and MS SQL Server 2008 R2 (SP2)
Delphi:
with TADOStoredProc.Create(Self) do
try
Connection := AConnection;
ProcedureName := ASPName;
Parameters.Refresh;
Parameters.ParamByName('#XML').Value := AXML;
try
ExecProc;
...
MS SQL Server:
ALTER PROCEDURE dbo.StoredProcName
#XML NVARCHAR(MAX)
,#ErrMsgOut NVARCHAR(MAX) = NULL OUT
AS BEGIN
SET NOCOUNT ON
DECLARE #RETURN INT = 0
,#idoc INT
BEGIN TRY
-- Prepare XML
DECLARE #XML_TEXT VARCHAR(MAX)
SET #XML_TEXT = CONVERT(VARCHAR(MAX), #XML)
EXEC sp_xml_preparedocument #idoc OUTPUT, #XML_TEXT
-- Open XML
SELECT *
FROM OPENXML (#idoc, '/ServicesList/ServicesItem', 2)
WITH
(
YourFields AndTypes
)
...
Software I'm working with uses a text field to store XML. From my searches online, the text datatype is supposed to hold 2^31 - 1 characters. Currently SQL Server is truncating the XML at 65,535 characters every time. I know this is caused by SQL Server, because if I add a 65,536th character to the column directly in Management Studio, it states that it will not update because characters will be truncated.
Is the max length really 65,535 or could this be because the database was designed in an earlier version of SQL Server (2000) and it's using the legacy text datatype instead of 2005's?
If this is the case, will altering the datatype to Text in SQL Server 2005 fix this issue?
that is a limitation of SSMS not of the text field, but you should use varchar(max) since text is deprecated
Here is also a quick test
create table TestLen (bla text)
insert TestLen values (replicate(convert(varchar(max),'a'), 100000))
select datalength(bla)
from TestLen
Returns 100000 for me
MSSQL 2000 should allow up to 2^31 - 1 characters (non unicode) in a text field, which is over 2 billion. Don't know what's causing this limitation but you might wanna try using varchar(max) or nvarchar(max). These store as many characters but allow also the regular string T-SQL functions (like LEN, SUBSTRING, REPLACE, RTRIM,...).
If you're able to convert the column, you might as well, since the text data type will be removed in a future version of SQL Server. See here.
The recommendation is to use varchar(MAX) or nvarchar(MAX). In your case, you could also use the XML data type, but that may tie you to certain database engines (if that's a consideration).
You should have a look at
XML Support in Microsoft SQL Server
2005
Beginning SQL Server 2005 XML
Programming
So I would rather try to use the data type appropriate for the use. Not make a datatype fit your use from a previous version.
Here's a little script I wrote for getting out all data
SELECT #data = N'huge data';
DECLARE #readSentence NVARCHAR (MAX) = N'';
DECLARE #dataLength INT = ( SELECT LEN (#data));
DECLARE #currIndex INT = 0;
WHILE #data <> #readSentence
BEGIN
DECLARE #temp NVARCHAR (MAX) = N'';
SET #temp = ( SELECT SUBSTRING (#data, #currIndex, 65535));
SELECT #temp;
SET #readSentence += #temp;
SET #currIndex += 65535;
END;