In theory varchar(max) and varbinary(max) columns should be capable of storing up to 2GB of data but I cannot store a unicode string 5000 characters long.
I've looked through other questions on this topic and they all suggest checking column sizes. I've done this and see that all related columns are declared with max size.
The key difference from similar questions is that, when storing I'm encrypting data using EncryptByKey and I think that it's the bottleneck I'm looking for. From MSDN I know that return type of EncryptByKey has max size of 8000 bytes, and it is not clear what is max size of #cleartext argument, but I suspect it's the same.
The following code gives me error :
OPEN SYMMETRIC KEY SK1 DECRYPTION BY CERTIFICATE Cert1;
DECLARE #tmp5k AS NVARCHAR(max);
SET #tmp5k = N'...5000 characters...';
SELECT EncryptByKey(Key_GUID('SK1'), #tmp5k);
GO
[22001][8152] String or binary data would be truncated.
How to encrypt and store big strings (around 5k unicode characters)?
So I ran into this issue when using C# and trying to encrypt and inserts a long JSON string into SQL. What ended up working was converting the plain-text string to binary and then using the same SQL EncryptByKey function to insert that instead.
If you're doing this is just SQL, I think you can use this function:
CONVERT(VARBINARY(MAX), #tmp5k) AS ToBinary
So using our example:
OPEN SYMMETRIC KEY SK1 DECRYPTION BY CERTIFICATE Cert1;
DECLARE #tmp5k AS NVARCHAR(max);
SET #tmp5k = N'...5000 characters...';
SELECT EncryptByKey(Key_GUID('SK1'), CONVERT(VARBINARY(MAX), #tmp5k));
GO
And here's an example of using SQL to convert the binary back to a string:
CONVERT(VARCHAR(100), CONVERT(VARBINARY(100), #TestString)) AS StringFromBinaryFromString ;
Related
Wondering which data type to select for my SQL Server to store the sha512 password hash. (hashed value will always have fixed length)
declare #hashedPasswd varchar(max)
set #hashedPasswd = convert(varchar(max), HASHBYTES('SHA2_512', 'any password with any langth'), 1)
select len (#hashedPasswd)
always returns length of 130.
What is the best choice for datatype of the column?
Variants are nvarchar(max), nvarchar(130), varchar, char.
If I understand correctly, nvarchar is a waste of space in my case, because It will be only ASCII symbols in hashed value.
Please assist.
SHA2_512 is 64 bytes long and internaly a varbinary so I would suggest using this datatype instead.
For more safty I also would recommend to use an additional salt for password encryption and decryption. You can find a useful description here:
https://www.mssqltips.com/sqlservertip/4037/storing-passwords-in-a-secure-way-in-a-sql-server-database/
Best regards, Stephan
According to the documentation, hasbytes returns a varbinary therefore your data type is varbinary.
Your length of 130 is only because you are casting to a varchar and is an inefficient way to store it. From the documentation, sha512 returns 64 bytes, therefore your length required is 64.
declare #hashedPasswd varbinary(max)
set #hashedPasswd = HASHBYTES('SHA2_512', 'any password with any length')
select len (#hashedPasswd)
What I need is to ensure that a string gets encoded in a known character encoding. So far, my research and testing with MS SQL Server has revealed that the documented encoding is 'UCS-2', however the actual encoding (on the server in question) is 'UCS-2LE'.
Which doesn't seem very reliable. What I would love is an ENCODE function as found in PERL, Node, or most anything, so that regardless of upgrades or settings changes, my hash function will be working on known input.
We can limit the hashing string to HEX, so at worst, we could manually map the 16 possible input characters to the proper bytes. Anyone have a recommendation on this?
Here's the PERL I'm using:
use Digest::SHA qw/sha256/;
use Encode qw/encode/;
$seed = 'DDFF5D36-F14D-495D-BAA6-3688786D6CFA';
$string = '123456789';
$target = '57392CD6A5192B6185C5999EB23D240BB7CEFD26E377D904F6FEF262ED176F97';
$encoded = encode('UCS-2LE', $seed.$string);
$sha256 = uc(unpack("H*", sha256($encoded)));
print "$target\n$sha256\n";
Which matches MS SQL:
HASHBYTES('SHA_256', 'DDFF5D36-F14D-495D-BAA6-3688786D6CFA123456789')
But what I really want is:
HASHBYTES('SHA_256', ENCODE('UCS2-LE', 'DDFF5D36-F14D-495D-BAA6-3688786D6CFA123456789'))
So that no matter what MS SQL happens to be encoding the input string as, the HASHBYTES will always operate on a known byte array.
SQL Server uses UCS-2 only on columns, variables and literals that were declared as nvarchar. In all other cases it uses 8-bit ASCII with the encoding of the current database, unless specified otherwise (using the collate clause, for example).
So, you either have to specify a Unicode literal:
select HASHBYTES('SHA_256', N'DDFF5D36-F14D-495D-BAA6-3688786D6CFA123456789');
Or, you can use a variable or table column of the nvarchar data type:
-- Variable
declare #var nvarchar(128) = N'DDFF5D36-F14D-495D-BAA6-3688786D6CFA123456789';
select HASHBYTES('SHA_256', #var);
-- Table column
declare #t table(
Value nvarchar(128)
);
insert into #t
select #var;
select HASHBYTES('SHA_256', t.Value)
from #t t;
P.S. Of course, since Wintel is a little-endian platform, SQL Server uses the same version of the encoding as the OS / hardware. Unless something new will come out in SQL Server 2017, there is no way to get big-endian representation in this universe natively.
I'm having an issue with SQL Server truncating encrypted values. It seems very strange as the length of the string it's encrypting isn't long at all.
If I execute the following SQL in a query window:
OPEN SYMMETRIC KEY MySymmetricKey DECRYPTION BY CERTIFICATE MyCertificate;
SELECT
CONVERT(nvarchar, DecryptByKey(
ENCRYPTBYKEY(KEY_GUID('MySymmetricKey'), convert(varbinary(8000), cast('hellothisisastringwhichisntverylong' AS nvarchar(max))))
))
the output is: "hellothisisastringwhichisntver"
I've looked through lots of examples, however I can't see what I'm doing wrong?
I should point out also that I have tried variations of not using nvarchar(max) and varbinary(8000) and they don't change the outcome.
Cheers!
Your CONVERT statement is not specifying the NVARCHAR length to use so it's using the default NVARCHAR(30)
https://technet.microsoft.com/en-us/library/ms187928(v=sql.100).aspx
Try this instead (or whatever other length you want):
OPEN SYMMETRIC KEY MySymmetricKey DECRYPTION BY CERTIFICATE MyCertificate;
SELECT
CONVERT(nvarchar(100), DecryptByKey(
ENCRYPTBYKEY(KEY_GUID('MySymmetricKey'), convert(varbinary(8000), cast('hellothisisastringwhichisntverylong' AS nvarchar(max))))
))
I have a table with a column data type nvarchar(max), the column will have data more than 8000 characters.
mytext navarchar(max)
I want to calculate hash value of that column, I am using the following code in MS SQL 2008/R2
select HASHBYTES('md5',column_name)
But I am getting error as,
String or binary data would be truncated.
Is that possible to calculate hash value in nvarchar(max) field in sql query.
Or is there any other ways to do it.
Thanks in advance.
Allowed input values are limited to 8000 bytes as was mentioned.
Try:
select master.sys.fn_repl_hash_binary(cast(column_name as varbinary(max)))
For this operation you have to disable FIPS validated cryptographic algorithms:
http://blog.aggregatedintelligence.com/2007/10/fips-validated-cryptographic-algorithms.html
Software I'm working with uses a text field to store XML. From my searches online, the text datatype is supposed to hold 2^31 - 1 characters. Currently SQL Server is truncating the XML at 65,535 characters every time. I know this is caused by SQL Server, because if I add a 65,536th character to the column directly in Management Studio, it states that it will not update because characters will be truncated.
Is the max length really 65,535 or could this be because the database was designed in an earlier version of SQL Server (2000) and it's using the legacy text datatype instead of 2005's?
If this is the case, will altering the datatype to Text in SQL Server 2005 fix this issue?
that is a limitation of SSMS not of the text field, but you should use varchar(max) since text is deprecated
Here is also a quick test
create table TestLen (bla text)
insert TestLen values (replicate(convert(varchar(max),'a'), 100000))
select datalength(bla)
from TestLen
Returns 100000 for me
MSSQL 2000 should allow up to 2^31 - 1 characters (non unicode) in a text field, which is over 2 billion. Don't know what's causing this limitation but you might wanna try using varchar(max) or nvarchar(max). These store as many characters but allow also the regular string T-SQL functions (like LEN, SUBSTRING, REPLACE, RTRIM,...).
If you're able to convert the column, you might as well, since the text data type will be removed in a future version of SQL Server. See here.
The recommendation is to use varchar(MAX) or nvarchar(MAX). In your case, you could also use the XML data type, but that may tie you to certain database engines (if that's a consideration).
You should have a look at
XML Support in Microsoft SQL Server
2005
Beginning SQL Server 2005 XML
Programming
So I would rather try to use the data type appropriate for the use. Not make a datatype fit your use from a previous version.
Here's a little script I wrote for getting out all data
SELECT #data = N'huge data';
DECLARE #readSentence NVARCHAR (MAX) = N'';
DECLARE #dataLength INT = ( SELECT LEN (#data));
DECLARE #currIndex INT = 0;
WHILE #data <> #readSentence
BEGIN
DECLARE #temp NVARCHAR (MAX) = N'';
SET #temp = ( SELECT SUBSTRING (#data, #currIndex, 65535));
SELECT #temp;
SET #readSentence += #temp;
SET #currIndex += 65535;
END;