I want to update a varbinary column with base64 string (an image) to bytes in SQL Server.
Does anyone know how to do it?
I need to have the image in bytes.
Thanks!
You can use the XML Query function xs:base64Binary() to encode and decode Base64 data for you:
When given varbinary(max) input it returns a varchar(max) result, i.e.: Base64 encoded data.
When given varchar(max) input it returns a varbinary(max) result. If the input isn't valid Base64 encoded data it returns NULL.
e.g.:
create table dbo.Demo (
ID int not null identity(1,1),
ImageData varbinary(max)
);
insert dbo.Demo (ImageData) values (null);
declare #Base64Data varchar(max) = 'LzlqLzRB';
update dbo.Demo
set ImageData = cast('' as xml).value('xs:base64Binary(sql:variable("#Base64Data"))', 'varbinary(max)')
where ID = 1;
select * from dbo.Demo;
=====
Edit: if you have already stored your base64 data into varbinary(max) you will need to cast it to varchar(max) before supplying it to xs:base64Binary(), e.g.:
create table dbo.Demo (
ID int not null identity(1,1),
ImageData varbinary(max)
);
insert dbo.Demo (ImageData) values ( cast('LzlqLzRB' as varbinary(max)) );
select * from dbo.Demo; -- base64 characters as varbinary
update Dmo
set ImageData = cast('' as xml).value('xs:base64Binary(sql:column("Base64Data"))', 'varbinary(max)')
from dbo.Demo Dmo
outer apply ( select Base64Data = cast(ImageData as varchar(max)) ) Cst
where ID = 1;
select * from dbo.Demo; -- decoded data as varbinary
Related
I have extracted text data from a rtf file, there are string in CP 1049
lang1049\''ea\''e0\''f6\''e0\''ef\''e8
I have tried to decode it to a string but received not string what I know but other characters
declare #out table (id int not null identity, string varchar(128) collate Cyrillic_General_CI_AS)
insert into #out(string) select (char(0xEA)+char(0xE0)+char(0xF6)+char(0xE0)+char(0xEF)+char(0xE8))
select * from #out
GO
The string should be 'кацапи' .
How to do it correctly?
You can do this using NCHAR()
NCHAR
insert into #out(string) select NCHAR(0xEA)+NCHAR(0xE0)+NCHAR(0xF6)+NCHAR(0xE0)+NCHAR(0xEF)+NCHAR(0xE8)
Alternatively, for UNICODE chars, you can use convert()
insert into #out(string) select convert(varchar(128), 0xEA)+convert(varchar(128), 0xE0)+convert(varchar(128), 0xF6)+convert(varchar(128), 0xE0)+convert(varchar(128), 0xEF)+convert(varchar(128), 0xE8)
I have an encrypted varbinary(MAX) field in my DB called ACCT_FName_encrypt.
I can successfully decrypt this field with:
CONVERT(nvarchar(MAX), DecryptByKey(ACCT_FName_encrypt)) AS 'ACCT_FName_Denc'
But if I try to decrypt the actual value from the column, I get NULL:
CONVERT(nvarchar(MAX), DecryptByKey('0x001D25D87D3D8E49A97863ADC4958E790100000021E26DD2305384AE49EC9329EF2AF8758134F7C946EC9FE024805B8DF21472C4545D461DA9F2B7F96094C2AED09BF4A9')) AS 'ACCT_FName_Denc'
How can I get the decrypted value from the straight varbinary, without calling the field?
It should not be passed as string and it needs to be cast to the original type after. Full, working example:
OPEN SYMMETRIC KEY StackOverflow
DECRYPTION BY PASSWORD = 'pass123_#pass123_#'
GO
DECLARE #ColumnValue NVARCHAR(MAX);
DECLARE #EncrpytionValue VARBINARY(8000);
SET #ColumnValue = REPLICATE (N'A', 12)
SET #EncrpytionValue = ENCRYPTBYKEY( KEY_GUID('StackOverflow'), #ColumnValue )
SELECT #EncrpytionValue
SELECT CONVERT(NVARCHAR(MAX), DECRYPTBYKEY(#EncrpytionValue));
SELECT CONVERT(NVARCHAR(MAX), DECRYPTBYKEY(0x00B08017838E6C48889DD12542E4C52002000000A8C910DA1CBFFE30E446358940177F03F912EE36FACF91FA2044BE5C75C9AA69BC15E6425DE52C2A193BA13AEDA90AE2276C244E56692B75CB2D4FDEC8D596F9));
--DROP SYMMETRIC KEY StackOverflow;
and in your code it will be just:
CONVERT(nvarchar(MAX), DecryptByKey(0x001D25D87D3D8E49A97863ADC4958E790100000021E26DD2305384AE49EC9329EF2AF8758134F7C946EC9FE024805B8DF21472C4545D461DA9F2B7F96094C2AED09BF4A9)) AS 'ACCT_FName_Denc'
Try converting the value to Varbinary
select convert(varbinary, '0x001D25D87D3D8E49A97863ADC4958E790100000021E26DD2305384AE49EC9329EF2AF8758134F7C946EC9FE024805B8DF21472C4545D461DA9F2B7F96094C2AED09BF4A9')
Is there a simple way to convert a utf-8 encoded varbinary(max) column to varchar(max) in T-SQL. Something like CONVERT(varchar(max), [MyDataColumn]). Best would be a solution that does not need custom functions.
Currently, i convert the data on the client side, but this has the downside, that correct filtering and sorting is not as efficient as done server-side.
XML trick
Following solution should work for any encoding.
There is a tricky way of doing exactly what the OP asks. Edit: I found the same method discussed on SO (SQL - UTF-8 to varchar/nvarchar Encoding issue)
The process goes like this:
SELECT
CAST(
'<?xml version=''1.0'' encoding=''utf-8''?><![CDATA[' --start CDATA
+ REPLACE(
LB.LongBinary,
']]>', --we need only to escape ]]>, which ends CDATA section
']]]]><![CDATA[>' --we simply split it into two CDATA sections
) + ']]>' AS XML --finish CDATA
).value('.', 'nvarchar(max)')
Why it works: varbinary and varchar are the same string of bits - only the interpretation differs, so the resulting xml truly is utf8 encoded bitstream and the xml interpreter is than able to reconstruct the correct utf8 encoded characters.
BEWARE the 'nvarchar(max)' in the value function. If you used varchar, it would destroy multi-byte characters (depending on your collation).
BEWARE 2 XML cannot handle some characters, i.e. 0x2. When your string contains such characters, this trick will fail.
Database trick (SQL Server 2019 and newer)
This is simple. Create another database with UTF8 collation as the default one. Create function that converts VARBINARY to VARCHAR. The returned VARCHAR will have that UTF8 collation of the database.
Insert trick (SQL Server 2019 and newer)
This is another simple trick. Create a table with one column VARCHAR COLLATE ...UTF8. Insert the VARBINARY data into this table. It will get saved correctly as UTF8 VARCHAR. It is sad that memory optimized tables cannot use UTF8 collations...
Alter table trick (SQL Server 2019 and newer)
(don't use this, it is unnecessary, see Plain insert trick)
I was trying to come up with an approach using SQL Server 2019's Utf8 collation and I have found one possible method so far, that should be faster than the XML trick (see below).
Create temporary table with varbinary column.
Insert varbinary values into the table
Alter table alter column to varchar with utf8 collation
drop table if exists
#bin,
#utf8;
create table #utf8 (UTF8 VARCHAR(MAX) COLLATE Czech_100_CI_AI_SC_UTF8);
create table #bin (BIN VARBINARY(MAX));
insert into #utf8 (UTF8) values ('Žluťoučký kůň říčně pěl ďábelské ódy za svitu měsíce.');
insert into #bin (BIN) select CAST(UTF8 AS varbinary(max)) from #utf8;
select * from #utf8; --here you can see the utf8 string is stored correctly and that
select BIN, CAST(BIN AS VARCHAR(MAX)) from #bin; --utf8 binary is converted into gibberish
alter table #bin alter column BIN varchar(max) collate Czech_100_CI_AI_SC_UTF8;
select * from #bin; --voialá, correctly converted varchar
alter table #bin alter column BIN nvarchar(max);
select * from #bin; --finally, correctly converted nvarchar
Speed difference
The Database trick together with the Insert trick are the fastest ones.
The XML trick is slower.
The Alter table trick is stupid, don't do it. It loses out when you change lots of short texts at once (the altered table is large).
The test:
first string contains one replace for the XML trick
second string is plain ASCII with no replaces for XML trick
#TextLengthMultiplier determines length of the converted text
#TextAmount determines how many of them at once will be converted
------------------
--TEST SETUP
--DECLARE #LongText NVARCHAR(MAX) = N'český jazyk, Tiếng Việt, русский язык, 漢語, ]]>';
--DECLARE #LongText NVARCHAR(MAX) = N'JUST ASCII, for LOLZ------------------------------------------------------';
DECLARE
#TextLengthMultiplier INTEGER = 100000,
#TextAmount INTEGER = 10;
---------------------
-- TECHNICALITIES
DECLARE
#StartCDATA DATETIME2(7), #EndCDATA DATETIME2(7),
#StartTable DATETIME2(7), #EndTable DATETIME2(7),
#StartDB DATETIME2(7), #EndDB DATETIME2(7),
#StartInsert DATETIME2(7), #EndInsert DATETIME2(7);
drop table if exists
#longTexts,
#longBinaries,
#CDATAConverts,
#DBConverts,
#INsertConverts;
CREATE TABLE #longTexts (LongText VARCHAR (MAX) COLLATE Czech_100_CI_AI_SC_UTF8 NOT NULL);
CREATE TABLE #longBinaries (LongBinary VARBINARY(MAX) NOT NULL);
CREATE TABLE #CDATAConverts (LongText VARCHAR (MAX) COLLATE Czech_100_CI_AI_SC_UTF8 NOT NULL);
CREATE TABLE #DBConverts (LongText VARCHAR (MAX) COLLATE Czech_100_CI_AI_SC_UTF8 NOT NULL);
CREATE TABLE #InsertConverts (LongText VARCHAR (MAX) COLLATE Czech_100_CI_AI_SC_UTF8 NOT NULL);
insert into #longTexts --make the long text longer
(LongText)
select
REPLICATE(#LongText, #TextLengthMultiplier)
from
TESTES.dbo.Numbers --use while if you don't have number table
WHERE
Number BETWEEN 1 AND #TextAmount; --make more of them
insert into #longBinaries (LongBinary) select CAST(LongText AS varbinary(max)) from #longTexts;
--sanity check...
SELECT TOP(1) * FROM #longTexts;
------------------------------
--MEASURE CDATA--
SET #StartCDATA = SYSDATETIME();
INSERT INTO #CDATAConverts
(
LongText
)
SELECT
CAST(
'<?xml version=''1.0'' encoding=''utf-8''?><![CDATA['
+ REPLACE(
LB.LongBinary,
']]>',
']]]]><![CDATA[>'
) + ']]>' AS XML
).value('.', 'Nvarchar(max)')
FROM
#longBinaries AS LB;
SET #EndCDATA = SYSDATETIME();
--------------------------------------------
--MEASURE ALTER TABLE--
SET #StartTable = SYSDATETIME();
DROP TABLE IF EXISTS #AlterConverts;
CREATE TABLE #AlterConverts (UTF8 VARBINARY(MAX));
INSERT INTO #AlterConverts
(
UTF8
)
SELECT
LB.LongBinary
FROM
#longBinaries AS LB;
ALTER TABLE #AlterConverts ALTER COLUMN UTF8 VARCHAR(MAX) COLLATE Czech_100_CI_AI_SC_UTF8;
--ALTER TABLE #AlterConverts ALTER COLUMN UTF8 NVARCHAR(MAX);
SET #EndTable = SYSDATETIME();
--------------------------------------------
--MEASURE DB--
SET #StartDB = SYSDATETIME();
INSERT INTO #DBConverts
(
LongText
)
SELECT
FUNCTIONS_ONLY.dbo.VarBinaryToUTF8(LB.LongBinary)
FROM
#longBinaries AS LB;
SET #EndDB = SYSDATETIME();
--------------------------------------------
--MEASURE Insert--
SET #StartInsert = SYSDATETIME();
INSERT INTO #INsertConverts
(
LongText
)
SELECT
LB.LongBinary
FROM
#longBinaries AS LB;
SET #EndInsert = SYSDATETIME();
--------------------------------------------
-- RESULTS
SELECT
DATEDIFF(MILLISECOND, #StartCDATA, #EndCDATA) AS CDATA_MS,
DATEDIFF(MILLISECOND, #StartTable, #EndTable) AS ALTER_MS,
DATEDIFF(MILLISECOND, #StartDB, #EndDB) AS DB_MS,
DATEDIFF(MILLISECOND, #StartInsert, #EndInsert) AS Insert_MS;
SELECT TOP(1) '#CDATAConverts ', * FROM #CDATAConverts ;
SELECT TOP(1) '#DBConverts ', * FROM #DBConverts ;
SELECT TOP(1) '#INsertConverts', * FROM #INsertConverts;
SELECT TOP(1) '#AlterConverts ', * FROM #AlterConverts ;
SQL-Server does not know UTF-8 (at least all versions you can use productivly). There is limited support starting with v2014 SP2 (and some details about the supported versions)
when reading an utf-8 encoded file from disc via BCP (same for writing content to disc).
Important to know:
VARCHAR(x) is not utf-8. It is 1-byte-encoded extended ASCII, using a codepage (living in the collation) as character map.
NVARCHAR(x) is not utf-16 (but very close to it, it's ucs-2). This is a 2-byte-encoded string covering almost any known characters (but exceptions exist).
utf-8 will use 1 byte for plain latin characters, but 2 or even more bytes to encoded foreign charsets.
A VARBINARY(x) will hold the utf-8 as a meaningless chain of bytes.
A simple CAST or CONVERT will not work: VARCHAR will take each single byte as a character. For sure this is not the result you would expect. NVARCHAR would take each chunk of 2 bytes as one character. Again not the thing you need.
You might try to write this out to a file and read it back with BCP (v2014 SP2 or higher). But the better chance I see for you is a CLR function.
you can use the following to post string into varbinary field
Encoding.Unicode.GetBytes(Item.VALUE)
then use the following to retrive data as string
public string ReadCString(byte[] cString)
{
var nullIndex = Array.IndexOf(cString, (byte)0);
nullIndex = (nullIndex == -1) ? cString.Length : nullIndex;
return System.Text.Encoding.Unicode.GetString(cString);
}
I'm curently storing a list of ids in a column as a CSV string value ('1;2;3').
I'd like to optimize with a better approach (I believe) which would use varbinary(max).
I'm looking for tsql functions
1 . That would merge side by side a set of integer rows into a varbinary(max)
2 . That would split the varbinary field into a set of int rows
Any tips appreciated, thanks
Solution is very questionable. I'd also suggest to normalize the data.
However, if you still want to store your data as VARBINARY, here is the solution:
CREATE FUNCTION dbo.fn_String_to_Varbinary(#Input VARCHAR(MAX))
RETURNS VARBINARY(MAX) AS
BEGIN
DECLARE #VB VARBINARY(MAX);
WITH CTE as (
SELECT CAST(CAST(LEFT(IT,CHARINDEX(';',IT)-1) AS INT) as VARBINARY(MAX)) as VB, RIGHT(IT,LEN(IT) - CHARINDEX(';',IT)) AS IT
FROM (VALUES (#Input)) as X(IT) union all
SELECT VB + CAST(CAST(LEFT(IT,CHARINDEX(';',IT)-1) AS INT) as VARBINARY(MAX)) as VB, RIGHT(IT,LEN(IT) - CHARINDEX(';',IT)) AS IT FROM CTE WHERE LEN(IT) > 1
)
SELECT TOP 1 #VB = VB FROM CTE
ORDER BY LEN(VB) DESC
RETURN #VB
END
GO
DECLARE #Input VARCHAR(MAX) = '421;1;2;3;5000;576;842;375;34654322;18;67;000001;1232142334;'
DECLARE #Position INT = 9
DECLARE #VB VARBINARY(MAX)
SELECT #VB = dbo.fn_String_to_Varbinary(#Input)
SELECT #VB, CAST(SUBSTRING(#VB,4*(#Position-1)+1,4) AS INT)
GO
The function converts string into VARBINARY and then script extracts 9th number from that VARBINARY value.
Do not run this function against a data set with million records and million numbers in each line.
i've data in XML like
<Values>
<Id>7f8a5d20-d171-42f5-a222-a01b5186a048</Id>
<DealAttachment>
<AttachmentId>deefff3f-f63e-4b4c-8e76-68b6db476628</AttachmentId>
<IsNew>true</IsNew>
<IsDeleted>false</IsDeleted>
<DealId>7f8a5d20-d171-42f5-a222-a01b5186a048</DealId>
<AttachmentName>changes2</AttachmentName>
<AttachmentFile>991049711010310132116104101321011099710510832115117981061019911645100111110101131011711210097116101329811110012132116101120116321151171031031011151161011003298121326610510810845100111110101131010011132110111116321151011101003210110997105108321051103299971151013211110232117112100971161011003299108105101110116115329811711632115101110100321051103299971151013211110232100101108101116101100329910810510111011611545321001111101011310991141019711610132973211510111297114971161013211697981081013211611132991111101169710511032117115101114115321161113211910411110932101109971051083211910510810832981013211510111011645100111110101131011611411711099971161013211610410132108111103321169798108101329710211610111432111110101321091111101161044510011111010113108411497110115108971161051111101154532100111110101131097100100321129711497109101116101114321161113210611198321161113211411711032115105108101110116108121451001011089712110110013101310131013101310131010511010211111410932981051081083210511032999711510132981111161043211711210097116105111110471001011081011161051111104432117112100971161051111103211910510810832110111116329810132105110102111114109101100</AttachmentFile>
<AttachmentType>.txt</AttachmentType>
</DealAttachment>
</Values>
where AttachmentFile is varbinary(max)
DECLARE #AttachmentId uniqueidentifier,
#DealId uniqueidentifier,
#IsNew bit,
#IsDeleted bit,
#AttachmentName varchar(100),
#AttachmentFile varbinary(max),
#AttachmentType varchar(50)
SET #DealId = #SubmitXml.value('(Values/Id/node())[1]', 'uniqueidentifier')
SET #AttachmentId = #SubmitXml.value('(Values/DealAttachment/AttachmentId/node())[1]', 'uniqueidentifier')
SET #IsNew = #SubmitXml.value('(Values/DealAttachment/IsNew/node())[1]', 'bit')
SET #IsDeleted = #SubmitXml.value('(Values/DealAttachment/IsDeleted/node())[1]', 'bit')
SET #AttachmentName = #SubmitXml.value('(Values/DealAttachment/AttachmentName/node())[1]', 'varchar(100)')
SET #AttachmentFile = #SubmitXml.value('(Values/DealAttachment/AttachmentFile/node())[1]', 'varbinary(max)')
SET #AttachmentType = #SubmitXml.value('(Values/DealAttachment/AttachmentType/node())[1]', 'varchar(50)')
But, after above statement #AttachmentFile is NULL or blankspace.
Binary data types in SQL Server (including varbinary) are represented as hexadecimal in queries which read and write them.
I think the problem here is that, rather than writing to the database directly from the byte stream (as in the example you linked to in your comment, which would implicitly cast the byte array to a hexadecimal value), it's being written to an intermediate XML block. When the data is written to the XML block, it looks like the byte stream is being converted to a string made up of concatenated list of integers of the byte values in decimal. Because the byte values are not delimited, it might not be possible to reconstruct the original data from this stream.
If you need to use an intermediate XML file, a more conventional approach would be to encode the file data as Base64 in the XML block (as discussed in this question - and doubtless many others). You could then decode it using the xs:base64Binary function:
SET #AttachmentFile = #SubmitXml.value('xs:base64Binary((Values/DealAttachment/AttachmentFile/node())[1])','varbinary(max)')
#Ed is correct in that you have somehow stored the decimal ASCII values for each character instead of the hex value for each. You can see that by decoding each one:
SELECT CHAR(99) + CHAR(104) + CHAR(97) + CHAR(110) + CHAR(103) + CHAR(101) +
CHAR(32) + CHAR(116) + CHAR(104) + CHAR(101);
-- change the
But as you can also see, there is no way to decode that programmatically because it is a mix of 2 digit and 3 digit values.
If you really were storing hex bytes in that XML element, you could turn it into VARBINARY(MAX) having the same binary bytes by first extracting it from the XML as a plain VARCHAR(MAX) and then converting it to VARBINARY(MAX) using the CONVERT built-in function, specifying a "style" of 2.
SELECT CONVERT(VARBINARY(MAX), 'this is a test, yo!');
-- 0x74686973206973206120746573742C20796F21
DECLARE #SomeXML XML = N'
<Values>
<DealAttachment>
<AttachmentFile>74686973206973206120746573742C20796F21</AttachmentFile>
</DealAttachment>
</Values>';
SELECT CONVERT(VARBINARY(MAX),
#SomeXML.value('(/Values/DealAttachment/AttachmentFile/text())[1]',
'VARCHAR(MAX)'),
2)
-- 0x74686973206973206120746573742C20796F21 (but as VARBINARY)
However, that all being said, ideally you would just Base64 encode the binary file on the way in (also as mentioned by #Ed) using Convert.ToBase64String (since you already have a byte[]).