I have a table in SQL Server, with binary data stored as a string in a varchar(max) field.
The table name is attachment, and the field is named "documentbody".
select id, mimetype, documentbody
from attachment
The files stored in the table are mostly PDF, but also include JPG and PNG and probably some other file types too.
Here is a sample of what one of the "files" looks like, when queried (first 100 characters only):
JVBERi0xLjQKJeLjz9MNCjEgMCBvYmoKPDwvVHlwZSAvUGFnZQovUGFyZW50IDIgMCBSCi9NZWRpYUJveCBbIDAgMCA2MTIuMDAw
How can I convert this data into actual binary data?
When one wishes to convert data from one datatype to another, and no implicit convert exists, one uses either CAST or CONVERT.
e.g.
select cast(MyColumn as varbinary(max)), convert(varbinary(max), MyColumn)
from MyTable;
CAST is ANSI-SQL for what it is worth whereas CONVERT is SQL Server specific. However CONVERT handles many other cases including specific formatting, which CAST doesn't handle.
OK, taking a total guess here, many people encode binary data as base64 so try this:
SELECT CAST(CAST(N'' AS XML).value('xs:base64Binary(sql:column("MyColumn"))', 'VARBINARY(MAX)') AS VARCHAR(MAX))
FROM MyTable;
I've created one store procedure that returns the JSON. But it not return complete JSON only limited JSON returned. I used "For JSON auto" after Select statement. Have any Solution to get all JSON?
if you are using any Cast or covert operations. Use VARCHAR(MAX) instead of VARCHAR().
I have seen this issue in such cases.
Also if you are using the print output, then it is possible that the text might get truncated, but you can use the Select or Output parameter instead.
make the output parameter of type nvarchar(max)
or check this link :
Format Query Results as JSON with FOR JSON (SQL Server)
This might help
I found my solution here: https://learn.microsoft.com/en-us/sql/relational-databases/json/format-query-results-as-json-with-for-json-sql-server?view=sql-server-ver15#output-of-the-for-json-clause (same as above, but a specific section)
The issue was that I thought SQL Server was returning a single row/cell of JSON data because that is how SSMS displayed it. The truth is that it chops it into multiple rows.
I was retrieving the data in .NET using ExecuteScalar(), but I needed to use ExecuteReader(), and concatenate all rows together. Once I did that, I could deserialize the JSON without issue.
A piece of tsql code doesnt behave the same from production to Test environment. When the code below is executed on prod it brings back data
SELECT [col1xml]
FROM [DBName].[dbo].[Table1] (NOLOCK)
WHERE (cast([col1xml] as xml).value('(/Payment/****/trn1)[1]','nvarchar(20)') ='123456'))
However that same code brings back the below error when ran in Test.
Msg 9402, Level 16, State 1, Line 9
XML parsing: line 1, character 38, unable to switch the encoding
I have seen the fix provided by this site of conversion of UTF and this works in both prod and test. See below. However i need to provide an answer to the developers of why this behavior is occurring and a rationale why they should change their code(if that is the case)
WHERE CAST(
REPLACE(CAST(col1xml AS VARCHAR(MAX)), 'encoding="utf-16"', 'encoding="utf-8"')
AS XML).value('(/Payment/****/trn1)[1]','NVARCHAR(max)') ='123456')
I have compared both DB's and looked for anything obvious such as ansi nulls and ansi padding. Everything is the same and the version of SQL Server. This is SQL SERVER 2012 11.0.5388 version. Data between environments is different but the table schema is identical and the data type for col1xml is ntext.
In SQL Server you should store XML in a column typed XML. This native type has a got a lot of advantages. It is much faster and has implicit validity checks.
From your question I take, that you store your XML in NTEXT. This type is deprecated for centuries and will not be supported in future versions! You ought to change this soon!
SQL-Server knows two kinds of strings:
1 byte strings (CHAR or VARCHAR), which is extended ASCII
Important: This is not UTF-8! Native UTF-8 support will be part of a coming version.
2 byte string (NCHAR or NVARCHAR), which is UTF-16 (UCS-2)
If the XML has a leading declaration with an encoding (in most cases this is utf-8 or utf-16) you can get into troubles.
If the XML is stored as 2-byte-string (at least the NTEXT tells me this), the declaration has to be utf-16. With a 1-byte-string it should be utf-8.
The best (and easiest) was to ommit the declaration completely. You do not need it. Storing the XML in the appropriate type will kill this declaration automatically.
What you should do: Create a new column of type XML and shuffle all your XMLs to this column. Get rid of any TEXT, NTEXT and IMAGE columns you might have!
The next step is: Be happy and enjoy the fast and easy going with the native XML type :-D
UPDATE Differences in environment
You write: Data between environments is different
The error happens here:
cast([col1xml] as xml)
If your column would store the XML in the native type, you would not need a cast (which is very expensive!!) at all. But in your case this cast depends on the actual XML. As this is stored in NTEXT it is 2-byte-string. If your XML starts with a declaration stating a non-supported encoding (in most cases utf-8), this will fail.
Try this:
This works
DECLARE #xml2Byte_UTF16 NVARCHAR(100)='<?xml version="1.0" encoding="utf-16"?><root>test1</root>';
SELECT CAST(#xml2Byte_UTF16 AS XML);
DECLARE #xml1Byte_UTF8 VARCHAR(100)='<?xml version="1.0" encoding="utf-8"?><root>test2</root>';
SELECT CAST(#xml1Byte_UTF8 AS XML);
This fails
DECLARE #xml2Byte_UTF8 NVARCHAR(100)='<?xml version="1.0" encoding="utf-8"?><root>test3</root>';
SELECT CAST(#xml2Byte_UTF8 AS XML);
DECLARE #xml1Byte_UTF16 VARCHAR(100)='<?xml version="1.0" encoding="utf-16"?><root>test4</root>';
SELECT CAST(#xml1Byte_UTF16 AS XML);
Play around with VARCHAR and NVARCHAR and utf-8 and utf-16...
Our application updates and accesses data using SQL Server 2014.
I have a table of which the last column ('Contents') is created as VARCHAR(MAX).
We are using Delphi XE8, and are using a FireDAC TFDQuery component to update this column.
.....
FDquery.ParamByName('Contents').AsString:=Contents;
FDquery.ExecSQL;
When running this update, I get the following error:
Exception raised with message [FireDAC][Phys][ODBC]-345. Data too large for variable [CONTENTS]. Max len = [8002], actual len = [13829] Hint: set the TFDParam.Size to a greater value.
'Contents' can be string of varying length.
Browsing the web, the only reasonably simple solution that I found is changing the query as follows:
FDquery.ParamByName('Contents').AsWideMemo:=Contents;
FDquery.ExecSQL;
Is this acceptable, or should I be handling this differently?
There is nothing fancy with 'Contents', as I mentioned, it is simply a long string.
Is it acceptable to access a VARCHAR(MAX) type field parameter value by the AsWideMemo property?
Not especially. For VARCHAR(MAX) field parameter use AsMemo access. It is because you could be sending to your DBMS Unicode values to a non Unicode field. From the reference:
The Unicode encoded parameter value is converted to a Unicode
character set, which is supported by the DBMS, and sent to the DBMS.
This does not depend on a client character set or on a Delphi version.
If your field would be NVARCHAR(MAX), using AsWideMemo access to the parameter value would be the right choice.
Why am I getting “data too large for variable” error when having assigned a more than 8k chars long string by the AsString property?
Some background to why this happens. By accessing a certain parameter value by As<T> property, the engine also sets the parameter DataType, if you don't explicitly do that before. In this particular case you hinted the engine that it's fine for you if it sets the parameter data type to ftWideString or ftString just by accessing parameter value by the AsString property.
And thanks to data type mapping such parameter is treated as the VARCHAR[n] or NVARCHAR[n] data type including its limits (hence you got a string length limit error here).
Similar, just more specific data type hint is used when you access the parameter value by AsMemo property, it defaults to ftMemo data type which maps to VARCHAR(MAX). And as you may predict,
AsWideMemo access defaults to ftWideMemo which maps to NVARCHAR(MAX) data type. If you don't want to explicitly set the parameter data types but use this hinting, consult the manual to see how each used access property sets the default parameter data type.
I'm trying to insert into XML column (SQL SERVER 2008 R2), but the server's complaining:
System.Data.SqlClient.SqlException (0x80131904):
XML parsing: line 1, character 39, unable to switch the encoding
I found out that the XML column has to be UTF-16 in order for the insert to succeed.
The code I'm using is:
XmlSerializer serializer = new XmlSerializer(typeof(MyMessage));
StringWriter str = new StringWriter();
serializer.Serialize(str, message);
string messageToLog = str.ToString();
How can I serialize object to be in UTF-8 string?
EDIT: Ok, sorry for the mixup - the string needs to be in UTF-8. You were right - it's UTF-16 by default, and if I try to insert in UTF-8 it passes. So the question is how to serialize into UTF-8.
Example
This causes errors while trying to insert into SQL Server:
<?xml version="1.0" encoding="utf-16"?>
<MyMessage>Teno</MyMessage>
This doesn't:
<?xml version="1.0" encoding="utf-8"?>
<MyMessage>Teno</MyMessage>
Update
I figured out when the SQL Server 2008 for its Xml column type needs utf-8, and when utf-16 in encoding property of the xml specification you're trying to insert:
When you want to add utf-8, then add parameters to SQL command like this:
sqlcmd.Parameters.Add("ParamName", SqlDbType.VarChar).Value = xmlValueToAdd;
If you try to add the xmlValueToAdd with encoding=utf-16 in the previous row it would produce errors in insert. Also, the VarChar means that national characters aren't recognized (they turn out as question marks).
To add utf-16 to db, either use SqlDbType.NVarChar or SqlDbType.Xml in previous example, or just don't specify type at all:
sqlcmd.Parameters.Add(new SqlParameter("ParamName", xmlValueToAdd));
This question is a near-duplicate of 2 others, and surprisingly - while this one is the most recent - I believe it is missing the best answer.
The duplicates, and what I believe to be their best answers, are:
Using StringWriter for XML Serialization (2009-10-14)
https://stackoverflow.com/a/1566154/751158
Trying to store XML content into SQL Server 2005 fails (encoding problem) (2008-12-21)
https://stackoverflow.com/a/1091209/751158
In the end, it doesn't matter what encoding is declared or used, as long as the XmlReader can parse it locally within the application server.
As was confirmed in Most efficient way to read XML in ADO.net from XML type column in SQL server?, SQL Server stores XML in an efficient binary format. By using the SqlXml class, ADO.net can communicate with SQL Server in this binary format, and not require the database server to do any serialization or de-serialization of XML. This should also be more efficient for transport across the network.
By using SqlXml, XML will be sent pre-parsed to the database, and then the DB doesn't need to know anything about character encodings - UTF-16 or otherwise. In particular, note that the XML declarations aren't even persisted with the data in the database, regardless of which method is used to insert it.
Please refer to the above-linked answers for methods that look very similar to this, but this example is mine:
using System.Data;
using System.Data.SqlClient;
using System.Data.SqlTypes;
using System.IO;
using System.Xml;
static class XmlDemo {
static void Main(string[] args) {
using(SqlConnection conn = new SqlConnection()) {
conn.ConnectionString = "...";
conn.Open();
using(SqlCommand cmd = new SqlCommand("Insert Into TestData(Xml) Values (#Xml)", conn)) {
cmd.Parameters.Add(new SqlParameter("#Xml", SqlDbType.Xml) {
// Works.
// Value = "<Test/>"
// Works. XML Declaration is not persisted!
// Value = "<?xml version=\"1.0\"?><Test/>"
// Works. XML Declaration is not persisted!
// Value = "<?xml version=\"1.0\" encoding=\"UTF-16\"?><Test/>"
// Error ("unable to switch the encoding" SqlException).
// Value = "<?xml version=\"1.0\" encoding=\"UTF-8\"?><Test/>"
// Works. XML Declaration is not persisted!
Value = new SqlXml(XmlReader.Create(new StringReader("<?xml version=\"1.0\" encoding=\"UTF-8\"?><Test/>")))
});
cmd.ExecuteNonQuery();
}
}
}
}
Note that I would not consider the last (non-commented) example to be "production-ready", but left it as-is to be concise and readable. If done properly, both the StringReader and the created XmlReader should be initialized within using statements to ensure that their Close() methods are called when complete.
From what I've seen, the XML declarations are never persisted when using an XML column. Even without using .NET and just using this direct SQL insert statement, for example, the XML declaration is not saved into the database with the XML:
Insert Into TestData(Xml) Values ('<?xml version="1.0" encoding="UTF-8"?><Test/>');
Now in terms of the OP's question, the object to be serialized still needs to be converted into an XML structure from the MyMessage object, and XmlSerializer is still needed for this. However, at worst, instead of serializing to a String, the message could instead be serialized to an XmlDocument - which can then be passed to SqlXml through a new XmlNodeReader - avoiding a de-serialization/serialization trip to a string. (See http://blogs.msdn.com/b/jongallant/archive/2007/01/30/how-to-convert-xmldocument-to-xmlreader-for-sqlxml-data-type.aspx for details and an example.)
Everything here was developed against and tested with .NET 4.0 and SQL Server 2008 R2.
Please don't make waste by running XML through extra conversions (de-deserializations and serializations - to DOM, strings, or otherwise), as shown in other answers here and elsewhere.
Although a .net string is always UTF-16 you need to serialize the object using UTF-16 encoding.
That sould be something like this:
public static string ToString(object source, Type type, Encoding encoding)
{
// The string to hold the object content
String content;
// Create a memoryStream into which the data can be written and readed
using (var stream = new MemoryStream())
{
// Create the xml serializer, the serializer needs to know the type
// of the object that will be serialized
var xmlSerializer = new XmlSerializer(type);
// Create a XmlTextWriter to write the xml object source, we are going
// to define the encoding in the constructor
using (var writer = new XmlTextWriter(stream, encoding))
{
// Save the state of the object into the stream
xmlSerializer.Serialize(writer, source);
// Flush the stream
writer.Flush();
// Read the stream into a string
using (var reader = new StreamReader(stream, encoding))
{
// Set the stream position to the begin
stream.Position = 0;
// Read the stream into a string
content = reader.ReadToEnd();
}
}
}
// Return the xml string with the object content
return content;
}
By setting the encoding to Encoding.Unicode not only the string will be UTF-16 but you should also get the xml string as UTF-16.
<?xml version="1.0" encoding="utf-16"?>
Isn't the easiest solution to tell the serializer not to ouput the XML declaration? .NET and SQL should sort the rest out between them.
XmlSerializer serializer = new XmlSerializer(typeof(MyMessage));
StringWriter str = new StringWriter();
using (XmlWriter writer = XmlWriter.Create(str, new XmlWriterSettings { OmitXmlDeclaration = true }))
{
serializer.Serialize(writer, message);
}
string messageToLog = str.ToString();
It took me forever to re-solve this problem.
I was doing an INSERT statement into SQL Server as something like:
UPDATE Customers
SET data = '<?xml version="1.0" encoding="utf-16"?><MyMessage>Teno</MyMessage>';
and this gives the error:
Msg 9402, Level 16, State 1, Line 2
XML parsing: line 1, character 39, unable to switch the encoding
And the really, very simple fix is to:
UPDATE Customers
SET data = N'<?xml version="1.0" encoding="utf-16"?><MyMessage>Teno</MyMessage>';
The difference is prefixing the Unicode string with N:
N'<?xml version="1.0" encoding="utf-16"?>Teno</MyMessage>'
In the former case, an unprefixed string is assumed to be varchar (e.g. Windows-1252 code-page). When it encounters the encoding="utf-16" inside the string, there is a conflict (and rightly so, since the string isn't utf-16).
The fix is to pass the string to SQL server as an nvarchar (i.e. UTF-16):
N'<?xml version="1.0" encoding="utf-16"?>'
That way the string is UTF-16, which matches the utf-16 encoding that the XML says it is. The carpet matches the curtains, so to speak.
#ziesemer's answer (above) is the only fully correct answer to this question and the linked duplicates of this question. However, it could still use a little more explanation and some clarification. Consider this as an extension of #ziesemer's answer.
Even if they produce the desired result, most answers to this question (including the duplicate question) are convoluted and go through many unnecessary steps. The main issue here is the overall lack of understanding regarding how the XML datatype actually works in SQL Server (not surprising given that it isn't well documented). The XML type:
Is a highly optimized (for storage) type that converts the incoming XML into a binary format (which is documented somewhere in the msdn site). The optimizations include:
Converting numbers and dates from string (as they are in the XML) into binary representations IF the element or attribute is tagged with the type info (this might require specifying an XML Schema Collection). Meaning, the number "1234567" is stored as a 4-byte "int" instead of a 14-byte UTF-16 string of 7 digits.
Element and Attribute names are stored in a dictionary and given a numeric ID. That numeric ID is used in the XML tree structure. Meaning, "<ElementName>...</ElementName>" takes up 27 character (i.e. 54 bytes) in string form, but only 11 characters (i.e. 22 bytes) when stored in the XML type. And that is for a single instance of it. Multiple instances take up additional multiples of the 54 bytes. But in the XML type, each instance only takes up the space of that numeric ID, most likely a 4-byte int.
Stores strings as UTF-16 Little Endian, always. This is most likely why the XML declaration is not stored: it is entirely unnecessary as it is always the same since the "Encoding" attribute cannot ever change.
No XML declaration assumes the encoding to be UTF-16, not UTF-8.
Can have 8-bit / non-UTF-16 data passed in. In this case, you need to make sure that the string is not an NVARCHAR string (i.e. not prefixed with an upper-case "N" for literals, not declared as NVARCHAR when dealing with T-SQL variables, and not declared as SqlDbType.NVarChar in .NET). AND, you need to make sure that you do have the XML declaration, and that it specifies the correct encoding.
PRINT 'VARCHAR / UTF-8:';
DECLARE #XML_VC_8 XML;
SET #XML_VC_8 = '<?xml version="1.0" encoding="utf-8"?><test/>';
PRINT 'Success!'
-- Success!
GO
PRINT '';
PRINT 'NVARCHAR / UTF-8:';
DECLARE #XML_NVC_8 XML;
SET #XML_NVC_8 = N'<?xml version="1.0" encoding="utf-8"?><test/>';
PRINT 'Success!'
/*
Msg 9402, Level 16, State 1, Line XXXXX
XML parsing: line 1, character 38, unable to switch the encoding
*/
GO
PRINT '';
PRINT 'VARCHAR / UTF-16:';
DECLARE #XML_VC_16 XML;
SET #XML_VC_16 = '<?xml version="1.0" encoding="utf-16"?><test/>';
PRINT 'Success!'
/*
Msg 9402, Level 16, State 1, Line XXXXX
XML parsing: line 1, character 38, unable to switch the encoding
*/
GO
PRINT '';
PRINT 'NVARCHAR / UTF-16:';
DECLARE #XML_NVC_16 XML;
SET #XML_NVC_16 = N'<?xml version="1.0" encoding="utf-16"?><test/>';
PRINT 'Success!'
-- Success!
As you can see, when the input string is NVARCHAR, then the XML declaration can be included, but it needs to be "UTF-16".
When the input string is VARCHAR then the XML declaration can be included, but it cannot be "UTF-16". It can, however, be any valid 8-bit encoding, in which case the bytes for that encoding will be converted into UTF-16, as shown below:
DECLARE #XML XML;
SET #XML = '<?xml version="1.0" encoding="utf-8"?><test attr="'
+ CHAR(0xF0) + CHAR(0x9F) + CHAR(0x98) + CHAR(0x8E) + '"/>';
SELECT #XML;
-- <test attr="😎" />
SET #XML = '<?xml version="1.0" encoding="Windows-1255"?><test attr="'
+ CONVERT(VARCHAR(10), 0xF9ECE5ED) + '"/>';
SELECT #XML AS [XML from Windows-1255],
CONVERT(VARCHAR(10), 0xF9ECE5ED) AS [Latin1_General / Windows-1252];
/*
XML from Windows-1255 Latin1_General / Windows-1252
<test attr="שלום" /> ùìåí
*/
The first example specifies the 4-byte UTF-8 sequence for Smiling Face with Sunglasses and it get converted correctly.
The second example uses 4 bytes to represent 4 Hebrew letters making up the word "Shalom", which is converted correctly, and displayed correctly given that the "F9" byte, which is first, is the ש character, which is on the right-side of the word (since Hebrew is a right-to-left language). Yet those same 4 bytes display as ùìåí when selected directly since the default Collation for the current DB is Latin1_General_100_CS_AS_SC.
A string is always UTF-16 in .NET, so as long as you stay inside your managed app you don't have to care about which encoding it is.
The problem is more likely where you talk to the SQL server. Your question doesn't show that code so it's hard to pin point the exact error. My suggestion is you check if there's a property or attribute you can set on that code that specifies the encoding of the data sent to the server.
You are serializing to a string rather than a byte array so, at this point, any encoding hasn't happened yet.
What does the start of "messageToLog" look like? Is the XML specifying an encoding (e.g. utf-8) which subsequently turns out to be wrong?
Edit
Based on your further info it sounds like the string is automatically converted to utf-8 when it is passed to the database, but the database chokes because the XML declaration says it is utf-16.
In which case, you don't need to serialize to utf-8. You need to serialize with the "encoding=" omitted from the XML. The XmlFragmentWriter (not a standard part of .Net, Google it) lets you do this.
Default encoding for a xml serializer should be UTF-16. Just to make sure you can try -
XmlSerializer serializer = new XmlSerializer(typeof(YourObject));
// create a MemoryStream here, we are just working
// exclusively in memory
System.IO.Stream stream = new System.IO.MemoryStream();
// The XmlTextWriter takes a stream and encoding
// as one of its constructors
System.Xml.XmlTextWriter xtWriter = new System.Xml.XmlTextWriter(stream, Encoding.UTF16);
serializer.Serialize(xtWriter, yourObjectInstance);
xtWriter.Flush();