I'm looking for a way for retrieving Entity Data Model (EDM) from __MigrationHistory table using only T-SQL (so anyone, using Microsoft SQL Server Management Studio only, could do the same).
I want to have a valid BASE64 string value.
I don't want to fully decompress it to EDMX.
I don't want to get it from *.resx migration file.
When I select whole __MigrationHistory in SSMS, Model column value is like following
0x1F8B0800000000000400CD57DB6EDB38107D5F60FF81E0D31648C5247DD906528BD4898BA075525469DF6969EC10CB8BCA4B607F5B1FFA49FB0B3BBACB521CB7E9A228020422357366E6F0CC50FEF7EBB7F8F54649720FD609A3137A121D53023A33B9D0EB8406BF7AFE377DFDEACF3FE2CB5C6DC8E7D6EE4569879EDA25F4CEFBE28C3197DD81E22E5222B3C699958F32A318CF0D3B3D3E7EC94E4E182004452C42E28F417BA1A05AE072667406850F5C2E4C0ED235FBF826AD50C93557E00A9E414253AE0A0917DCF325771035CB254278D8784ACEA5E098550A724509D7DA78EE31E7B34F0E526F8D5EA7056E7079BB2D00ED565C3A686A39EBCDBFB7ACE3D3B22CD63BB6505970DEA81F043C79D1F0C4C6EE4F629B763C229397C8B8DF9655576C26746E0C25E3386733694B9B433C47159C001721CC11D9353EEA84827A2AFF8EC82C481F2C241A82B75C1E910F612945F60EB6B7E61FD0890E520ED3C584F1DDCE066E7DB0A600EBB71F61D514719553C276FDD8D8B1731BF8D435BE0D029FAF31365F4AE8C4C01E752FFFB700A8286C144A167CF31EF4DADF25141F29998B0DE4ED4E83FA490BEC2B74F2368C82C4AC3F9DE999959473A1C136094C243F3944D47A738EAE09B75B518D9B82EFA5E028E973A8DB27AA14F2509E5D467DA3B2BA53DB8E667B5A3A5EF0A240CA062DDEEC90B4EEEFD9F3F4C7C5AE6A0C96B90734DF65DB45F2C6F2358CDE6268CC742EACF3AD902999E56A6236E67F0FB76DB401C5633DF78CB7C6E5F3CE29EFEBBFF26C46703D97732C4F81F655A5D06554F7FCC4AB1AB35C72FB409FCC8C0C4AEFEBB5C7BCEB3619FAD73B5384988DD21ED3C4263C8D3A7ECCF963921D9B74D13BE98E241A3772397C354DF4539B5082D4DC8BBCD2CED67950516910A55FE44C0AACB73758702D56E07C3D1429DE08A7A31BEDF7B95D9873B93C7CC5FCF2A11EB4F81200D9C46C5602EC4F0D787DCF6D76C7ED5F8A6F9E0D917E76884F67CDC111BD6F42D72A4B68BEC4C56D9D5E3DD19F38BBA7828FD9F08B2DBE0027D63D44F9FDA6212B95D483B636577A655A82B19C6146ADC988FF05789E232BE7160F90671E5F67E05C75D17EE632A0C9A55A427EA56F822F823F770ED4526E87F5C6ECF1F8D505B59B737C53942BF77F9480690A2C016EF49B2064DEE53D9FEA6F1F442990B780FB5587E28706C2ADB71DD2B5D1DF09D4D0770105E8B2296E016F120473373AE5F7F094DCF0FBE23DAC79B66DE7D67E90C307B14B7B7C21F8DA72E51A8CDEBFFC15C2CA9F21AFFE038574340FB80C0000
This is my sample database, so I don't mind sharing it.
The column is of type varbinary(max), I googled how can convert it to UTF8*-ish* text.
varbinary to string on SQL Server
SQL Server: Convert a string into a hex string and back #sql #sqlserver
For
SELECT CONVERT(VARCHAR(max), 0x1F8B0800000000000400CD57DB6EDB3810..... , 0)
or even more straightforward
SELECT CONVERT(VARCHAR(max), (SELECT TOP (1) [t0].[Model]FROM [__MigrationHistory] AS [t0]), 0);
I get the following result
literally this: ‹
I checked my Model if it's not broken or incomplete in both LinqPad5 and using this tool found here, but it looks ok.
Finally found this article:
https://social.technet.microsoft.com/wiki/contents/articles/36388.transact-sql-convert-varbinary-to-base64-string-and-vice-versa.aspx#Convert_VARBINARY_to_Base64_String
So, running the query gets what I wanted, valid Base64.
Using XML and the hint "for xml path"
select Model, baze64
from __MigrationHistory
cross apply (select Model as '*' for xml path('')) T (baze64)
Other presented queries in article will also work
Using XML XQuery
Using JSON
Based on your solution I'm sharing the code for 2 scalar function for conversion in both direction:
Base64Decode
CREATE FUNCTION [dbo].[fnBase64ToBinary]
(
#Str AS NVARCHAR(MAX)
)
RETURNS VARBINARY(MAX)
AS
BEGIN
RETURN (
SELECT
CONVERT(
VARBINARY(MAX), CAST('' AS XML).value('xs:base64Binary(sql:column("BASE64_COLUMN"))', 'VARBINARY(MAX)')
)
FROM (SELECT #Str AS BASE64_COLUMN) A
);
END;
Base64 encode
CREATE FUNCTION [dbo].[fnBinaryToBase64]
(
#Var AS VARBINARY(MAX)
)
RETURNS NVARCHAR(MAX)
AS
BEGIN
RETURN (
SELECT #Var AS '*' FOR XML PATH('')
);
END;
It's apparently gzip'd xml. eg
SELECT MigrationId
,ContextKey
,cast(decompress(model) as xml) model
FROM __MigrationHistory
You can Convert binary to Base64 using XML and the hint "for xml path"
select Column1,ColumnWithBinary,ColumnToSwFinalResult
from TableWithBinary
cross apply (select ColumnWithBinaryas '*' for xml path('')) T (ColumnToSwFinalResult)
GO
Necromancing.
You can also use for JSON PATH similar to how for XML PATH is used:
SELECT
T_AP_Dokumente.DK_UID
,T_AP_Dokumente.DK_Thumbnail
,tBase64.JSON_Thumbnail
,NULLIF(T.XML_Thumbnail, '') AS base64
,COMPRESS(T_AP_Dokumente.DK_Thumbnail) AS gzipped
,DATALENGTH(COMPRESS(T_AP_Dokumente.DK_Thumbnail)) AS dlt_datalength_gzipped
,DATALENGTH(T_AP_Dokumente.DK_Thumbnail) AS dlt_datalength
,DATALENGTH(DECOMPRESS(COMPRESS(T_AP_Dokumente.DK_Thumbnail))) AS should_equal_dlt_datalength
FROM T_AP_Dokumente
CROSS APPLY
(
SELECT T_AP_Dokumente.DK_Thumbnail AS '*' FOR XML PATH('')
) AS T(XML_Thumbnail)
CROSS APPLY
(
SELECT * FROM
OPENJSON
(
(
SELECT T_AP_Dokumente.DK_Thumbnail AS JSON_Thumbnail
FOR JSON PATH
)
) WITH(JSON_Thumbnail varchar(MAX)) AS t
) AS tBase64
Related
I got a Varchar column which holds XML. I would like to convert them to JSON so I did a quick test.
Here is what worked so far:
DECLARE #xml XML
SET #XML = '<content>
<value>123456</value>
</content>'
SELECT
'{"foo":"' + b.value('(./value)[1]', 'VARCHAR(50)') + '"}'
FROM
#xml.nodes('/content') AS a(b)
Result:
{"foo":"123456"}
So far so good, but this does not do half the job that has to be done.
I have lots of rows, and I want to convert all rows at once.
I have multiple values inside a content tag and therefore I cannot just add a json prefix and suffix since inside that array there will be more than one ID.
I read this article : https://learn.microsoft.com/en-us/sql/t-sql/xml/nodes-method-xml-data-type?view=sql-server-ver15 but since my target column is a VARCHAR and has to be cast, it actually didn't help me...
So first I cast the values to xml:
SELECT CAST(mytextField.TEXTVALUE AS XML) AS xmlData
FROM myDataTable
WHERE mytextFieldId = 12345
This returns the values as XML:
<content><value>12345</value></content>
<content><value>9874</value></content>
<content><value>Foo</value><value>Bar</value></content>
Now I'm stuck.
How can I select this result in a new query?
What would be the best approach to convert this to json format?
In the first snippet I added a prefix and a suffix to the value of the xmls value tag. How could I do this with the cast rows I selected?
I have a table in SQL that contains a list of settings for an application per serverID
I want write a SQL statement that produce a specific XML layout.
The basic SQL statement to retrieve this data:
SELECT SettingName, SettingValue
FROM dbo.ServerSettings
WHERE ServerID = #ServerID
I just need to know the correct FOR XML options.
I want to get back a result like the following. Basically using the value of the SettingName field to be the name of the node.
<ROOT>
<COSTRECOVERYSYSTEM_CONNECTION_STRING></COSTRECOVERYSYSTEM_CONNECTION_STRING>
<COSTRECOVERYSYSTEM_EXTRACT_INTERVAL>60</COSTRECOVERYSYSTEM_EXTRACT_INTERVAL>
<COSTRECOVERYSYSTEM_FILE_DATESTAMP>yyyyMMdd</COSTRECOVERYSYSTEM_FILE_DATESTAMP>
<COSTRECOVERYSYSTEM_FILE_EXTENSION>txt</COSTRECOVERYSYSTEM_FILE_EXTENSION>
<COSTRECOVERYSYSTEM_FILE_NAME>txt</COSTRECOVERYSYSTEM_FILE_NAME>
<COSTRECOVERYSYSTEM_FILE_PATH>txt</COSTRECOVERYSYSTEM_FILE_PATH>
</ROOT>
As in any other query you cannot use a column's value as the output column name. This would need some dynamically created statement and EXEC() for its execution.
But you might do something along this:
DECLARE #tbl TABLE(SettingName VARCHAR(100),SettingValue VARCHAR(100));
INSERT INTO #tbl VALUES ('Setting1','1'),('Setting2','2'),('ForbiddenValue','Huh! What about & and <?');
SELECT CAST((
SELECT '<' + UPPER(t.SettingName) + '>' +
--this embedded FOR XML will implicitly do the escaping for you
(SELECT t.SettingValue AS [*] FOR XML PATH('')) +
'</' + UPPER(t.SettingName) + '>'
FROM #tbl t
FOR XML PATH(''),TYPE
).value('.','nvarchar(max)') AS XML)
FOR XML PATH('ROOT');
In general I would never create XML with string methods. There are so many possible draw backs and traps. But in this case it might be the best choice.
Hint: Be sure, that the setting names are valid XML element names. There are some XML element naming rules
You can see these links for more information:
1. https://learn.microsoft.com/en-us/sql/relational-databases/xml/for-xml-sql-server?view=sql-server-2017
2. https://www.red-gate.com/simple-talk/sql/learn-sql-server/using-the-for-xml-clause-to-return-query-results-as-xml/
I have xml like this:
<root>
<name></name>
</root>
When I save it do database to XML column and query e.g in SSMS, it is formatted with self closing tags:
<root>
<name />
</root>
Is it possible to keep original formatting, or determine the formatting in SELECT statement?
If you are getting the data in a non-xml format, and don't want a Self closing tag, you'll need to replace the NULL with an empty string: ISNULL([YourColumn],'').
For example:
CREATE TABLE #Sample ([name] char(1));
INSERT INTO #Sample
VALUES(NULL);
SELECT ISNULL([Name],'') AS [name]
FROM #Sample
FOR XML PATH('root');
DROP TABLE #Sample;
If, however, you're inserting that xml into SQL Server, as an xml type, and then returning it, then SQL Server will use self-closing tags (as per my comment on the question).
As #DavidG said, any good xml parser will be able to read both self closing and non-self closing tags. If your parser can't read self closing tags, you need to consider updating your parser. If it's purely for display purposes... Well why are you using the "old" way of doing it for display?
The self-closing element <SomeElement/> is - semantically - the same as <SomeElement></SomeElement>. You should not bother about this... If your reading tool (or a third party requirement) needs this, you should rather replace this tool or discuss this with your partner.
The problem is: You have no control, that things stay as they are. Even in cases, where you are able to store the empty value with an opening and a closing tag this might be changed implicitly with a later call.
Try this:
DECLARE #tbl TABLE(ID INT IDENTITY,YourXML XML);
INSERT INTO #tbl VALUES
(
(SELECT '' AS [SomeTag] FOR XML PATH('RowNode'),ROOT('RootNode'),TYPE)
)
,(
(SELECT '' AS [SomeTag] FOR XML RAW('RowNode'),ROOT('RootNode'), ELEMENTS)
)
,(
N'<RootNode>
<RowNode>
<SomeTag></SomeTag>
</RowNode>
</RootNode>'
)
,
(CAST(
N'<RootNode>
<RowNode>
<SomeTag></SomeTag>
</RowNode>
</RootNode>'AS XML)
);
DECLARE #FirstXml XML=(SELECT YourXml FROM #tbl WHERE ID=1);
INSERT INTO #tbl VALUES(#FirstXml);
INSERT INTO #tbl SELECT #FirstXml;
INSERT INTO #tbl SELECT #FirstXml.query(N'.');
SELECT * FROM #tbl
The result
ID YourXML
--FOR XML PATH created - why ever - both tags
1 <RootNode><RowNode><SomeTag></SomeTag></RowNode></RootNode>
--FOR XML AUTO and all implicit casts use self-closing tags
2 <RootNode><RowNode><SomeTag /></RowNode></RootNode>
3 <RootNode><RowNode><SomeTag /></RowNode></RootNode>
4 <RootNode><RowNode><SomeTag /></RowNode></RootNode>
--Here we insert the first node *as is*
5 <RootNode><RowNode><SomeTag></SomeTag></RowNode></RootNode>
6 <RootNode><RowNode><SomeTag></SomeTag></RowNode></RootNode>
--But `.query()` will reformat this
7 <RootNode><RowNode><SomeTag /></RowNode></RootNode>
Some background:
XML is not stored as string string representation you see but as a hierarchy-table. Whenever you get the XML displayed on screen, its string representation is rebuilt from scratch. This can be slightly different each time you call it (e.g. attribute's order, CDATA sections, empty elements).
If you need this you can only enforce this format on string level. You might use some kind of RegEx approach, to replace any <abc/> with <abc></abc>.
But again: You should not have to think about that...
UPDATE
Try the code above with this SELECT:
SELECT *
,CAST(YourXML AS NVARCHAR(MAX)) AS CastedToString
FROM #tbl
The result
ID YourXML
CastedToString
--implicitly changed to self-closing tags
1 <RootNode><RowNode><SomeTag></SomeTag></RowNode></RootNode>
<RootNode><RowNode><SomeTag/></RowNode></RootNode>
--Self-closing **without a blank!!!**
2 <RootNode><RowNode><SomeTag /></RowNode></RootNode>
<RootNode><RowNode><SomeTag/></RowNode></RootNode>
You can see, that a comparisson on string level is not that easy... and rather hard to predict... You might use CAST(YourXml.query(N'.') AS NVARCHAR(MAX)) to get the same action done on each of your XMLs...
I want to generate an XML in UTF-8 encoding. But by default it is generating in UCS-2.
Please help me to generate XML in UTF-8 encoding. Below is my query:
select
isnull(cast('5678' as nvarchar(50)),'') [Vlootnummer],
isnull((select top 1 cast(EngineNo as nvarchar(50))
from VTS_DEMO.dbo.VehicleDetails v
join VTS_DEMO.dbo.VehicleDevice vd on v.VehicleId = vd.VehicleId
and ObuID = '353234023894171'), '') as Kenteken,
isnull((select top 1 cast(FillingStationName as nvarchar(50)) Units
from VTS_DEMO.dbo.FillingStation
where GeoFenceId = 3655),'') Locatie,
isnull((select top 1 GeofenceCode
from VTS_DEMO.dbo.GeoFence
where GeoFenceId = 3655), '') GeoFencingID,
isnull(cast(case when 1 = 0 then '' else '2017-02-07T23:15:25Z' end as nvarchar(50)),'') Aankomsttijd,
isnull(cast(case when 1 = 0 then '2017-02-07T23:15:25Z' else NULL end as nvarchar(50)),'') Vertrektijd
FOR XML PATH('Notificatie')
When I send this XML as attachment in mail using stored procedure msdb.dbo.sp_send_dbmail, when its opened in notepad++ then it shows UCS-2.
As per the MSDN documentation, It is a limitation on the XML datatype on MS SQL Server. SQL Server always saves an XML datatype with the character encoding of UCS-2.
The XML declaration PI, for example, , is not preserved when storing XML data in an xml data type instance. This is by design. The XML declaration () and its attributes (version/encoding/stand-alone) are lost after data is converted to type xml. The XML declaration is treated as a directive to the XML parser. The XML data is stored internally as ucs-2. All other PIs in the XML instance are preserved.
Considering the above, you can add this manually to take affect.
In an attempt to do this, you can use the following to gain advantage of FOR XML
SELECT CAST((SELECT [Columns] FROM [Tables] FOR XML AUTO) AS VARCHAR(MAX)) AS xmldata
You can check this answer for a discussion similar to this.
Hope this helps!
Is there a way to get the xml-safe version of an xml column in sql server ?
By xml-Safe i mean escaping special characters like <,>,', &, etc.
I'd like to avoid doing the replacements myself. Is there a build in function in sql server.
What I want to achieve is to store the xml content into another xml attribute.
It is not a direct answer to this question but to anyone who tries to xml-escape strings in TSQL, here is a little function I wrote :
CREATE FUNCTION escapeXml
(#xml nvarchar(4000))
RETURNS nvarchar(4000)
AS
BEGIN
declare #return nvarchar(4000)
select #return =
REPLACE(
REPLACE(
REPLACE(
REPLACE(
REPLACE(#xml,'&', '&')
,'<', '<')
,'>', '>')
,'"', '"')
,'''', ''')
return #return
end
GO
I assume that by xml-safe you mean escaping of XML special tags. If you have an XML column you wish to include in another XML document then you have two options:
project the column as [*]: select ..., xmlcolumn as [*], ... from ... for xml path... this will embed the XML content of the column in the result XMl. Eg. if the column has the value <element>value</element> then the result will be like <root><row><element>value</element></row></root>.
project the column as the column name: select ..., xmlcolumn, ... from ... for xml path... this will insert the content of the column as a value (ie. it will escape it). Eg. the same value as above will produce <root><row><xmlcolumn><element><value</element>.
If your question is about something else, then you're going to have to rephrase it in a proper manner and use terms correctly. Don't invent new terms no one understands but you.
Update:
If you are inserting XML values into the column, then you don't have to do anything at all. The client libraries know how to handle the proper escaping. As long as you write your code correctly. Remeber, XML is NOT a string and should never, ever be treated as one. If you write XML in your client, use an appropriate XML library (XmlWriter, XML DOM, Linq to XML etc). when passing in the XML into SQL Server, use the appropiate type: SqlXml. Stored procedures should use the appropiate parameter type: XML. When you read it, use the appropriate method to read XML: GetSqlXml(). Same goes for declaring the type in one of the miriad designers (LINQ to SQL , EF etc). Ultimately, there is never any need to escape XML characters manually. If you find yourself doing that, you're using the wrong API and you have to go back to the drawing board.
A good start reading is XML Support in Microsoft SQL Server 2005.
And finally, to manipulate XML as you describe (update XML column of table A with XML column of table B), you use XML methods, specifically modify (... insert...), and you bind the table B column inside the XQuery using sql:column:
update A
set somecolumn.modify('insert {sql:column("B.othercolumn")} before somenode')
from A join B on ...;
In you comment you threat XML as a string and, as I already said, you should never ever do that: strings and XML are as water and oil.
Another simpler way to xml escape a string is to use the following:
SELECT #String FOR XML PATH('')
e.g.
DECLARE #Input NVARCHAR(4000) = 'bacon & eggs'
DECLARE #String = (SELECT #Input FOR XML PATH(''))
then use #string from there
The contents of an XML column are XML. By definition, that is "XML-safe".
Do you need to include XML from a column in an XML element or attribute of another XML document? Then just save the outer XML as a string in the new document.