In a project using a MSSQL 2005 Database we are required to log all data manipulating actions in a logging table. One field in that table is supposed to contain the row before it was changed. We have a lot of tables so I was trying to write a stored procedure that would gather up all the fields in one row of a table that was given to it, concatenate them somehow and then write a new log entry with that information.
I already tried using FOR XML PATH and it worked, but the client doesn't like the XML notation, they want a csv field.
Here's what I had with FOR XML PATH:
DECLARE #foo varchar(max);
SET #foo = (SELECT * FROM table WHERE id = 5775 FOR XML PATH(''));
The values for "table", "id" and the actual id (here: 5775) would later be passed in via the call to the stored procedure.
Is there any way to do this without getting XML notation and without knowing in advance which fields are going to be returned by the SELECT statement?
We used XML path and as you've discovered, it works very well. Since one of SQL's features is to store XML properly, the CSV makes no sense.
What you could try is a stored proc that reads out the XML in CSV format (fake it). I would. Since you won't likely be reading the data that much compared to saving it, the overhead is negligible.
How about:
Set #Foo = Stuff(
( Select ',' + MyCol1 + ',' + MyCol2 ...
From Table
Where Id = 5775
For Xml Path('')
), 1, 1, '')
This will produce a CSV line (presuming the inner SQL returns a single row). Now, this solves the second part of your question. As to the first part of "without knowing in advance which fields", there is no means to do this without using dynamic SQL. I.e., you have to build the SQL statement as a string on the fly. If you are going to do that you might as well build the entire CSV result on the fly.
Related
I have a text file that was created by dumping 8 SQL tables into it. Now I need to import this data back into SQL Server.
Using BULK insert I was able to load data into one table with single column 'FileData'.
DECLARE #FileTable TABLE (FileData NVARCHAR(MAX))
INSERT INTO #FileTable
SELECT BulkColumn
FROM OPENROWSET( BULK N'C:\My\Path\Name\FileName.txt', SINGLE_CLOB) AS Contents
SELECT * FROM #FileTable
So now I have this huge string that I need to organize into different tables.
For example this part of string corresponds to the below table :
FileData
00001 00000009716496000000000331001700000115200000000000
Table:
It also seems like all fields have a set length and I can get that length.
I can see doing something like this:
select SUBSTRING('00001 00000009716496000000000331001700000115200000000000 ', 1,5) as RecordKey
select SUBSTRING('00001 00000009716496000000000331001700000115200000000000 ', 6,17) as Filler
select SUBSTRING('00001 00000009716496000000000331001700000115200000000000 ', 23,16) as BundleAnnualPremium
But is any faster and better way to load this data into different tables?
You could just bulk insert with a format file right from the start. But since the data is already loaded into a big table, if you'd rather use pure TSQL, you can pull elements out of a string using left(), right(), and substring().
Running SQL Server 2014. I have a stored procedure that returns a quite large XML. It goes something like this:
SELECT(
...
FOR XML PATH (N''), ROOT, TYPE
Now, that query runs in 1 second. If I remove TYPE it runs in around half the time:
SELECT(
...
FOR XML PATH (N''), ROOT
Obviously, the latter returns an nvarchar(max) instead of an xml. I want xml data, but if I ask for xml it gets slower! If I want to fetch xml data on the client, is it really necessary to convert it to xml using the TYPE directive above?
Q: Anyway, why is FOR XML ... TYPE significantly slower than FOR XML ...? Is there any way to improve the conversion?
Did you try to set variables with the results as XML and as VARCHAR(MAX) without displaying them? Maybe the time difference you measure is bound to preparing the viewer? Pasting the first letters into a grid column is faster than creating a well formed, indented, displayable XML...
Sepcifying "TYPE" is not needed in most cases. You really need this with nested XML only. Just play around with aliases, PATH- and ROOT-literals and - of course - with or without TYPE:
And - very important! - try to call this with the surrounding SELECT and without:
SELECT
(
SELECT tbls.TABLE_NAME AS [#TableName]
,(
SELECT COLUMN_NAME AS [#ColumName]
FROM INFORMATION_SCHEMA.COLUMNS AS cols
WHERE cols.TABLE_NAME=Tbls.TABLE_NAME
FOR XML PATH('COLUMN') /*,TYPE*/
) /*AS alias*/
FROM INFORMATION_SCHEMA.TABLES AS Tbls
FOR XML PATH('TABLE'),ROOT('ALL_TABLES') /*,TYPE*/
) /*AS alias*/
I don't know, how you continue with your generated XML. If you transfer it to your application it will be a plain string anyway.
Conclusio: Take the faster approach :-)
By the way...
I do not know your Stored Procedure and what else is done there besides the SELECT...
In most cases it is a bad habit to use SPs just to read data.
If your SP is not more than a wrapper around your SELECT you should think about a (single-statement!) table valued function to retrieve your data.
This function is easily queried and transformed to XML with
SELECT *
FROM dbo.MyFunction(/*Parameters*/)
FOR XML PATH('TheRowsName'),ROOT('TheRootName') [,TYPE]
Or - if you need this as XML everytime, you might define a scalar function delivering XML or VARCHAR(MAX). The re-usability of functions is way better than with SPs...
I've currently got a C# application that responds to HTTP requests. The body of the HTTP request (XML) is passed to SQL Server, at which time the database engine performs the correct instruction. One of the instructions is used to load information about Invoices using the id of the customer(InvoiceLoad):
<InvoiceLoad ControlNumber="12345678901">
<Invoice>
<CustomerID>johndoe#gmail.com</CustomerID>
</Invoice>
</InvoiceLoad>
I need to perform a SELECT operation against the invoice table (which contains the associated email address).
I've tried using:
SELECT 'Date', 'Status', 'Location'
FROM Invoices
WHERE Email_Address = Invoice.A.value(.)
using an xml.nodes('InvoiceLoad/Invoice/CustomerId') Invoice(A)
command.
However, as this query may run THOUSANDS of times per minute, I want to make it as fast as possible. I'm hearing that one way to do this may be to use CROSS APPLY (which I have never used). Is that the solution? If not, how exactly would I go about making this query as fast as possible? Any and all suggestions are greatly appreciated!
I don't see why you would need a call to .nodes() at all - from what I understand, each XML fragment has just a single entry - right?
So given this XML:
<InvoiceLoad ControlNumber="12345678901">
<Invoice>
<CustomerID>johndoe#gmail.com</CustomerID>
</Invoice>
</InvoiceLoad>
you can use this SQL to get the value of the <CustomerID> node:
DECLARE #xmlvar XML
SET #xmlvar = '<InvoiceLoad ControlNumber="12345678901">
<Invoice>
<CustomerID>johndoe#gmail.com</CustomerID>
</Invoice>
</InvoiceLoad>'
SELECT
#xmlvar.value('(/InvoiceLoad/Invoice/CustomerID)[1]', 'varchar(100)')
and you can join this against your customer table or whatever you need to do.
If you have the XML stored in a table, and you always need to extract that value from <CustomerID>, you could also think about creating a computed, persisted column on that table that would extract that e-mail address into a separate column, which you could then use for easy joining. This requires a little bit of work - a stored function taking the XML as input - but it's really quite a nice way to "surface" certain important snippets of data from your XML.
Step 1: create your function
CREATE FUNCTION dbo.ExtractCustomer (#input XML)
RETURNS VARCHAR(255)
WITH SCHEMABINDING
AS BEGIN
DECLARE #Result VARCHAR(255)
SELECT
#Result = #Input.value('(/InvoiceLoad/Invoice/CustomerID)[1]', 'varchar(255)')
RETURN #result
END
So given your XML, you get the one <CustomerID> node and extract its "inner text" and return it as a VARCHAR(255).
Step 2: add a computed, persisted column to your table
ALTER TABLE dbo.YourTableWithTheXML
ADD CustomerID AS dbo.ExtractCustomer(YourXmlColumnHere) PERSISTED
Now, your table that has the XML column has a new column - CustomerID - which will automagically contain the contents of the <CustomerID> as a VARCHAR(255). The value is persisted, i.e. as long as the XML doesn't change, it doesn't have to be re-computed. You can use that column like any other on your table, and you can even index it to speed up any joins on it!
Note: I'm running under SQL Server 2008 R2...
I've taken the time to read dozens of posts on this site and other sites on how to execute dynamic SQL where the query is more than 4000 characters. I've tried more than a dozen solutions proposed. The consensus seems to be to split the query into 4000-character variables and then do:
EXEC (#SQLQuery1 + #SQLQuery2)
This doesn't work for me - the query is truncated at the end of #SQLQuery1.
Now, I've seen samples how people "force" a long query by using REPLICATE a bunch of spaces, etc., but this is a real query - but it gets a little more sophisticated than that.
I have SQL View with a name of "Company_A_ItemView".
I have 10 companies that I want to create the same exact view, with different names, e.g.
"Company_B_ItemView"
"Company_C_ItemView"
..etc.
If you offer help, please don't ask why there are multiple views - just accept that I need to do it this way, OK?
Each company has its own set of tables, and the CREATE VIEW statement references several tables by name. Here's BRIEF sample, but remember, the total length of the query is around 6000 characters:
CREATE view [dbo].[Company_A_ItemView] as
select
WE.[Item No_],
WE.[Location Code],
LOC.[Bin Number],
[..more fields, etc.]
from
[Company_A_Warehouse_Entry] WE
left join
[Company_A_Location] LOC
...you get the idea
So, what I am currently doing is:
a. Pulling the contents of the CREATE VIEW statement into 2 Declared Variables, e.g.
Set #SQLQuery1 = (select text
from syscomments
where ID = 1382894081 and colid = 1)
Set #SQLQuery2 = (select
from syscomments
where ID = 1382894081 and colid = 2)
Note that this is how SQL stores long definitions - when you create the view, it stores the text into multiple syscomments records. In my case, the view is split into a text chunk of 3591 characters into the first syscomment record and the rest of the text is in the second record. I have no idea why SQL doesn't use all 4000 characters in the syscomment field. And the statement is broken in the middle of a word.
Please note in all my examples, all #SQLQueryxxx variables are declared as varchar(max). I've also tried declaring them as nvarchar(max) and varchar(8000) and nvarchar(8000) with the same results.
b. I then do a "Search and Replace" for "Company_A" and replace it with "Company_B". In the code below, the variable "#CompanyID" is first set to "Company_B":
SET #SQLQueryNew1 = #SQLQuery1
SET #SQLQueryNew1 = REPLACE(#SQLQueryNew1, 'Company_A', #CompanyID)
SET #SQLQueryNew2 = #SQLQuery2
SET #SQLQueryNew2 = REPLACE(#SQLQueryNew2, 'Company_A',#CompanyID)
c. I then try:
EXEC (#SQLQueryNew1 + #SQLQueryNew2)
The message returned indicates that it's trying to execute the statement truncated at the end of #SQLQueryNew1, e.g. 80% (approx) of the query's text.
I've tried CAST'ing the final result into a new varchar(max) and nvarchar(max) - no luck
I've tried CAST'ing the original query a new varchar(max) and nvarchar(max)- no luck
I've looked at the result of retrieving the original CREATE VIEW statement, and it's fine.
I've tried various other ways of retrieving the original CREATE VIEW statement, such as:
Set #SQLQuery1 = (select VIEW_DEFINITION)
FROM [MY_DATABASE].[INFORMATION_SCHEMA].[VIEWS]
where TABLE_NAME = 'Company_A_ItemView')`
This one returns only the first 4000 characters of the CREATE VIEW
Set #SQLQuery1 = (SELECT (OBJECT_DEFINITION(#ObjectID))
If I do a
SELECT LEN(OBJECT_DEFINITION(#ObjectID))
it returns the correct length of the query (e.g. 5191), but if I look at #SQLQuery1, or try to
EXEC(#SQLQuery1), the statement is still truncated.
c. There are some references that state that since I'm manipulating the text of the query after retrieving it, the resulting variables are then truncated to 4000 characters. I've tried CAST'ing the result as I do the REPLACE, e.g.
SET #SQLQueryNew1 = SELECT (CAST(REPLACE(#SQLQueryNew1,
'Company_A',
#CompanyID) AS varchar(max))
Same result.
I know there are other methods, such as creating stored procedures for creating the views. But the views are being developed and are somewhat "in flux", so placing the text of the CREATE VIEW inside a stored proc is cumbersome. My goal is to be able to take Company_A's view and replicate it exactly - multiple times, except reference Company_B's view name and table names, Company_C's view name and table names, etc.
I'm wondering if there is anyone out there who has done this type of manipulation of a long SQL "CREATE VIEW" statement and try to execute it.
Just use VARCHAR(MAX) or NVARCHAR(MAX). They work fine for EXEC(string).
FYI,
Note that this is how SQL stores long definitions - when you create
the view, it stores the text into multiple syscomments records.
This is not correct. This is how it used to be done on SQL Server 2000. Since SQL Server 2005 and higher they are saved as NVARCHAR(MAX) in a single entry in sys.sql_modules.
syscomments is still around, but it is retained read-only solely for compatibility.
So all you should need to do is to change your #SQLQuery1,2,etc. variables to a single NVARCHAR(MAX) variable, and pull your View code from the [definition] column of the sys.sql_modules table instead.
Note that you should be careful with your string manipulations as there are certain functions that will revert to (N)VARCHAR(4000) output if all of their input arguments are not (N)VARCHAR(MAX). (Sorry, I do not know which ones, but REPLACE() may be one). In fact, this may be what has been causing so much confusion in your tests.
declare your sql variables (#SQLQuery1...) as nvarchar(4000)
be sure each sql part did't exceed 4000 byte (copy each part to a text file and test the file size in bytes)
Is there a way to get the xml-safe version of an xml column in sql server ?
By xml-Safe i mean escaping special characters like <,>,', &, etc.
I'd like to avoid doing the replacements myself. Is there a build in function in sql server.
What I want to achieve is to store the xml content into another xml attribute.
It is not a direct answer to this question but to anyone who tries to xml-escape strings in TSQL, here is a little function I wrote :
CREATE FUNCTION escapeXml
(#xml nvarchar(4000))
RETURNS nvarchar(4000)
AS
BEGIN
declare #return nvarchar(4000)
select #return =
REPLACE(
REPLACE(
REPLACE(
REPLACE(
REPLACE(#xml,'&', '&')
,'<', '<')
,'>', '>')
,'"', '"')
,'''', ''')
return #return
end
GO
I assume that by xml-safe you mean escaping of XML special tags. If you have an XML column you wish to include in another XML document then you have two options:
project the column as [*]: select ..., xmlcolumn as [*], ... from ... for xml path... this will embed the XML content of the column in the result XMl. Eg. if the column has the value <element>value</element> then the result will be like <root><row><element>value</element></row></root>.
project the column as the column name: select ..., xmlcolumn, ... from ... for xml path... this will insert the content of the column as a value (ie. it will escape it). Eg. the same value as above will produce <root><row><xmlcolumn><element><value</element>.
If your question is about something else, then you're going to have to rephrase it in a proper manner and use terms correctly. Don't invent new terms no one understands but you.
Update:
If you are inserting XML values into the column, then you don't have to do anything at all. The client libraries know how to handle the proper escaping. As long as you write your code correctly. Remeber, XML is NOT a string and should never, ever be treated as one. If you write XML in your client, use an appropriate XML library (XmlWriter, XML DOM, Linq to XML etc). when passing in the XML into SQL Server, use the appropiate type: SqlXml. Stored procedures should use the appropiate parameter type: XML. When you read it, use the appropriate method to read XML: GetSqlXml(). Same goes for declaring the type in one of the miriad designers (LINQ to SQL , EF etc). Ultimately, there is never any need to escape XML characters manually. If you find yourself doing that, you're using the wrong API and you have to go back to the drawing board.
A good start reading is XML Support in Microsoft SQL Server 2005.
And finally, to manipulate XML as you describe (update XML column of table A with XML column of table B), you use XML methods, specifically modify (... insert...), and you bind the table B column inside the XQuery using sql:column:
update A
set somecolumn.modify('insert {sql:column("B.othercolumn")} before somenode')
from A join B on ...;
In you comment you threat XML as a string and, as I already said, you should never ever do that: strings and XML are as water and oil.
Another simpler way to xml escape a string is to use the following:
SELECT #String FOR XML PATH('')
e.g.
DECLARE #Input NVARCHAR(4000) = 'bacon & eggs'
DECLARE #String = (SELECT #Input FOR XML PATH(''))
then use #string from there
The contents of an XML column are XML. By definition, that is "XML-safe".
Do you need to include XML from a column in an XML element or attribute of another XML document? Then just save the outer XML as a string in the new document.