Exporting XML Data to mutiple files but retain structure - sql-server

I have been searching on the net for the last couple of days and unfortunately cannot find quite what I am looking for. I have a SQL Server 2012 table that contains 2 columns, one is PurchaseOrder DataType varchar(10) the other is Data with a DataType of XML. Currently what is happening ever few hours a stored procedure is run to check a number of table for a value and then create a PurchaseOrder in the desired XML format. This has been achieved and validated by the company that I am going to be sending the data to. The issue I now have is I want to create a SSIS package that looks at this table and creates multiple xml files with the contents of the Data field "not putting the contents in to one long line" and then using the PurchaseOrder field for the file name. I have found various part solutions e.g. using a Script task as there is no native XML Destination in SSIS but struggling and being quite new to SSIS not getting anywhere fast. Can anyone offer any help, guidance, youtube links or even a part way there package.
Thanks P
Further to the above question I have tried to go about it using the bcp command, below is the following code that works great just looking at the top 1 record
DECLARE #fileName VARCHAR(50)
DECLARE #sqlStr VARCHAR(1000)
DECLARE #sqlCmd VARCHAR(1000)
SET #fileName = 'C:\Temp\test.xml'
SET #sqlStr = 'select TOP 1 Data from DatabaseName.dbo.OutputXML'
SET #sqlCmd = 'bcp "' + #sqlStr + '" queryout ' + #fileName + '
-w -T -S Server\Instance'
EXEC xp_cmdshell #sqlCmd
What I have then done is add this in to a cursor, which again works to a point, it creates my files, using the purchaseorder as the file name and outputs the relevant information but it no longer holds the proper xml format, it goes back to looking like a string with none of the original formatting, is this because I am imbedding it in the cursor?? Below is the code that I am using
DECLARE #fileName VARCHAR(50)
DECLARE #sqlStr VARCHAR(1000)
DECLARE #sqlCmd VARCHAR(1000)
DECLARE xml_cursor CURSOR FOR
SELECT PurchaseOrder from DatabaseName.dbo.OutputXML
OPEN xml_cursor
FETCH NEXT FROM xml_cursor INTO #fileName
WHILE ##FETCH_STATUS = 0
BEGIN
SET #fileName = 'C:\Temp\'+#fileName+'.xml'
SET #sqlStr = 'select Data from DatabaseName.dbo.OutputXML'
SET #sqlCmd = 'bcp "' + #sqlStr + '" queryout ' + #fileName + '
-w -T -S Server\Instance'
EXEC xp_cmdshell #sqlCmd
FETCH NEXT FROM xml_cursor into #fileName
END
CLOSE xml_cursor
DEALLOCATE xml_cursor

Since bcp is a command line utility it does not know about your cursor. You can only change the parameters passed. You already pass the #filename from the cursor, you will also need to filter your query. Below is how to fix your problem but it is far from a great solution, it will be quite slow. I don't know what your key is called, so I called it SomeKey
DECLARE #fileName VARCHAR(50)
DECLARE #sqlStr VARCHAR(1000)
DECLARE #sqlCmd VARCHAR(1000)
DECLARE #SomeKey VARCHAR(100) --Put your key datatype here
DECLARE xml_cursor CURSOR FOR
-- replace SomeKey with your key
SELECT SomeKey,PurchaseOrder FROM DatabaseName.dbo.OutputXML
OPEN xml_cursor
FETCH NEXT FROM xml_cursor INTO #SomeKey,#fileName
WHILE ##FETCH_STATUS = 0
BEGIN
SET #fileName = 'C:\Temp\' + #filename + '.xml'
SET #sqlStr = 'SELECT Data FROM DatabaseName.dbo.OutputXML WHERE SomeKey = ''' + #SomeKey +''''
SET #sqlCmd = 'bcp "' + #sqlStr + '" queryout ' + #fileName + '-w -T -S Server\Instance'
EXEC xp_cmdshell #sqlCmd
FETCH NEXT FROM xml_cursor into #SomeKey,#fileName
END
CLOSE xml_cursor
DEALLOCATE xml_cursor

Related

Not displaying select results from dynamic SQL within Cursor

I'm currently learning SQL and trying to think of exercises for myself and I can't seem to make this one work even though it seems simple:
I'm trying to run a cursor through all the filtered tables within my db so that then I could pass that table name to a variable which will be used within a DynamicSQL inside the cursor. The end result should be all values from every column that has the column 'empid' in them.
However, the message returns as "Commands completed successfully" but I get to see no results despite my select statement.
I'm trying to run something like this:
declare #tablename nvarchar(200);
declare #empid int;
declare #sql nvarchar(200) = N'select * from ' + #tablename + N' where empid = ' +#empid;
declare tablecursor cursor for select table_name from information_schema.tables where col_length(table_name, 'empid') is not null;
open tablecursor;
fetch next from tablecursor into #tablename;
while ##fetch_status = 0
begin
execute sp_executesql #sql, 825
fetch next from tablecursor into #tablename;
end
close tablecursor;
deallocate tablecursor;
I've been searching everywhere for answers to make this work but can't find anything. I've tried putting into a stored procedure and then executing it from there but that didn't work either.
Help would be highly appreciated.
DECLARE #SQL Should be outside but assigning the Variable inside the while loop
SET #SQL = N'SELECT * FROM ' + #tableName
Should be in while loop.
The other thing is to increase the length of #SQL Variable.
Thank you kindly for the help. After I listened to your advice I've encountered more errors but at least for these I was able to find answers online. What I also learnt is that you can't have your sql string in quotes when you execute it as that will make SSMS treat #SQL as an actual string and not a variable. I've managed to get it working and my code now looks something like this:
create proc cdr #empid nvarchar(5) as
declare #tablename nvarchar(200);
declare tablecursor cursor for select table_name from information_schema.tables where col_length(table_name, 'empid') is not null;
open tablecursor;
fetch next from tablecursor into #tablename;
declare #sql nvarchar(max)
while ##fetch_status = 0
begin
set #sql = N'select * from ' + #tablename + N' where empid = ' + #empid;
execute sp_executesql #sql
fetch next from tablecursor into #tablename;
end
close tablecursor;
deallocate tablecursor;

Why xp_cmdshell exports empty PDF files from database that stored in image datatype

I have query that extracts PDF files into directory. But for some reason all files are empty (0 KB).
Column PDF stored as image datatype. I know its obsolete but I am not the owner of the database.
Would you suggest any workaround?
CREATE TABLE #tblPolicyForms
(
FormID INT NOT NULL
,PDF varbinary(max) NULL
,PDF_FIlename VARCHAR(max)
)
INSERT INTO #tblPolicyForms
SELECT FormID,
PDF,
PDF_FileName
FROM [dbo].[tblPolicyForms]
WHERE FormID IN (19,20,21,22)
--select * from #tblPolicyForms
DECLARE #FormID varchar(4);
DECLARE #FileName VARCHAR(200);
DECLARE FormID_cursor CURSOR FOR
SELECT
FormID
,PDF_FIlename
FROM
#tblPolicyForms
WHERE FormID IN (19,20,21,22)
OPEN FormID_cursor
FETCH NEXT FROM FormID_cursor
INTO #FormID, #FileName
WHILE ##FETCH_STATUS = 0
BEGIN
DECLARE #cmd VARCHAR(8000) = 'BCP "SELECT PDF FROM #tblPolicyForms " queryout "C:\Users\oserdyuk\Documents\ExportDir\'+#FileName+'" -T -N'
--print #cmd
EXEC master.dbo.xp_cmdshell #cmd;
FETCH NEXT FROM FormID_cursor
INTO #FormID, #FileName
END
CLOSE FormID_cursor
DEALLOCATE FormID_cursor
The thing is BCP is opening another connection to SQL Server, and regular temp tables like your #tblPolicyForms are private to the connection. The BCP is probably generating an error which you don't get to see.
Can you change your temp table to a global one? This way the other connection (BCP) will be able to access it. You need to use two pound signs like this:
CREATE TABLE ##tblPolicyForms
(
FormID INT NOT NULL
,PDF varbinary(max) NULL
,PDF_FIlename VARCHAR(max)
)
You also need to change every reference from #tblPolicyForms to ##tblPolicyForms (single to double pounds). Be aware that if the procedure is executed in parallel by different sessions you will receive an error indicating that ##tblPolicyForms already exists.
Another thing, you are exporting every row into every file, so every file ends up the same size as you have seen. You need a WHERE clause in your BCP like this:
DECLARE #cmd VARCHAR(8000) = 'BCP "SELECT PDF FROM ##tblPolicyForms WHERE FormID = ''' + #FormID + ''' " queryout "C:\Users\oserdyuk\Documents\ExportDir\'+#FileName+'" -T -N'
But even then things might not work properly because bcp will prepend a 8 byte header to your row. Since bcp is an export/import tool it will create a file it can later import back into your database.

Trigger MSSQL - Output to File (XML)

we have a very old ERP system which is badly supported.
Now the warehouse want´s to buy a new "store system" for our goods. It´s a fully automatic store system which need´s data from our ERP system. The support of your ERP system can´t help us, so we have to build a solution of our own.
The idea was to "move" the items for the new storage system to a special storage place called (SHUT1) and output the "part number" and "quantity" to a file (xml) which can be read by the new software.
We can´t change anything in the software of our ERP system, so we have to do it on the SQL Server itself.
(I know, a trigger is not the "best" thing to do, but I have have no other choice)
CREATE TRIGGER tr_LagerShut ON dbo.Lagerverwaltung
AFTER INSERT
AS
BEGIN
IF (SELECT [Lagerort] from Inserted) = 'SHUT1'
BEGIN
DECLARE #Cmd VARCHAR(2000) ;
DECLARE #FormatDate4File VARCHAR(200);
SET #FormatDate4File = (SELECT(SYSUTCDATETIME()));
SET #FormatDate4File = (SELECT REPLACE(#FormatDate4File,' ','-'));
SET #FormatDate4File = (SELECT REPLACE(#FormatDate4File,':','-'));
SET #FormatDate4File = (SELECT REPLACE(#FormatDate4File,'.','-'));
SET #Cmd = ( SELECT [Artikelnummer],[Menge] FROM inserted FOR XML PATH('')) ;
SET #Cmd = 'Echo "' + #Cmd + '" >>"C:\Temp\' + #FormatDate4File +'.xml"' ;
EXEC xp_cmdshell #Cmd ;
END;
END;
The trigger "installs" fine, but if I change a storage place to a new one, the ERP system stalls with "ERROR" (there is no error description :(
If I drop the trigger the system is just running fine again. So I think there is a error in the trigger, but I can´t find it.
Can anybody help please?
Aleks.
Don't know what ERP system stalls with "ERROR" looks like... Frozend GUI? Timeout? Just no file created?
My magic glass bulb tells me the following: You are inserting more than one row at once. If so, this statement will break, because a comparison like this is only valid against a scalar value. If there is more than one row in inserted, this will not work:
IF (SELECT [Lagerort] from Inserted) = 'SHUT1'.
Your trigger can be simplified, but I doubt, that you will like the result. Check this with special characters (like üöä) and check for enclosing "-characters...
CREATE TRIGGER tr_LagerShut ON dbo.Lagerverwaltung
AFTER INSERT
AS
BEGIN
IF EXISTS(SELECT 1 FROM inserted WHERE [Lagerort]='SHUT1')
BEGIN
DECLARE #FileName VARCHAR(255) =REPLACE(REPLACE(REPLACE(SYSUTCDATETIME(),' ','-'),':','-'),'.','-');
DECLARE #Content XML=
(
SELECT [Artikelnummer],[Menge] FROM inserted WHERE [Lagerort]='SHUT1' FOR XML AUTO,ELEMENTS
);
DECLARE #Cmd VARCHAR(4000) = 'Echo "' + CAST(#Content AS VARCHAR(MAX)) + '" >>"c:\temp\' + #FileName + '.xml"' ;
PRINT #cmd;
EXEC xp_cmdshell #Cmd ;
END
END;
This might better be done with BCP.
UPDATE Your comment...
First you should check, if this works at all:
DECLARE #FileName VARCHAR(255) =REPLACE(REPLACE(REPLACE(SYSUTCDATETIME(),' ','-'),':','-'),'.','-');
DECLARE #Content XML=
(
SELECT TOP 5 * FROM sys.objects FOR XML AUTO,ELEMENTS
);
DECLARE #Cmd VARCHAR(4000) = 'Echo "' + CAST(#Content AS VARCHAR(MAX)) + '" >>"c:\temp\' + #FileName + '.xml"' ;
PRINT #cmd;
EXEC xp_cmdshell #Cmd ;
If you find no file in c:\temp\: Are you aware, that SQL-Server will always write in its own context? Might be, that you are awaiting a file in your local c-drive, but the file is written to the Server's machine acutally.
If this works isolatedly, it should work within a trigger too. You might pack the call into BEGIN TRY ... END TRY and add an appropriate CATCH block.
So okay, the "simple" trigger could be really a problem. Now I have this idea:
(one more info: time stamp is not inserted into table "Lagerverwaltung" when a new row is inserted)
Pseudo Code on:
Trigger on Table "Lagerverwaltung"
Check if the storage place(s) is "SHUT1"
If "yes"
DECLARE #FileName VARCHAR(255) =REPLACE(REPLACE(REPLACE(SYSUTCDATETIME(),' ','-'),':','-'),'.','-');
create a new table with name dbo + '.' + #filename
Insert all the data (inserted) + row(SYSUTCDATETIME()) AS TimeStamp where 'Lagerort' = 'SHUT1' into new table named 'dbo.' + #filename
DECLARE #Cmd varchar(4000) = 'bcp "select [Artikelnummer],[Menge],[TimeStamp] FROM [wwsbautest].[dbo].#filename WHERE [Lagerort]=''SHUT1'' AND [Menge] > ''0'' FOR XML AUTO, ELEMENTS" queryout "C:\temp\' + #FileName + '.xml" -c -T';
EXEC xp_cmdshell #Cmd;
Drop table dbo.#filename;
Could somthing like that work?

Using BCP and xp_cmdshell with T-SQL inside SSMS. "The syntax of the command is incorrect"

So, I am playing around with BCP to get to know it better, and ran into an issue I am having trouble figuring out. I am getting an error, "The Syntax of the command is incorrect".
I have tested the complete file path which is building correctly, and the select query by itself, which is completing correctly. Any thoughts?
The code I am using is:
DECLARE #fileName varchar(128)
DECLARE #filePath varchar(128)
DECLARE #completeFilePath varchar(128)
DECLARE #sqlCmd varchar(4000)
DECLARE #BCPSwitches VARCHAR(64)
SET #filePath = 'xxxxxx'
SELECT TOP 5 [EMP]
,[SSNO]
,[NAME]
,[STREET]
INTO #ParticipantIdentifiers
FROM xxxxxxxxx
SET #BCPSwitches = '-w -T -t |'
SET #fileName = 'xxxxx.txt'
SET #completeFilePath = #filePath + #fileName
SET #sqlCmd = 'bcp "SELECT * FROM #ParticipantIdentifiers" queryout "'+ #completeFilePath + '" ' + #BCPSwitches
EXEC DemoDB..xp_cmdshell #sqlCmd
I think the terminator "|" after -t (in the #BCPSwitches) needs to be enclosed in quotes as well since the shell might be interpreting this as a output pipe.

Saving image in database field to file on server

I am trying to save data from a table directly to the servers file system using this t-sql:
DECLARE #ID int
DECLARE #command varchar(1000)
DECLARE #Name varchar(100)
DECLARE #FileName varchar(100)
DECLARE MyCursor CURSOR FOR
SELECT Id, [FileName] FROM FilesTable
OPEN MyCursor
FETCH NEXT FROM MyCursor
INTO #ID, #Name
WHILE ##FETCH_STATUS = 0
BEGIN
SELECT #FileName = 'd:\Temp\' + convert(varchar(200), #Name)
SET #command = 'bcp "SELECT FileBytes FROM FilesTable WHERE ID = ' + convert(varchar(18), #Id) + '" queryout "' + #FileName + '" -T -n'
EXEC master..xp_cmdshell #command, no_output
FETCH NEXT FROM MyCursor
INTO #Id, #Name
END
CLOSE MyCursor
DEALLOCATE MyCursor
The table is filled with solid data. accessing this data with a C# program works.
I did configure advanced options gave permission to use xp_cmdshell
When I run this query i don't get any error but no file was written.
I also don't see anything in the logs, (maybe I am looking in the wrong location?)
What am I doing wrong? Or what can I do to checdk 'under the hood'?
Based on the comment from Patrick and the answer in fastest way to export blobs from table into individual files I was able to store the images on my fileserver.
we ended up writing a CLR function something like the one given.

Resources