I need to create multiple xml files (per row) from single query which I did. My query is generating 10,000 xml files as you know SQL Server assign different name for each xml file now I need to save them in c:\ automatically. I can see all the files but I can’t save them any advise or help will highly be appreciated. One more hint: from SQL server we can save each xml one by one from SQL server File tab.
CREATE TABLE dbo.sample(
[BTno] [nvarchar](25) NULL,
[First_Name] [nvarchar](35) NULL,
[Last_Name] [nvarchar](35) NULL,
[BTid] [nvarchar](15) NULL,
) ON [PRIMARY];
insert into dbo.sample values('1B','Vartan','Sarkis','69876');
insert into dbo.sample values('2B','Anoush','Eric','87656');
insert into dbo.sample values('3B','Lucine','Arpiar','65467');
insert into dbo.sample values('4B','Anum','Noor','98076');
insert into dbo.sample values('5B','Abercio','Banninq','34897');
insert into dbo.sample values('1C','Gaea','Nishan','29841');
insert into dbo.sample values('7B','Marilyn','Vahe','78903');
insert into dbo.sample values('2Z','Bansi','Aakarshan','34905');
insert into dbo.sample values('9S','Eric','Abban','45892');
insert into dbo.sample values('12B','Dave','Tate','19994');
-- Here is the Query that generates multiple xml files
select ((select * from dbo.sample a where a.BTid = b.BTid
for xml path('Row'),TYPE, ROOT('BT')
)) from dbo.sample b
I cheated and used someone else's function for the actual write. Note that you'll need 'Ole Automation Procedures' enabled for the write procedure to work. Some company security policies get a little touchy about these things being enabled. Basically I assume if this isn't enabled, you have the ability to do so. The write function will also overwrite files with the same filename, though that may not be relevant for your need.
declare #thexml xml
declare #xmlstring varchar(max)
declare #path varchar(255)
declare #filename_start varchar(100)
declare #filename varchar(100)
declare #count int
set #path='C:\testdata\'
set #filename_start = 'xmlname'
set #count = 0
declare xcursor cursor for
select ((select * from dbo.sample a where a.BTid = b.BTid
for xml path('Row'),TYPE, ROOT('BT')
)) from dbo.sample b
open xcursor
fetch xcursor into #thexml
while ##fetch_status <> -1 begin
set #count = #count + 1
set #filename = #filename_start + cast(#count as varchar) + '.xml'
set #xmlstring = cast(#thexml as varchar(max))
exec master.dbo.spWriteStringToFile #xmlstring,#path,#filename
fetch xcursor into #thexml
end
deallocate xcursor
Enabling OLE - http://msdn.microsoft.com/en-us/library/ms191188.aspx
Credit for write procedure - https://www.simple-talk.com/sql/t-sql-programming/reading-and-writing-files-in-sql-server-using-t-sql/
Actual write procedure (must change from alter to create) - https://www.simple-talk.com/code/WorkingWithFiles/spWriteStringTofile.txt
Try with this script, maybe will be useful:
CREATE PROCEDURE saveFile (#file varchar(1000), #txt VARCHAR(8000)) AS
BEGIN
DECLARE #sys_obj INT
DECLARE #rst INT
DECLARE #fID INT
EXECUTE #rst = sp_OACreate 'Scripting.FileSystemObject', #sys_obj OUT
EXECUTE sp_OAMethod #sys_obj, 'FileExists', #rst OUT, #file
IF #rst=0
BEGIN
EXECUTE sp_OAMethod #sys_obj, 'OpenTextFile', #fID OUT, #file, 8, 1
EXECUTE sp_OAMethod #fID, 'WriteLine', Null, #txt
EXECUTE sp_OADestroy #fID
EXECUTE sp_OADestroy #sys_obj
END
END
And with this line, could be save to disk a String (for example a XML):
EXEC saveFile #file = 'C:\test.xml', #txt = '<xml><test>123</test></xml>'
Here other query for save ALL your fields that returns your query:
DECLARE cur CURSOR LOCAL FAST_FORWARD FOR
select CAST(((select * from dbo.sample a where a.BTid = b.BTid
for xml path('Row'),TYPE, ROOT('BT')
)) AS VARCHAR(MAX)) AS TXT from dbo.sample b
--HERE GO YOUR QUERY
/*Example
SELECT '<xml><test>111</test></xml>' AS TXT
UNION
SELECT '<xml><test>222</test></xml>' AS TXT
UNION
SELECT '<xml><test>333</test></xml>' AS TXT
*/
DECLARE #v_count AS INTEGER
DECLARE #v_txt AS VARCHAR(8000)
DECLARE #v_filename AS VARCHAR(100)
SET #v_count = 1
SET #v_txt = ''
OPEN cur FETCH NEXT FROM cur INTO #v_txt
WHILE ##FETCH_STATUS = 0
BEGIN
SET #v_filename = 'C:\test' + CAST(#v_count AS VARCHAR) + '.xml'
EXEC saveFile #file = #v_filename, #txt = #v_txt
SET #v_count = #v_count + 1
FETCH NEXT FROM cur INTO #v_txt
END
CLOSE cur
DEALLOCATE cur
Related
I'm working with remote Microsoft SQL Server.
After executing following code using DataGrip on Ubuntu I receive an error:
CREATE TABLE #TempNullClass (
id nvarchar(10),
classCode nvarchar(10)
);
BULK INSERT #TempNullClass
FROM '/home/user/Downloads/data1.csv'
WITH (
FIELDTERMINATOR = ',',
ROWTERMINATOR = '0x0a'
);
[S0001][4861] Cannot bulk load because the file "/home/user/Downloads/data1.csv" could not be opened. Operating system error code 3(The system cannot find the path specified.).
File has all of the permissions:
-rwxrwxrwx 1 user user 73399 may 15 15:11 data1.csv
I cannot put the file on remote server.
The file should be placed on the remote server or in a folder that shared to the remote server.
You can use a local file, the file path needs to be relative to the server, so a file on another computer needs to be a UNC path, a server local path does not have to be a share (security concern and one more thing you need to configure and monitor).
I have to loop over import files sometimes like all other DBA's do. I solve that like this.
You can dump it in a SQL Job and schedule it. Can run every second, or every day, just depends what your business requirements are. Here is a sample of one of my scripts:
SET NOCOUNT ON
DECLARE #BackupDirectory SYSNAME = 'G:\'
IF OBJECT_ID('tempdb..#DirTree') IS NOT NULL
DROP TABLE #DirTree
CREATE TABLE #DirTree (
Id int identity(1,1),
SubDirectory nvarchar(255),
Depth smallint,
FileFlag bit,
ParentDirectoryID int
)
INSERT INTO #DirTree (SubDirectory, Depth, FileFlag)
EXEC master..xp_dirtree #BackupDirectory, 10, 1
UPDATE #DirTree
SET ParentDirectoryID = (
SELECT MAX(Id) FROM #DirTree d2
WHERE Depth = d.Depth - 1 AND d2.Id < d.Id
)
FROM #DirTree d
DECLARE
#ID INT,
#BackupFile VARCHAR(MAX),
#Depth TINYINT,
#FileFlag BIT,
#ParentDirectoryID INT,
#wkSubParentDirectoryID INT,
#wkSubDirectory VARCHAR(MAX)
if OBJECT_ID('dbo.ImportFiles')=0
create table dbo.ImportFiles
(
FileNamePath VARCHAR(MAX)
)
else
truncate table dbo.ImportFiles
DECLARE FileCursor CURSOR LOCAL FORWARD_ONLY FOR
SELECT * FROM #DirTree WHERE FileFlag = 1
OPEN FileCursor
FETCH NEXT FROM FileCursor INTO
#ID,
#BackupFile,
#Depth,
#FileFlag,
#ParentDirectoryID
SET #wkSubParentDirectoryID = #ParentDirectoryID
WHILE ##FETCH_STATUS = 0
BEGIN
--loop to generate path in reverse, starting with backup file then prefixing subfolders in a loop
WHILE #wkSubParentDirectoryID IS NOT NULL
BEGIN
SELECT #wkSubDirectory = SubDirectory, #wkSubParentDirectoryID = ParentDirectoryID
FROM #DirTree
WHERE ID = #wkSubParentDirectoryID
SELECT #BackupFile = #wkSubDirectory + '\' + #BackupFile
END
--no more subfolders in loop so now prefix the root backup folder
SELECT #BackupFile = #BackupDirectory + #BackupFile
INSERT INTO ImportFiles (FileNamePath) VALUES(#BackupFile)
FETCH NEXT FROM FileCursor INTO
#ID,
#BackupFile,
#Depth,
#FileFlag,
#ParentDirectoryID
SET #wkSubParentDirectoryID = #ParentDirectoryID
END
CLOSE FileCursor
DEALLOCATE FileCursor
SET NOCOUNT ON
print 'loading files from file table'
-- =============================================
-- Declare and using a READ_ONLY cursor
-- =============================================
DECLARE eoddata_cursor CURSOR
READ_ONLY
FOR SELECT FileNamePath FROM ImportFiles
where FileNamePath like '%.csv'
DECLARE #name nvarchar(4000), #sql nvarchar(2000), #rows int
OPEN eoddata_cursor
FETCH NEXT FROM eoddata_cursor INTO #name
WHILE (##fetch_status <> -1)
BEGIN
IF (##fetch_status <> -2)
BEGIN
print 'file '+#name
begin try
truncate table [Tickers].[DataClient]
set #sql = '
bulk insert [Tickers].[DataClient]
from N'''+#name+'''
with (
FIELDTERMINATOR ='','',
ROWTERMINATOR =''\n'' ,
FIRSTROW = 2,
CODEPAGE = ''RAW'',
MAXERRORS = 2000000,
ERRORFILE = '''+#name+'.err''
)'
insert into [Tickers].[DataClient]
exec(#sql)
exec [Tickers].[MoveDataClientData] #filename = #name, #rows = #rows output
delete from ImportFiles where FileNamePath = #name
print 'loaded :'+str(#rows)
end try
begin catch
print 'error:'+ error_message()
end catch
END
FETCH NEXT FROM eoddata_cursor INTO #name
END
CLOSE eoddata_cursor
DEALLOCATE eoddata_cursor
GO
Subject:
I've got a report that I create every month. The creation of report consists of 2 steps:
Get an XML from our service and store it in DB;
Parse XML and create file.
For the last few month I've created report in manual mode. And now I want to automate this stuff.
But here comes a
Problem:
The second step (parsing XML and file creation) runs like a charm, but with first step I'm observing weird behaviour.
I got Stored Procedure which gets XML:
ALTER PROCEDURE [Structure].[GetXML]
#LastActDate date,
#CurActDate date
AS
BEGIN
SET NOCOUNT ON;
begining:
DECLARE #URI varchar(2000),
#methodName varchar(50),
#objectID int,
#hResult int,
#setTimeouts nvarchar(255),
#serv nvarchar(255) = 'http://example.com/docs/',
#result nvarchar(max) = ''
DECLARE #t TABLE(Resp nvarchar(max))
declare #timeStamp nvarchar(50) = convert(nvarchar(50),CURRENT_TIMESTAMP,127)
declare #CurDate date = dateadd(day,0,getdate())
--EXEC #hResult = sp_OACreate 'WinHttp.WinHttpRequest.5.1', #objectID OUT
EXEC #hResult = sp_OACreate 'MSXML2.XMLHTTP', #ObjectID OUT
SELECT #URI = #serv + '.newchange?ds='+CONVERT(nvarchar(10),#LastActDate,104)+'&df='+CONVERT(nvarchar(10),#CurActDate,104)+'&pardaily=1',
#methodName='GET',
#setTimeouts = 'setTimeouts(9000,90000,900000,9000000)'
EXEC #hResult = sp_OAMethod #objectID, 'open', null, #methodName, #URI, 'false'
EXEC #hResult = sp_OAMethod #objectID, #setTimeouts
EXEC #hResult = sp_OAMethod #objectID, 'send', null
INSERT INTO #t
EXEC sp_OAGetProperty #objectID, 'responseText'
SELECT top 1 #result = Resp
FROM #t
if #result is null
begin
delete from #t
exec sp_OAGetErrorInfo #objectID
exec sp_OADestroy #objectID
goto begining
end
else
begin
INSERT INTO Structure.MonthlyRow
SELECT #timeStamp, #result
end
END
When I run this SP like
EXEC [Structure].[GetXML] '2016-06-01', '2016-07-01'
I got a row in Structure.MonthlyRow table with correct timestamp and response (the average length is ~70k symbols)
Here is creation script of a table:
CREATE TABLE Structure.MonthlyRow(
[timestamp] nvarchar(50) NOT NULL,
[RowResp] nvarchar(max) NULL,
CONSTRAINT [PK_dDayly] PRIMARY KEY CLUSTERED ([timestamp] DESC))
If I create a job that launch this SP I get a row in table with results, and the length of result is 512 symbols! It is a proper part of XML that looks like it was truncated from nvarchar(max) to nvarchar(512), but I have no variables or table columns with length of 512 that are used.
What have I tried:
Run as user with my account in Job Step properties;
Job was started by schedule or manually;
Add WITH EXECUTE AS OWNER in SP;
Tried using WinHttp.WinHttpRequest.5.1 and MSXML2.XMLHTTP.
Question:
What possibly could be a problem? Why I am getting correct results when I run my SP manually, and got only 512 symbols of response when run SP as job step?
Note:
Yes, I know that getting XML from web-service is better handled by PHP, C# or even PowerShell and if I can not find a solution I will use one of them.
add this line at the top of your sp or in the job before EXEC of your sp
SET TEXTSIZE 2147483647;
the problem is that jobs set a default
SET TEXTSIZE 1024
this limits data returned to 1024 chars (512 for nchars)
I have a scenario wherein i have to execute an SP for specified number of time(number of execution will be mentioned by user) without using loop.
My SP is setting an OUTPUT variable of varchar type. I am willing to insert the output of my SP into a temp table and use it for further processing.
I am unable to modify this SP into function as it contain an Update statement.
Kindly suggest if we can do so without loop.
Whit this solution you do not need an output; #res is your result directly set in a temp table.
CREATE PROCEDURE [dbo].[myStoredProc]
#Counter int,
#params nvarchar(64),
#CreateTable bit
AS
DECLARE #res varchar(64)
IF #CreateTable = 1
BEGIN
IF EXISTS (SELECT 1 FROM dbo.sysobjects WHERE id = object_id(N'[#tempTable]'))
DROP TABLE #tempTable
CREATE TABLE #tempTable ([Res] [nvarchar] (64))
END
SET #res = CONVERT(varchar(64), #Counter)
SET #Counter = #Counter - 1
IF #Counter > 0
exec myStoredProc #Counter, #params, 0
INSERT #tempTable VALUES (#res)
IF #CreateTable = 1
BEGIN
SELECT * FROM #tempTable
DROP TABLE #tempTable
END
GO
DECLARE #o varchar(64)
exec [myStoredProc] 5, '', 1
I am trying to read an XML file using OpenRowSet from a folder and have been unable to do so and get the error
Cannot bulk Load since the "' #FullFilename'' does not exist.
Would appreciate if one could suggest how I can correct the problem
to obtain all data from each of the XML files.
Thanks.
Code:
declare #Directory varchar(50)
select #Directory = 'E:\XML\'
DECLARE #CD TABLE (XMLData XML);
declare #FileExist int
DECLARE #FileName varchar(500),
#DeleteCommand varchar(1000),
#FullFileName varchar(500)
DECLARE #SQL NVARCHAR(1000),#xml xml
--This is so that we know how long the loop lasts
declare #LoopID int, #MaxID int
SELECT #LoopID = min(id),#MaxID = max(ID)
FROM #tempList
WHILE #LoopID <= #MaxID
BEGIN
SELECT #FileNAme = filename
from #tempList
where id = #LoopID
SELECT #FullFileName = #Directory + #FileName
print #FULLFileName
exec xp_fileexist #FullFileName , #FileExist output
if #FileExist =1 --sanity check in case some evil person removed the file
begin
---********************************Problem with #FullFileName----------------
INSERT INTO #CD
SELECT *
FROM OPENROWSET(BULK ''' + #FullFileName +''' ,Single_BLOB) as rs
---********************************------------
select * from #CD
--SET #DeleteCommand = 'del ' + #Directory + #FileName
--maybe you want to delete or move the file to another directory
-- ** here is how to delete the files you just imported
-- uncomment line below to delete the file just inserted
--EXEC MASTER..XP_CMDSHELL #DeleteCommand
-- ** end of here is how to delete the files
end
--Get the next id, instead of +1 we grab the next value in case of skipped id values
SELECT #LoopID = min(id)
FROM #tempList
where id > #LoopID
END
select * from #tempList
This works and I am able to get the XML data from the specified file
DECLARE #CD TABLE (XMLData XML);
Declare #get_GeneralID bigint
INSERT INTO #CD
SELECT *
FROM OPENROWSET(BULK N'E:\XML\TestResult.XML', SINGLE_BLOB) rs;
select * from #CD
PS: I have put together from code that I found from the web.
I don't think you can use T-SQL variables in your OPENROWSET command - those things need to be fully and explicitly spelled out.
If you need to do this over a list of XML files, you'll have to create your T-SQL command as a string and then use dynamic SQL to execute that command.
I have an arbitrary list of values and I want to delete records across multiple tables using T-SQL. I would like to re-use the script in the future with different lists of values. This is for debugging purposes only (I just want to clear out records so they can be re-imported with the new version of the software), so it doesn't need to be pretty.
So far I have:
DECLARE #RequestIDList table(Request_ID nvarchar(50) NOT NULL)
INSERT INTO #RequestIDList (Request_ID) VALUES
('00987172'),
('01013218'),
('01027886'),
('01029552'),
('01031476'),
('01032882'),
('01033085'),
('01034446'),
('01039261')
DELETE FROM Request WHERE Request_ID IN (SELECT Request_ID FROM #RequestIDList)
DELETE FROM RequestTest WHERE Request_ID IN (SELECT Request_ID FROM #RequestIDList)
It seems to work, but is there a better way? I can't seem to work out how to use a variable directly with an IN clause (e.g. "WHERE Request_ID IN #RequestIDList").
Quick script:
SET NOCOUNT ON
-- Temp table so it can be joined against in dynamic SQL
IF OBJECT_ID('tempdb..#RequestIDList') IS NOT NULL
DROP TABLE #RequestIDList
GO
CREATE TABLE #RequestIDList (Request_ID nvarchar(50) NOT NULL)
INSERT INTO #RequestIDList (Request_ID) VALUES
('00987172'),('01013218'),('01027886'),('01029552'),
('01031476'),('01032882'),('01033085'),('01034446'),
('01039261')
DECLARE #TableList TABLE (TableName NVARCHAR(128) NOT NULL)
INSERT #TableList VALUES
('Request'),
('RequestTest')
DECLARE
#sqlcmd VARCHAR(4000),
#table VARCHAR(128)
-- Loop through the tables in your delete list
DECLARE c CURSOR LOCAL FORWARD_ONLY STATIC READ_ONLY FOR
SELECT TableName
FROM #TableList
ORDER BY TableName
OPEN c
FETCH NEXT FROM c INTO #table
WHILE ##FETCH_STATUS = 0
BEGIN
-- Assuming all tables in schema dbo
-- Assuming all tables have column Request_ID
SET #sqlcmd = 'DELETE FROM t FROM ' + QUOTENAME(#table)
+ ' t JOIN #RequestIDList r ON r.Request_ID = t.Request_ID'
-- PRINT #sqlcmd
EXEC (#sqlcmd)
FETCH NEXT FROM c INTO #table
END
CLOSE c
DEALLOCATE c
-- Clean up
DROP TABLE #RequestIDList
First you need to create a function which parses the input
CREATE FUNCTION inputParser (#list nvarchar(MAX))
RETURNS #tbl TABLE (number int NOT NULL) AS
BEGIN
DECLARE #pos int,
#nextpos int,
#valuelen int
SELECT #pos = 0, #nextpos = 1
WHILE #nextpos > 0
BEGIN
SELECT #nextpos = charindex(',', #list, #pos + 1)
SELECT #valuelen = CASE WHEN #nextpos > 0
THEN #nextpos
ELSE len(#list) + 1
END - #pos - 1
INSERT #tbl (number)
VALUES (convert(int, substring(#list, #pos + 1, #valuelen)))
SELECT #pos = #nextpos
END
RETURN
END
Then use that function in the SP
CREATE PROCEDURE usp_delete
#RequestIDList varchar(50)
AS
Begin
DELETE FROM Request as req inner join
inputParser (#RequestIDList) i on req.Request_ID = i.number
End
EXEC usp_delete '1, 2, 3, 4'
For furthur details please have a look at this article .It explains differnt methods depending on the sql server version .For SQl server 2008 it uses TVP which further simplifies the input parser