Subject:
I've got a report that I create every month. The creation of report consists of 2 steps:
Get an XML from our service and store it in DB;
Parse XML and create file.
For the last few month I've created report in manual mode. And now I want to automate this stuff.
But here comes a
Problem:
The second step (parsing XML and file creation) runs like a charm, but with first step I'm observing weird behaviour.
I got Stored Procedure which gets XML:
ALTER PROCEDURE [Structure].[GetXML]
#LastActDate date,
#CurActDate date
AS
BEGIN
SET NOCOUNT ON;
begining:
DECLARE #URI varchar(2000),
#methodName varchar(50),
#objectID int,
#hResult int,
#setTimeouts nvarchar(255),
#serv nvarchar(255) = 'http://example.com/docs/',
#result nvarchar(max) = ''
DECLARE #t TABLE(Resp nvarchar(max))
declare #timeStamp nvarchar(50) = convert(nvarchar(50),CURRENT_TIMESTAMP,127)
declare #CurDate date = dateadd(day,0,getdate())
--EXEC #hResult = sp_OACreate 'WinHttp.WinHttpRequest.5.1', #objectID OUT
EXEC #hResult = sp_OACreate 'MSXML2.XMLHTTP', #ObjectID OUT
SELECT #URI = #serv + '.newchange?ds='+CONVERT(nvarchar(10),#LastActDate,104)+'&df='+CONVERT(nvarchar(10),#CurActDate,104)+'&pardaily=1',
#methodName='GET',
#setTimeouts = 'setTimeouts(9000,90000,900000,9000000)'
EXEC #hResult = sp_OAMethod #objectID, 'open', null, #methodName, #URI, 'false'
EXEC #hResult = sp_OAMethod #objectID, #setTimeouts
EXEC #hResult = sp_OAMethod #objectID, 'send', null
INSERT INTO #t
EXEC sp_OAGetProperty #objectID, 'responseText'
SELECT top 1 #result = Resp
FROM #t
if #result is null
begin
delete from #t
exec sp_OAGetErrorInfo #objectID
exec sp_OADestroy #objectID
goto begining
end
else
begin
INSERT INTO Structure.MonthlyRow
SELECT #timeStamp, #result
end
END
When I run this SP like
EXEC [Structure].[GetXML] '2016-06-01', '2016-07-01'
I got a row in Structure.MonthlyRow table with correct timestamp and response (the average length is ~70k symbols)
Here is creation script of a table:
CREATE TABLE Structure.MonthlyRow(
[timestamp] nvarchar(50) NOT NULL,
[RowResp] nvarchar(max) NULL,
CONSTRAINT [PK_dDayly] PRIMARY KEY CLUSTERED ([timestamp] DESC))
If I create a job that launch this SP I get a row in table with results, and the length of result is 512 symbols! It is a proper part of XML that looks like it was truncated from nvarchar(max) to nvarchar(512), but I have no variables or table columns with length of 512 that are used.
What have I tried:
Run as user with my account in Job Step properties;
Job was started by schedule or manually;
Add WITH EXECUTE AS OWNER in SP;
Tried using WinHttp.WinHttpRequest.5.1 and MSXML2.XMLHTTP.
Question:
What possibly could be a problem? Why I am getting correct results when I run my SP manually, and got only 512 symbols of response when run SP as job step?
Note:
Yes, I know that getting XML from web-service is better handled by PHP, C# or even PowerShell and if I can not find a solution I will use one of them.
add this line at the top of your sp or in the job before EXEC of your sp
SET TEXTSIZE 2147483647;
the problem is that jobs set a default
SET TEXTSIZE 1024
this limits data returned to 1024 chars (512 for nchars)
Related
I have used the following code for years to convert a Bank Routing Number into the Bank Name that gets inserted into a Payroll Stub. It recently stopped returning a value. No error. Just an empty string.
Has something changed with T-SQL that renders this obsolete? ^Is there a better way to do this?
-- stored procedure params: sample bank ID = 121000358
#bankid varchar(9)
#bankname varchar(100) OUTPUT
exec sp_configure 'Ole Automation Procedures', 1;
RECONFIGURE;
declare #obj int
declare #url varchar(60)
declare #response varchar(500)
declare #bankinfo varchar(500)
set #url = 'https://www.routingnumbers.info/api/data.json?rn=' + #bankid
exec sp_OACreate 'MSXML2.XMLHTTP',#obj OUT;
exec sp_OAMethod #obj,'open',NULL,'get',#url,'false'
exec sp_OAMethod #obj,'send'
exec sp_OAMethod #obj,'responseText',#response OUTPUT
set #bankinfo = (select #response [response])
declare #left varchar(500)
declare #right varchar(500)
declare #delimiter1 varchar(18) = '"customer_name": "'
declare #delimiter2 varchar(6) = '"'
declare #outStr varchar(200)
declare #pos int = (charindex(#delimiter1,#bankinfo) + 18)
set #bankinfo = substring(#bankinfo,#pos,len(#bankinfo))
set #pos = charindex(#delimiter2,#bankinfo)
set #bankinfo = substring(#bankinfo,0,#pos)
set #bankname = #bankinfo
--select #bankname
exec sp_OADestroy #obj
exec sp_configure 'Ole Automation Procedures', 0;
RECONFIGURE;
I figured it out. The Ops team at my company isolated this SQL server from the internet. Thus the call to the URL comes back empty, but when I manually make the call from my laptop, it makes the trip there and back.
I have my sql returning information and building a url.
SELECT Address,Address2,'https://maps.googleapis.com/maps/api/distancematrix/xml?units=imperial&origins='+Address+'&destinations='+Address2+'&key=xxx' URL
FROM table
And if I take the URL that's created above and copy it into a browser the needed XML appears in the screen. But how can I have the XML within SQL so I can parse only specific data such as travel time?
So now I have the following code that provides me with the time:
DECLARE #Address0 varchar(100) =null
DECLARE #Address1 varchar(100)=null
SET #Address0 = 'xxx'
SET #Address1 = 'yyy'
Declare #Object as Int;
Declare #ResponseText as Varchar(8000);
Declare #serviceUrl as varchar(500)
set #serviceUrl = 'https://maps.googleapis.com/maps/api/distancematrix/xml?units=imperial&origins=' +#Address0+
'&destinations=' +#Address1 +'&key=keyfromgoogle'
Exec sp_OACreate 'MSXML2.XMLHTTP', #Object OUT;
Exec sp_OAMethod #Object, 'open', NULL, 'get',
#serviceUrl, --Your Web Service Url (invoked)
'false'
Exec sp_OAMethod #Object, 'send'
Exec sp_OAMethod #Object, 'responseText', #ResponseText OUTPUT
Declare #Response as XML
SET #Response = CAST(#ResponseText AS XML);
Declare #Time as varchar(20)
Begin
set #Time=#Response.value('(DistanceMatrixResponse/row/element/duration/text)[1]', 'varchar(20)')
End
select
#Time as Time
So I'm thinking I create a temp table to hold my addresses and then a cursor to run through and add the time to each row of the temp table?
Good day.
I have main task to do is next:
"Some" programm that has databases MonitorEDTest and DecNet in MSSQL 2012, i need to make an xml-file with detailed information from 4-5 tables in this database, which creates when StatusId of one entry becomes 150.
Now, i'm trying to solve this like this:
I made trigger on table with column StatusId, so it triggers when StatusId is 150. Made it with cursor for multiple entries with this status.
ALTER TRIGGER [dbo].[DT_Update_NEW]
ON [MonitorEDTest].[dbo].[LOG_DECL]
FOR UPDATE
AS
BEGIN
SET NOCOUNT ON;
if exists (select * from inserted where StatusId = 150)
begin
declare #nd nvarchar (100), #ProcessId nvarchar (100)
declare DTcursor cursor for
select [ND], [ProcessId] from inserted
open DTcursor;
FETCH NEXT FROM DTcursor into #nd, #ProcessId
WHILE ##FETCH_STATUS = 0
BEGIN
exec dbo.DT_info_Broker #nd, #ProcessId
FETCH NEXT FROM DTcursor into #nd, #ProcessId
END
CLOSE DTcursor
deallocate DTcursor
end
END
Then this trigger uses procedure dbo.DT_info_Broker that creates full xml-file with select...join..for xml statements, after that full xml code sends with service broker to himself (made ServiceBroker on base MonitorEDTest which contains table for trigger, and whole info takes from other DB DecNet).
ALTER PROCEDURE [dbo].[DT_Info_Broker]
#nd nvarchar (200), #ProcessId nvarchar (200)
AS
BEGIN
SET NOCOUNT ON;
declare #base nvarchar(100), #g07 nvarchar(1000), #sql nvarchar (MAX), #flag int, #code1 nvarchar (2000), #code2 nvarchar (1000), #code3 nvarchar (1000), #code4 nvarchar(1000), #code nvarchar(4000), #codeold nvarchar (1000),
#bat nvarchar(1000), #pwshell nvarchar(1000), #batdel nvarchar(1000), #msg xml
set #base = 'DecNet'
set #msg = (SELECT (select.....)for xml path(''), root('DT'))
declare #ch uniqueidentifier
begin dialog conversation #ch from service Monitor to service 'Monitor' on contract [Monitor_Contract] with encryption = off
select #ch, #nd, #g07, #msg
;send on conversation #ch message type [Monitor_Message] (#msg)
EXEC [MonitorEDTest].[dbo].[Que_Broker]
END
Then i had two ideas, first i tried to make activation procedure for Service Broker (SB) to make it like trigger for sending xml from 2., but it didn’t work, so after sending to SB I executed another procedure dbo.Que_Broker.
This procedure is similar to how people usually “receives” message from SB, code is above. Also on that procedure I take from ready xml-code one tag g07 for making name of xml output file like #g07.xml ~ 10000000-170718-0000000.xml
ALTER PROCEDURE [dbo].[Que_Broker]
AS
BEGIN
while 1 = 1
begin
declare #command nvarchar(2000), #xmlint int, #XmlText nvarchar(max), #g07 nvarchar (100)
declare #count int, #ch uniqueidentifier, #retvalue bit --, #msgtype sysname, #body varbinary(max)
select #count = count(*) from Monitor_Queue
if (#count = 0) break
set #xmlint = 0
select top (1) #XmlText = cast(message_body as nvarchar(max)), #ch = conversation_handle from Monitor_Queue where message_type_name = 'Monitor_Message' order by conversation_handle
exec sp_xml_preparedocument #xmlint OUTPUT, #XmlText
select #g07 = [text] from OPENXML (#xmlint, 'DT', 1) where [parentid] = 4
exec sp_XML_removedocument #xmlint
set #g07 = (select replace (#g07, '/', '-'))
set #command = 'bcp "receive top (1) cast(message_body as xml) from Monitor_Queue" queryout D:\' + #g07 + '.xml -T -x -c -C 1251 -d MonitorEDTest'
select #command, #g07
--waitfor delay '00:00:01'
EXEC master..xp_cmdshell #command
--break
end conversation #ch with cleanup
end
END
So the main problem is next, i wrote on last procedure code which goes to Queues of SB and while it has count>0 it makes bcp “receive top (1) cast(message_body as xml) from Monitor_Queue”...queryout “*.xml”
Then the conversation ends.
When i was checking how it works, everything worked fine, but bcp didn’t copy any entry of BS and when i check messages in table queue, the message still there. When i’m trying to execute that procedure manually bcp works fine.
output
------------
NULL
Starting copy...
NULL
0 rows copied.
Network packet size (bytes): 4096
Clock Time (ms.) Total : 1
NULL
Also before idea with SB, i tried to do something like that with creating dynamic steps in job with sp_update_jobstep and sp_start_job, but it didn’t work, because of multiple entry within one time (like update top 5 set statusid = 150 where statusid = 150). Job couldn’t be opened by next entry because first didn’t finish.
Sorry for some mistakes, but first time writing here, hope to see some help and learn more of using sql, because I learned everything by myself and from some courses.
I need to create multiple xml files (per row) from single query which I did. My query is generating 10,000 xml files as you know SQL Server assign different name for each xml file now I need to save them in c:\ automatically. I can see all the files but I can’t save them any advise or help will highly be appreciated. One more hint: from SQL server we can save each xml one by one from SQL server File tab.
CREATE TABLE dbo.sample(
[BTno] [nvarchar](25) NULL,
[First_Name] [nvarchar](35) NULL,
[Last_Name] [nvarchar](35) NULL,
[BTid] [nvarchar](15) NULL,
) ON [PRIMARY];
insert into dbo.sample values('1B','Vartan','Sarkis','69876');
insert into dbo.sample values('2B','Anoush','Eric','87656');
insert into dbo.sample values('3B','Lucine','Arpiar','65467');
insert into dbo.sample values('4B','Anum','Noor','98076');
insert into dbo.sample values('5B','Abercio','Banninq','34897');
insert into dbo.sample values('1C','Gaea','Nishan','29841');
insert into dbo.sample values('7B','Marilyn','Vahe','78903');
insert into dbo.sample values('2Z','Bansi','Aakarshan','34905');
insert into dbo.sample values('9S','Eric','Abban','45892');
insert into dbo.sample values('12B','Dave','Tate','19994');
-- Here is the Query that generates multiple xml files
select ((select * from dbo.sample a where a.BTid = b.BTid
for xml path('Row'),TYPE, ROOT('BT')
)) from dbo.sample b
I cheated and used someone else's function for the actual write. Note that you'll need 'Ole Automation Procedures' enabled for the write procedure to work. Some company security policies get a little touchy about these things being enabled. Basically I assume if this isn't enabled, you have the ability to do so. The write function will also overwrite files with the same filename, though that may not be relevant for your need.
declare #thexml xml
declare #xmlstring varchar(max)
declare #path varchar(255)
declare #filename_start varchar(100)
declare #filename varchar(100)
declare #count int
set #path='C:\testdata\'
set #filename_start = 'xmlname'
set #count = 0
declare xcursor cursor for
select ((select * from dbo.sample a where a.BTid = b.BTid
for xml path('Row'),TYPE, ROOT('BT')
)) from dbo.sample b
open xcursor
fetch xcursor into #thexml
while ##fetch_status <> -1 begin
set #count = #count + 1
set #filename = #filename_start + cast(#count as varchar) + '.xml'
set #xmlstring = cast(#thexml as varchar(max))
exec master.dbo.spWriteStringToFile #xmlstring,#path,#filename
fetch xcursor into #thexml
end
deallocate xcursor
Enabling OLE - http://msdn.microsoft.com/en-us/library/ms191188.aspx
Credit for write procedure - https://www.simple-talk.com/sql/t-sql-programming/reading-and-writing-files-in-sql-server-using-t-sql/
Actual write procedure (must change from alter to create) - https://www.simple-talk.com/code/WorkingWithFiles/spWriteStringTofile.txt
Try with this script, maybe will be useful:
CREATE PROCEDURE saveFile (#file varchar(1000), #txt VARCHAR(8000)) AS
BEGIN
DECLARE #sys_obj INT
DECLARE #rst INT
DECLARE #fID INT
EXECUTE #rst = sp_OACreate 'Scripting.FileSystemObject', #sys_obj OUT
EXECUTE sp_OAMethod #sys_obj, 'FileExists', #rst OUT, #file
IF #rst=0
BEGIN
EXECUTE sp_OAMethod #sys_obj, 'OpenTextFile', #fID OUT, #file, 8, 1
EXECUTE sp_OAMethod #fID, 'WriteLine', Null, #txt
EXECUTE sp_OADestroy #fID
EXECUTE sp_OADestroy #sys_obj
END
END
And with this line, could be save to disk a String (for example a XML):
EXEC saveFile #file = 'C:\test.xml', #txt = '<xml><test>123</test></xml>'
Here other query for save ALL your fields that returns your query:
DECLARE cur CURSOR LOCAL FAST_FORWARD FOR
select CAST(((select * from dbo.sample a where a.BTid = b.BTid
for xml path('Row'),TYPE, ROOT('BT')
)) AS VARCHAR(MAX)) AS TXT from dbo.sample b
--HERE GO YOUR QUERY
/*Example
SELECT '<xml><test>111</test></xml>' AS TXT
UNION
SELECT '<xml><test>222</test></xml>' AS TXT
UNION
SELECT '<xml><test>333</test></xml>' AS TXT
*/
DECLARE #v_count AS INTEGER
DECLARE #v_txt AS VARCHAR(8000)
DECLARE #v_filename AS VARCHAR(100)
SET #v_count = 1
SET #v_txt = ''
OPEN cur FETCH NEXT FROM cur INTO #v_txt
WHILE ##FETCH_STATUS = 0
BEGIN
SET #v_filename = 'C:\test' + CAST(#v_count AS VARCHAR) + '.xml'
EXEC saveFile #file = #v_filename, #txt = #v_txt
SET #v_count = #v_count + 1
FETCH NEXT FROM cur INTO #v_txt
END
CLOSE cur
DEALLOCATE cur
Can you please tell how to loop through a folder having .txt files and get the latest file and add the content into the table using SQL Server 2005 stored procedure??
Thanks in Advance.
SateeshChandra.
I would suggest that this is a better job for SQL CLR or external tools like a C# command line app. You can do this in various ways within SQL but they're inherently insecure and potentially problematic. My approach is usually xp_cmdshell if CLR or external tools are not an option. It needs to be enabled first:
EXEC sp_configure 'show adv', 1;
GO
RECONFIGURE WITH OVERRIDE;
GO
EXEC sp_configure 'xp_cmdshell', 1;
GO
RECONFIGURE WITH OVERRIDE;
GO
EXEC sp_configure 'show adv', 0;
GO
RECONFIGURE WITH OVERRIDE;
GO
Then you can do something like this:
SET NOCOUNT ON;
DECLARE
#folder NVARCHAR(2048),
#cmd NVARCHAR(MAX);
SET #folder = N'C:\path\';
SET #cmd = N'dir ' + #folder + '*.txt';
CREATE TABLE #x(n NVARCHAR(2048));
INSERT #x EXEC [master].dbo.xp_cmdshell #cmd;
DECLARE #filename NVARCHAR(2048);
;WITH x(n) AS (SELECT n FROM #x WHERE ISDATE(LEFT(n, 20)) = 1)
SELECT TOP 1 #filename = n FROM x
ORDER BY CONVERT(DATETIME, LEFT(n, 20)) DESC;
SET #cmd = N'type ' + #folder + SUBSTRING(#filename,
LEN(#filename) - CHARINDEX(' ', REVERSE(#filename)) + 2,
2048);
CREATE TABLE #y(n NVARCHAR(MAX));
INSERT #y EXEC [master].dbo.xp_cmdshell
-- no idea what "add the content into the table" means
-- but you can work with this:
SELECT n FROM #y;
DROP TABLE #x, #y;
Note 1: The width of the date information in the #x.n column is going to vary depending on your regional settings / locale. You may need to experiment.
Note 2: The determination of the file name assumes that your file names do not have spaces. If they do, then at least one line above will need to be revisited.
There is a handy udf located in this blog post which you could probably adapt. I'm going to post the contents of the script and usage incase the site below dies:
Create FUNCTION [dbo].[uftReadfileAsTable]
(
#Path VARCHAR(255),
#Filename VARCHAR(100)
)
RETURNS
#File TABLE
(
[LineNo] int identity(1,1),
line varchar(8000))
AS
BEGIN
DECLARE #objFileSystem int
,#objTextStream int,
#objErrorObject int,
#strErrorMessage Varchar(1000),
#Command varchar(1000),
#hr int,
#String VARCHAR(8000),
#YesOrNo INT
select #strErrorMessage='opening the File System Object'
EXECUTE #hr = sp_OACreate 'Scripting.FileSystemObject' , #objFileSystem OUT
if #HR=0 Select #objErrorObject=#objFileSystem, #strErrorMessage='Opening file "'+#path+'\'+#filename+'"',#command=#path+'\'+#filename
if #HR=0 execute #hr = sp_OAMethod #objFileSystem , 'OpenTextFile'
, #objTextStream OUT, #command,1,false,0--for reading, FormatASCII
WHILE #hr=0
BEGIN
if #HR=0 Select #objErrorObject=#objTextStream,
#strErrorMessage='finding out if there is more to read in "'+#filename+'"'
if #HR=0 execute #hr = sp_OAGetProperty #objTextStream, 'AtEndOfStream', #YesOrNo OUTPUT
IF #YesOrNo<>0 break
if #HR=0 Select #objErrorObject=#objTextStream,
#strErrorMessage='reading from the output file "'+#filename+'"'
if #HR=0 execute #hr = sp_OAMethod #objTextStream, 'Readline', #String OUTPUT
INSERT INTO #file(line) SELECT #String
END
if #HR=0 Select #objErrorObject=#objTextStream,
#strErrorMessage='closing the output file "'+#filename+'"'
if #HR=0 execute #hr = sp_OAMethod #objTextStream, 'Close'
if #hr<>0
begin
Declare
#Source varchar(255),
#Description Varchar(255),
#Helpfile Varchar(255),
#HelpID int
EXECUTE sp_OAGetErrorInfo #objErrorObject,
#source output,#Description output,#Helpfile output,#HelpID output
Select #strErrorMessage='Error whilst '
+coalesce(#strErrorMessage,'doing something')
+', '+coalesce(#Description,'')
insert into #File(line) select #strErrorMessage
end
EXECUTE sp_OADestroy #objTextStream
-- Fill the table variable with the rows for your result set
RETURN
END
Usage:
Select line from
Dbo.uftReadfileAsTable('MyPath','MyFileName')
where line not like '#%'
Just fill in an existing file name and path to the file you wish to read, instead of 'MyPath' and 'MyFileName', and away you go.
(Note: I've included the original source because I've been downvoted because a link to a solution for another question died : ( )
My first thought was that this is an ideal candidate for SSIS - except that SSIS has a fairly steep learning curve if you are new to it
TSQL Solution - a couple of excellent articles to get you started:
http://www.mssqltips.com/tip.asp?tip=1263
http://www.simple-talk.com/sql/t-sql-programming/reading-and-writing-files-in-sql-server-using-t-sql/
SSIS Solution: - one article to get you started:
http://www.sqlis.com/post/Looping-over-files-with-the-Foreach-Loop.aspx
I know that this is a very old post but I found that the solution in the following link worked perfectly for me: http://www.databaseskill.com/2219220/