I am calling an (GET) API from SQL Server and storing the results in JSON. My running code is as follow:
DECLARE #Object AS Int;
DECLARE #hr INT
DECLARE #json AS TABLE(Json_Table nvarchar(max))
declare #username varchar(50), #password varchar(50), #encoded_base64 varchar(max), #auth varchar(max)
declare #source varbinary(max)
set #username = 'demousername'
set #password = 'demopassword'
SET #source = CONVERT(varbinary(max), #username + ':' + #password)
SET #encoded_base64 = CAST(N'' AS xml).value('xs:base64Binary(sql:variable("#source"))', 'varchar(max)')
SET #auth = 'Basic ' + #encoded_base64
Exec #hr=sp_OACreate 'MSXML2.ServerXMLHTTP.6.0', #Object OUT;
IF #hr <> 0 EXEC sp_OAGetErrorInfo #Object
Exec #hr=sp_OAMethod #Object, 'open', NULL, 'get',
'http://demourl',
'false'
IF #hr <> 0 EXEC sp_OAGetErrorInfo #Object
EXEC #hr = sp_OAMethod #Object, 'setRequestHeader', NULL, 'Authorization', #auth;
IF #hr <> 0 EXEC sp_OAGetErrorInfo #Object
Exec #hr=sp_OAMethod #Object, 'send'
IF #hr <> 0 EXEC sp_OAGetErrorInfo #Object
Exec #hr=sp_OAMethod #Object, 'responseText', #json OUTPUT
IF #hr <> 0 EXEC sp_OAGetErrorInfo #Object
INSERT into #json (Json_Table) exec sp_OAGetProperty #Object, 'responseText'
select * from #json
SELECT * FROM OPENJSON((select * from #json), N'$.data')
WITH (
[Column1] nvarchar(max) N'$.column1.id'
)
EXEC sp_OADestroy #Object
The issue is that my API is returning a good amount of data (around 4 MB while storing the results in a file on disk manually) and SQL Server is presenting me with the following error:
Json_Table
-----------------------------------------------------------------------------------------
Response size 1164KB has exceeded maximum supported size 1024KB. However, the output is still available to subsequent steps in the workflow.
However, my code is running fine without any issues if I have smaller data.
Please help me to address this error and move on.
Thanks in advance.
P.S I need to use the above mentioned methodology (i.e. using SQL to call my API) as a must so using any other tool like VS, SSIS etc. won't be helpful.
Related
I try to create a new procedure to getting real currency exchange rates from website. To do this one, i use OLE Automation Stored Procedures. But i have encountered a problem when i trying to get data with sp_OAGetProperty from XML.
When i try to inserted a XML into a temporary table with sp_OASetProperty, it gives me an error. I tried many things to fix that but that solutions mostly about XML and i don't know about XML very much. Here is the problematic part of my code:
DECLARE #OBJ AS INT
DECLARE #RESULT AS INT
EXEC #RESULT=sp_OACreate 'MSXML2.XMLHTTP', #OBJ OUT
EXEC #RESULT=sp_OAMethod #OBJ , 'open' , null , 'GET', #url, false
EXEC #RESULT=sp_OAMethod #OBJ, SEND, NULL,''
IF OBJECT_ID('tempdb..#XML') IS NOT Null DROP TABLE #XML
CREATE TABLE #XML (STRXML VARCHAR(max))
INSERT INTO #XML (STRXML) EXEC #RESULT = sp_OASetProperty #OBJ,'responseXML.xml' ---> I am getting an error in this stage.
SELECT * FROM #XML
I expect the xml like this:
<?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="isokur.xsl"?> <Tarih_Date Tarih="18.09.2019" Date="09/18/2019" Bulten_No="2019/175"> ....... </Tarih_Date>
But the output is an empty result. When i exec sp_OAGetErrorInfo to get error it shows that error:
Error Code: 0x80042727
Description: sp_OASetProperty usage: ObjPointer int IN, PropertyName varchar IN, #setval <any> IN [, additional indexing IN params].
I would advise you to use an SQL CLR stored procedure to make web service calls and avoid MSXML2.XMLHTTP. MSXML2.XMLHTTP in particular, and OLE objects in general, are not thread safe meaning that concurrent calls from SQL may cause problems.
The following, based on Call a webservice from TSQL (Stored Procedure) using MSXML, does work on SQL Server 2017 (64-bit)...
/*
exec sp_configure 'Ole Automation Procedures', 1
go
reconfigure
go
*/
declare #TableVariable table (result nvarchar(max));
declare #hResult int, #object int, #Status nvarchar(max), #StatusText nvarchar(max), #Response nvarchar(max)
declare #url nvarchar(max) = 'http://www.geoplugin.net/xml.gp?ip=8.8.8.8'
declare #FailPoint nvarchar(max) = 'Create'
exec #hResult = sp_OACreate 'MSXML2.XMLHTTP', #object out
if (#hResult = 0)
begin
set #FailPoint = 'Open'
exec #hResult = sp_OAMethod #object , 'Open', null, 'GET', #url, 0
end
if (#hResult = 0)
begin
set #FailPoint = 'Send'
exec #hResult = sp_OAMethod #object, 'Send'
end
if (#hResult = 0)
begin
set #FailPoint = 'Status'
delete #TableVariable
insert #TableVariable exec #hResult = sp_OAMethod #object, 'Status'
select #Status=result from #TableVariable;
end
if (#hResult = 0)
begin
set #FailPoint = 'StatusText'
delete #TableVariable
insert #TableVariable exec #hResult = sp_OAMethod #object, 'StatusText'
select #StatusText=result from #TableVariable
end
if (#hResult = 0)
begin
set #FailPoint = 'ResponseText'
delete #TableVariable
insert #TableVariable exec #hResult = sp_OAMethod #object, 'ResponseText'
select #Response=result from #TableVariable
end
if (#hResult != 0)
begin
declare #Source nvarchar(max), #Description nvarchar(max)
exec sp_OAGetErrorInfo #object, #Source out, #Description out
select
hResult = convert(varbinary(4), #hResult),
Source = #Source,
Description = #Description,
FailPoint = #FailPoint
goto Destroy
end
select [Status]=#Status, [StatusText]=#StatusText, [Response]=#Response
Destroy:
exec #hResult = sp_OADestroy #object
go
If you're confident that the response is valid XML then you can follow up with cast(#Response as xml).
I have my sql returning information and building a url.
SELECT Address,Address2,'https://maps.googleapis.com/maps/api/distancematrix/xml?units=imperial&origins='+Address+'&destinations='+Address2+'&key=xxx' URL
FROM table
And if I take the URL that's created above and copy it into a browser the needed XML appears in the screen. But how can I have the XML within SQL so I can parse only specific data such as travel time?
So now I have the following code that provides me with the time:
DECLARE #Address0 varchar(100) =null
DECLARE #Address1 varchar(100)=null
SET #Address0 = 'xxx'
SET #Address1 = 'yyy'
Declare #Object as Int;
Declare #ResponseText as Varchar(8000);
Declare #serviceUrl as varchar(500)
set #serviceUrl = 'https://maps.googleapis.com/maps/api/distancematrix/xml?units=imperial&origins=' +#Address0+
'&destinations=' +#Address1 +'&key=keyfromgoogle'
Exec sp_OACreate 'MSXML2.XMLHTTP', #Object OUT;
Exec sp_OAMethod #Object, 'open', NULL, 'get',
#serviceUrl, --Your Web Service Url (invoked)
'false'
Exec sp_OAMethod #Object, 'send'
Exec sp_OAMethod #Object, 'responseText', #ResponseText OUTPUT
Declare #Response as XML
SET #Response = CAST(#ResponseText AS XML);
Declare #Time as varchar(20)
Begin
set #Time=#Response.value('(DistanceMatrixResponse/row/element/duration/text)[1]', 'varchar(20)')
End
select
#Time as Time
So I'm thinking I create a temp table to hold my addresses and then a cursor to run through and add the time to each row of the temp table?
I've created a stored procedure that uses a temp table to take the massive results of a Bing maps XML and convert it to varchar so that I can use it to retrieve several fields. The reason for converting it to varchar was because the XML was too large for the sp_OAMethod and my variable was always blank.
set #serviceUrl = 'https://dev.virtualearth.net/REST/v1/Routes/Truck?wp.0=' + #ToAddress + '&wp.1=' + #FromAddress + '&vehicleHazardousMaterials=Flammable&output=xml&key=XXX-000-XXX'
Exec sp_OACreate 'MSXML2.XMLHTTP', #Object OUT;
Exec sp_OAMethod #Object, 'open', NULL, 'get',#serviceUrl, 'false'
Exec sp_OAMethod #Object, 'send'
Exec sp_OAMethod #Object, 'responseText', #ResponseText OUTPUT
exec #returnCode = sp_oamethod #Object, 'read', #ResponseText out, -1
Create table #tmp(dt xml)
insert into #tmp
exec #hr = sp_OAGetProperty #Object, 'responseXML.XML'
Set #ResponseText = Convert(varchar(max), (SELECT dt from #tmp))
Drop Table #tmp
select REPLACE(SUBSTRING(#ResponseText,PATINDEX('%<TravelDistance>%',#ResponseText),21),'<TravelDistance>','') Miles
,REPLACE(REPLACE(SUBSTRING(#ResponseText,PATINDEX('%<TravelDuration>%',#ResponseText),21),'<TravelDuration>',''),'<','')/60 TravelMinutes
Above is not the complete code; I've edited it severely to make it easier to read.
I'm sure there is a better way to do it but this works and it's very exciting.
However, the plan was to create a function that could be used along with other ad-hoc queries to retrieve the same data. Not possible as a function because of the temp table or populating a table from within the function.
So does anyone have a way to make this possible?
You really should use CLR for this. You're a long way from a correct stored procedure here, and the sp_OAxxx procedures are best avoided. You're not handling the return code correctly, you're using the wrong HTTP component, and not parsing the results using the SQL Server's XML parser. It may seem initially like a CLR implementation is more work, but maintaining code that uses sp_OAxxx stored procedures and COM interop will always be painful.
There are very few people who understand how this code works, how to change it, and whether it's safe and reliable. The next person to own your codebase is probably not one of those people. It's not good to have code like that in your project.
If you do use CLR, you still shouldn't use a function (although you technically can). You shouldn't perform any external access in the middle of a query.
Anyway here's some fixes to your existing approach:
declare #serviceURL varchar(max)
declare #returnCode int
declare #errorMessage nvarchar(max)
declare #Object int
declare #hr int
declare #key varchar(2000) = 'Aoj...nFS'
declare #FromAddress nvarchar(max) = '7000 N SH 161, Irving TX 75039'
declare #ToAddress nvarchar(max) = '8617 Garland Rd, Dallas, TX 75218'
set #serviceUrl = 'https://dev.virtualearth.net/REST/v1/Routes/Truck?wp.0=' + #ToAddress + '&wp.1=' + #FromAddress + '&vehicleHazardousMaterials=Flammable&output=xml&key=' + #key
exec #hr = sp_OACreate 'MSXML2.ServerXMLHTTP', #Object OUT;
IF #hr <> 0
begin
set #errorMessage = concat('sp_OACreate failed ', convert(varchar(20),cast(#hr as varbinary(4)),1));
throw 60000, #errorMessage, 1;
end;
begin try
Exec #hr = sp_OAMethod #Object, 'open', NULL, 'get',#serviceUrl, 'false'
IF #hr <> 0
begin
set #errorMessage = concat('open failed ', convert(varchar(20),cast(#hr as varbinary(4)),1));
throw 60000, #errorMessage, 1;
end;
Exec #hr = sp_OAMethod #Object, 'send'
IF #hr <> 0
begin
set #errorMessage = concat('send failed ', convert(varchar(20),cast(#hr as varbinary(4)),1));
throw 60000, #errorMessage, 1;
end;
declare #responseCode int;
Exec #hr = sp_OAGetProperty #Object, 'status', #responseCode out
IF #hr <> 0 or #responseCode <> 200
begin
set #errorMessage = concat('send failed hr:', convert(varchar(20),cast(#hr as varbinary(4)),1),' http response code: ', #responseCode);
throw 60000, #errorMessage, 1;
end;
declare #tmp table(doc xml)
insert into #tmp
exec #hr = sp_OAGetProperty #Object, 'ResponseXML.XML'
IF #hr <> 0
begin
set #errorMessage = concat('ResponseXML.XML failed ', convert(varchar(20),cast(#hr as varbinary(4)),1));
throw 60000, #errorMessage, 1;
end;
WITH XMLNAMESPACES (DEFAULT 'http://schemas.microsoft.com/search/local/ws/rest/v1')
select doc.value('(//TravelDistance)[1]', 'float') TravelDistance,
doc.value('(//DistanceUnit)[1]', 'nvarchar(20)') DistanceUnit,
doc.value('(//TravelDuration)[1]', 'float') TravelDuration,
doc.value('(//DurationUnit)[1]', 'nvarchar(20)') DurationUnit
from #tmp;
end try
begin catch
exec #hr = sp_OADestroy #Object ;
throw;
end catch
I am using sql server 2016 to consume webservice through store procedure. I have a webservice in JSON returning the following:
["Name":"Rebecca","email":"rebecca#hotmail.com","ra":"12345"},
{"name":"Caroline","email":"caroline#hotmail.com","ra":"23456"},
{"name":"Vanessa","email":"vanessa#yahoo.com.br","ra":"99999"}]
I can consume it by passing a variable. If I leave my webservice to receive a parameter and pass it I can return the name and email of the student. But if I leave my webservice without the need to receive parameter and remove the procedure parameter pass and execute, my procedure returns null. What do I need to change in my code to be able to return the complete list of students my webservice exposes?
Here's my code:
create PROCEDURE webservice_parametros (#RA as varchar (5))
AS
BEGIN
DECLARE #OBJ INT;
DECLARE #URL VARCHAR(200);
DECLARE #RESPONSE VARCHAR(8000);
SET #URL = 'http://dominio:8080/v1/alunos/' + #RA
EXEC SP_OACREATE 'MSXML2.ServerXMLHttp', #OBJ out
EXEC SP_OAMETHOD #OBJ, 'OPEN', NULL, 'GET', #URL, FALSE
EXEC SP_OAMETHOD #OBJ, 'SEND'
exec SP_OAGETPROPERTY #OBJ, 'responseText', #RESPONSE out
EXEC SP_OADESTROY #OBJ
SELECT JSON_VALUE(#RESPONSE, '$.nome') as nome,
JSON_VALUE(#RESPONSE, '$.email') as email
END
execute webservice_parametros '12345'
My code that returns NULL
create PROCEDURE webservice
AS
BEGIN
DECLARE #OBJ INT;
DECLARE #URL VARCHAR(200);
DECLARE #RESPONSE VARCHAR(8000);
SET #URL = 'http://dominio:8080/v1/alunos/'
EXEC SP_OACREATE 'MSXML2.ServerXMLHttp', #OBJ out
EXEC SP_OAMETHOD #OBJ, 'OPEN', NULL, 'GET', #URL, FALSE
EXEC SP_OAMETHOD #OBJ, 'SEND'
exec SP_OAGETPROPERTY #OBJ, 'responseText', #RESPONSE out
EXEC SP_OADESTROY #OBJ
SELECT JSON_VALUE(#RESPONSE, '$.nome') as nome,
JSON_VALUE(#RESPONSE, '$.email') as email
END
execute webservice
Declare #Object as Int;
Declare #ResponseText as Varchar(8000);
Declare #Url as Varchar(MAX);
set #Url = 'http://mysite.ru/cgi-bin/my_xml.cgi'
Exec sp_OACreate 'MSXML2.XMLHTTP', #Object OUT;
Exec sp_OAMethod #Object, 'open', NULL, 'get', #Url, 'false'
Exec sp_OAMethod #Object, 'send'
Exec sp_OAMethod #Object, 'responseText', #ResponseText OUTPUT
Exec sp_OADestroy #Object
SELECT #ResponseText
XML length in url is 4210 and #ResponseText return NULL , when I change length to 3970 #ResponseText return me data. Does sp_OACreate have limit 400 ? If yes if it possible to avoid ?
despite the subject of your post i think that the issue is likely with sp_OAMethod and not sp_OACreate itself.
also IMHO accessing the web from sql code should be avoided at all costs but this is just my opinion because i don't like the idea having a RDBMS 'surfing the web'. ^^
to circumvent the limitation of sp_OAMethod you can try to elaborate an answer present on msdn.
your code should become something like this:
Declare #Object as Int;
Declare #ResponseText as Varchar(8000);
Declare #Url as Varchar(MAX);
set #Url = 'http://mysite.ru/cgi-bin/my_xml.cgi'
Exec sp_OACreate 'MSXML2.XMLHTTP', #Object OUT;
Exec sp_OAMethod #Object, 'open', NULL, 'get', #Url, 'false'
Exec sp_OAMethod #Object, 'send'
--Exec sp_OAMethod #Object, 'responseText', #ResponseText OUTPUT
INSERT #temptable ( appropriatefield )
EXEC #Result = sp_OAGetProperty #Obj, 'YourPropertyName'
Exec sp_OADestroy #Object
the solution requires a temp table with appropriate structure and datatype to store the value produced by the remote page and this should allow you to get more than 4k of data.
According to this thread on sqlservercentral.com, sp_OACreate is limited to 4000 characters.
A workaround is to split up the read into smaller "chunks" that are then concated together in SQL. Here is a code snippet from the above link, that might help you although it reads XML from file instead of through HTTP:
EXECUTE #hResult = sp_OACreate ''Scripting.FileSystemObject'' , #objFileSystem OUT
IF #hResult <> 0
BEGIN
EXEC sp_OAGetErrorInfo #objFileSystem, #ErrorSource OUT, #ErrorDesc OUT
SET #ErrorFailPoint = ''Creating FSO''
GOTO DestroyFSO
RETURN
END
SET #FileNameAndPath = #Path + ''\'' + #FileName
-- Read file
EXECUTE #hResult = sp_OAMethod #objFileSystem, ''OpenTextFile'', #objTextStream OUT, #FileNameAndPath, 1, false, 0--for reading, FormatASCII
IF #hResult <> 0
BEGIN
EXEC sp_OAGetErrorInfo #objFileSystem, #ErrorSource OUT, #ErrorDesc OUT
SET #ErrorFailPoint = ''Opening Reponse File''
GOTO Destroy
RETURN
END
SET #ResponseText = ''''
WHILE #hResult = 0
BEGIN
EXECUTE #hResult = sp_OAGetProperty #objTextStream, ''AtEndOfStream'', #YesOrNo OUTPUT
IF #hResult <> 0
BEGIN
EXEC sp_OAGetErrorInfo #objTextStream, #ErrorSource OUT, #ErrorDesc OUT
SET #ErrorFailPoint = ''Checking AtEndOfStream''
GOTO Destroy
RETURN
END
IF #YesOrNo <> 0
BREAK
EXECUTE #hResult = sp_OAMethod #objTextStream, ''Read'', #Chunk OUTPUT, 4000
IF #hResult <> 0
BEGIN
EXEC sp_OAGetErrorInfo #objTextStream, #ErrorSource OUT, #ErrorDesc OUT
SET #ErrorFailPoint = ''Reading Chunk''
GOTO Destroy
RETURN
END
SET #ResponseText = #ResponseText + ISNULL(#Chunk, '''')
END
EXECUTE #hResult = sp_OAMethod #objTextStream, ''Close''
IF #hResult <> 0
BEGIN
EXEC sp_OAGetErrorInfo #objTextStream, #ErrorSource OUT, #ErrorDesc OUT
SET #ErrorFailPoint = ''Closing Response File''
GOTO Destroy
RETURN
END
-- Record response info
SET #ResponseXml = CAST(#ResponseText AS XML)
Destroy:
EXEC sp_OADestroy #objTextStream
DestroyFSO:
EXEC sp_OADestroy #objFileSystem
This is what I use to overcome the limitation. I use it for a RESTful api communication. I can receive varchar(max) but am still limited on the amount of data I can send. This might get you where you need to be. The top 5 variables are the arguments I use for the sproc.
Declare #url as varchar(1024)
Declare #connection_type as varchar(6)='GET' --POST, PUT, GET DELETE
Declare #post_string as varchar(max)=null
Declare #response_text as Varchar(max)
Declare #content_type varchar(254)='application/json'
Declare #oa_object as Int;
Declare #err_code as Int
Declare #result_table Table (xml_result varchar(max))
Select #post_string=dbo.fn_regex_replace('([ ]{2,10})|\r|\n', #post_string,'') --remove carriage returns and multiple spaces
Exec #err_code=sp_OACreate 'MSXML2.ServerXMLHTTP.3.0', #oa_object OUT;
If #err_code<>0
Set #response_text=dbo.fn_oa_error_message(#oa_object)
Else
Begin
Exec #err_code=sp_OAMethod #oa_object, 'open', NULL, #connection_type, #url,'false','d0b1a0aaed2a529356471de4fe99cae2','8e7aa1a91fa68d06cd027914d3aa1140'
If #err_code<>0
Set #response_text='Open '+dbo.fn_oa_error_message(#oa_object)
Else
Begin
Exec #err_code=sp_OAMethod #oa_object, 'setRequestHeader', NULL, 'User-Agent', 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)'
If #err_code<>0
Set #response_text='setRequestHeader:User-Agent '+dbo.fn_oa_error_message(#oa_object)
Else
Begin
Exec #err_code=sp_OAMethod #oa_object, 'setRequestHeader', NULL, 'Content-Type', #content_type
If #err_code<>0
Set #response_text='setRequestHeader:Content-Type '+dbo.fn_oa_error_message(#oa_object)
Else
Begin
Exec #err_code=sp_OAMethod #oa_object, 'send', Null, #post_string
If #err_code<>0
Set #response_text='Send '+dbo.fn_oa_error_message(#oa_object)
Else
Begin
Set #response_text=null--make sure we don't return garbage
INSERT #result_table (xml_result)
Exec #err_code = sp_OAGetProperty #oa_object, 'responseText'
If #err_code<>0
Set #response_text='responseText '+dbo.fn_oa_error_message(#oa_object)
Else
SELECT #response_text=xml_result FROM #result_table
End
End
End
End
End
Exec sp_OADestroy #oa_object
Although you don't need it, the error handler is below. It helps with troubleshooting.
CREATE FUNCTION
dbo.n_oa_error_message(#oa_object int)
RETURNS varchar(max)
AS
BEGIN
Declare #source varchar(255)
Declare #description varchar(255)
exec sp_OAGetErrorInfo #oa_object, #source OUT, #description OUT
return 'Error: '+IsNull(#description,'no description')
END
I used the following query to solve this issue. The problem is probably not sp_OACreate or sp_OAMethod, but the way to return the #ResponseText. Inserting the data into a table variable instead of using "#ResponseText OUTPUT" is the key. Note that I changed the #Response to VARCHAR(MAX).
DECLARE #TABLEVAR TABLE (responseXml VARCHAR(MAX))
DECLARE #URL VARCHAR(200)
SELECT #URL = 'http://mysite/php-start/callxml.php'
DECLARE #Response NVARCHAR(MAX)
DECLARE #Xml XML
DECLARE #Obj INT
DECLARE #Result INT
EXEC #Result = sp_OACreate 'MSXML2.XMLHttp', #Obj OUT
EXEC #Result = sp_OAMethod #Obj, 'open', NULL, 'GET', #URL, false
EXEC #Result = sp_OAMethod #Obj, 'setRequestHeader', NULL, 'Content-Type', 'application/x-www-form-urlencoded'
EXEC #Result = sp_OAMethod #Obj, SEND, NULL, ''
INSERT INTO #TABLEVAR
EXEC #Result = sp_OAGetProperty #Obj, 'responseXML.xml'--, #Response OUT
EXEC sp_OADestroy #Obj
SELECT #Response = responseXml FROM #TABLEVAR
SELECT #Xml = CONVERT(XML, #Response, 2)
DECLARE #handle INT
EXEC sp_xml_preparedocument #handle OUTPUT, #Xml
SELECT *
FROM OPENXML(#handle, '/data/record', 2)
WITH [dbo].[tblDialogTechTemp]
EXEC sp_xml_removedocument #handle
My query suddenly returned null without changing anything. After changing 'MSXML2.XMLHttp' to 'MSXML2.ServerXMLHTTP', it started to work again. To know more about the difference between these two, see this article and Microsoft documentation.