I'm trying to execute a stored procedure to SELECT data from a SQL Server 2008 database using Mule 3.3.0.
In the Mule docs there is info about doing this with Oracle. I'm not sure if this is possible with SQL Server.
This is my Mule endpoint config
<jdbc:outbound-endpoint exchange-pattern="request-response" queryTimeout="-1" connector-ref="sqlServerConnector" queryKey="selectCoupons" doc:name="Database">
<jdbc:query key="selectCoupons" value="call sp_get_coupons()"/>
</jdbc:outbound-endpoint>
This is the output
Root Exception stack trace:
java.sql.SQLException: The executeUpdate method must not return a result set.
at net.sourceforge.jtds.jdbc.JtdsStatement.processResults(JtdsStatement.java:603)
at net.sourceforge.jtds.jdbc.JtdsStatement.executeSQL(JtdsStatement.java:546)
at net.sourceforge.jtds.jdbc.JtdsPreparedStatement.executeUpdate(JtdsPreparedStatement.java:506)
+ 3 more (set debug level logging or '-Dmule.verbose.exceptions=true' for everything)
I'm using the jTDS driver. Testing the stored procedure with a JDBC client I get the expected resultSet.
Any suggestions?
<db:stored-procedure config-ref="Oracle_Configuration1" doc:name="Database">
<db:parameterized-query><![CDATA[{call apps.create_sales_Order(:p_header_rec_oper,:P_order_number,:P_ordered_date,:P_line_id,:p_flow_Status_code,:P_return_status)}]]></db:parameterized-query>
<db:in-param name="p_header_rec_oper" value="CREATE"/>
<db:out-param name="P_order_number" type="INTEGER"/>
<db:out-param name="P_ordered_date" type="DATE"/>
<db:out-param name="P_line_id" type="VARCHAR"/>
<db:out-param name="p_flow_Status_code" type="VARCHAR"/>
<db:out-param name="P_return_status" type="VARCHAR"/>
</db:stored-procedure>
{
• P_return_status: "S",
• P_line_id: "684229",
• p_flow_Status_code: "ENTERED",
• P_ordered_date: "2015-05-22",
• P_order_number: 69393
}
Related
I'm trying to import some data from different SqlServer databases using ExecuteSQL in NiFi, but it's returning me an error. I've already imported a lot of other tables from MySQL databases without any problem and I'm trying to use the same workflow structure for the SqlServer dbs.
The structure is as follows:
There's a file .txt with the list of tables to be imported
This file is fetched, splitted and uptaded; so there's a FlowFile for each table of each db that has to be imported,
These FlowFiles are passed into ExecuteSQL which executes their contents
For example:
file.txt
table1
table2
table3
is being updated into 3 different FlowFiles:
FlowFile1
SELECT * FROM table1
FlowFile2
SELECT * FROM table2
FlowFile3
SELECT * FROM table3
which are passed to ExecuteSQL.
Here follows the configuration of ExecuteSQL (identical for SqlServer tables and MySQL ones)
ExecuteSQL
As the only difference with the import from MySQL db is in the connectors, this is how a generic MySQL connector has been configured:
SETTINGSPROPERTIES
Database Connection URL jdbc:mysql://00.00.00.00/DataBase?zeroDateTimeBehavior=convertToNull&autoReconnect=true
Database Driver Class Name com.mysql.jdbc.Driver
Database Driver Location(s) file:///path/mysql-connector-java-5.1.47-bin.jar
Database User user
PasswordSensitive value set
Max Wait Time 500 millis
Max Total Connections 8
Validation query No value set
And this is how a SqlServer connector has been configured:
SETTINGSPROPERTIES
Database Connection URL jdbc:jtds:sqlserver://00.00.00.00/DataBase;useNTLMv2=true;integratedSecurity=true;
Database Driver Class Name net.sourceforge.jtds.jdbc.Driver
Database Driver Location(s) /path/connectors/jtds-1.3.1.jar
Database User user
PasswordSensitive value set
Max Wait Time -1
Max Total Connections 8
Validation query No value set
It has to be noticed that one (only one!) SqlServer connector works and the ExecuteSQL processor imports the data without any problem. The even stranger thing is that the database that is being connected via this connector is located in the same place as other two (the connection URL and user/psw are identical), but only the first one is working.
Notice that I've tried appending ?zeroDateTimeBehavior=convertToNull&autoReconnect=true also to the SqlServer connections, supposing it was a problem of date type, but it didn't give any positive change.
Here is the error that is being returned:
12:02:46 CEST ERROR f1553b83-a173-1c0f-93cb-1c32f0f46d1d
00.00.00.00:0000 ExecuteSQL[id=****] ExecuteSQL[id=****] failed to process session due to null; Processor Administratively Yielded for 1 sec: java.lang.AbstractMethodError
Error retrieved from logs:
ERROR [Timer-Driven Process Thread-49] o.a.nifi.processors.standard.ExecuteSQL ExecuteSQL[id=****] ExecuteSQL[id=****] failed to process session due to java.lang.AbstractMethodError; Processor Administratively Yielded for 1 sec: java.lang.AbstractMethodError
java.lang.AbstractMethodError: null
at net.sourceforge.jtds.jdbc.JtdsConnection.isValid(JtdsConnection.java:2833)
at org.apache.commons.dbcp2.DelegatingConnection.isValid(DelegatingConnection.java:874)
at org.apache.commons.dbcp2.PoolableConnection.validate(PoolableConnection.java:270)
at org.apache.commons.dbcp2.PoolableConnectionFactory.validateConnection(PoolableConnectionFactory.java:389)
at org.apache.commons.dbcp2.BasicDataSource.validateConnectionFactory(BasicDataSource.java:2398)
at org.apache.commons.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:2381)
at org.apache.commons.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:2110)
at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1563)
at org.apache.nifi.dbcp.DBCPConnectionPool.getConnection(DBCPConnectionPool.java:305)
at org.apache.nifi.dbcp.DBCPService.getConnection(DBCPService.java:49)
at sun.reflect.GeneratedMethodAccessor1696.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:84)
at com.sun.proxy.$Proxy449.getConnection(Unknown Source)
at org.apache.nifi.processors.standard.AbstractExecuteSQL.onTrigger(AbstractExecuteSQL.java:195)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165)
at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
What I am using
Ubuntu 16.04
Python 3.6
FreeTDS, TDS Version 7.3
SQLAlchemy 1.2.5
Windows server 2012
SQL Server 2008 Enterprise
My purpose
I write code in Python on Ubuntu machine to insert and execute stored procedure on MS SQL Server 2008. I create an order for customer. An order may have many main ingredients, toppings. When finish order, I run a stored procedure to process data to user_order and employee_order.
The stored procedure
In stored procedure, when select data from source tables and process data, if any error is happened, transaction is rolled back.
My code snippet
def process():
engine = get_engine() # my method get engine by connection string
session_maker = sessionmaker(bind=engine.execution_options(isolation_level='SERIALIZABLE'))
session = session_maker()
ref = 'REF0000001'
try:
# Create order
order = Order(id=1, ref=ref)
# Add main ingredients
main1 = Main(order=1, name='coffee')
main2 = Main(order=1, name='milk')
# Topup
topup1 = TopUp(order=1, name='cookies')
topup2 = TopUp(order=1, name='chocolate')
session.add(order)
session.flush()
session.add_all([main1, main2])
session.flush()
session.add_all([topup1, topup2])
session.flush()
session.commit()
except:
session.rollback()
reraise
finally:
session.close()
del session
time.sleep(1)
session = session_maker()
session.execute('EXEC finish_order %a' % ref)
session.commit()
session.close()
del session
And result is
There is no error, but there is no data in user_order and employee_order even though stored procedure finish_order is run.
But, if I run the stored procedure again as a simple query in terminal or SQL Studio Management, the data is imported to destination tables.
Doubts
Is there any chance that data has not been finished inserting into origin tables yet when stored procedure is called?
Please help me with this case.
Thank you!
I am trying to get distributed queries to run on SQL Server 2012 with a linked server to Teradata.
Connection works fine and query returns quickly if I pass the where clause into the remote SQL using openquery e.g.
select *
from openquery(td, 'select * from lib.purchases where z_PO = ''123456''')
However the below does not run as expected: SQL Server loads the entire table and performs a local filter:
select *
from openquery(td, 'select * from lib.purchases') where z_PO = '123456'
The source table has 100M records.
Obviously index play no role here as query runs just fine on TD side.
What I have tried:
sp_configure 'Ad Hoc Distributed Queries', 1
set Collation Compatible" = True on the linked server properties
Instead of 2, set Collation Name = Latin1_BIN to match (closely?) TD character set (ASCII).
Not sure collation is the issue as I get same result when filtering on numeric fields.
Somehow the so-called query optimizer in SQL Server does not push simple filtering down to the remote server.
Is this the ODBC driver's fault (using 16.10) - a setting, a bug? SQL Server 2012 (v11.0.6248.0) setting I am missing (or path req'd)?
Below is the OLEDB for ODBC properties that I capture in SQL Profiler:
<ProviderInformation>
<Provider>MSDASQL</Provider>
<LinkedServer>td</LinkedServer>
<ProviderCapabilitiesAndSettings>
<Ansi92EntrySupport>0</Ansi92EntrySupport>
<ODBCCoreSupport>1</ODBCCoreSupport>
<ODBCMinimumSupport>1</ODBCMinimumSupport>
<SimpleGrammarSupport>0</SimpleGrammarSupport>
<AnsiLikeSupport>0</AnsiLikeSupport>
<SQLLikeSupport>1</SQLLikeSupport>
<DateLiteralsSupport>0</DateLiteralsSupport>
<GroupBySupport>0</GroupBySupport>
<InnerJoinSupport>0</InnerJoinSupport>
<SubqueriesSupport>0</SubqueriesSupport>
<SimpleUpdatesSupport>0</SimpleUpdatesSupport>
<HistogramsSupport>0</HistogramsSupport>
<ColumnLevelCollationSupport>0</ColumnLevelCollationSupport>
<ConnectionSharingSupport>0</ConnectionSharingSupport>
<MultipleActiveRowsetsSupport>0</MultipleActiveRowsetsSupport>
<MultipleResultsSupport>1</MultipleResultsSupport>
<AllowLimitingRowsReturned>1</AllowLimitingRowsReturned>
<NullConcatenationYieldsNull>0</NullConcatenationYieldsNull>
<StructuredStorageAccessToLargeObjects>1</StructuredStorageAccessToLargeObjects>
<MultipleConcurrentLargeObjectSupport>0</MultipleConcurrentLargeObjectSupport>
<DynamicParametersSupport>1</DynamicParametersSupport>
<NestedQueriesSupport>1</NestedQueriesSupport>
<IndicesAvailableAsAccessPath>0</IndicesAvailableAsAccessPath>
<AllowDataAccessByReference>1</AllowDataAccessByReference>
<RowsetChangesAreVisible>0</RowsetChangesAreVisible>
<RowsetSupportsAppendOnly>0</RowsetSupportsAppendOnly>
<UseLevelZeroOledbInterfacesOnly>0</UseLevelZeroOledbInterfacesOnly>
<RowsetUpdatability>1</RowsetUpdatability>
<AsynchronousRowsetProcessingSupport>0</AsynchronousRowsetProcessingSupport>
<DataSourceUnicodeLocaleId>0</DataSourceUnicodeLocaleId>
<DataSourceUnicodeComparisonStyle>0</DataSourceUnicodeComparisonStyle>
<DataSourceCollationComparisonFlags>0</DataSourceCollationComparisonFlags>
<DataSourceCharacterset></DataSourceCharacterset>
<DataSourceSortOrder></DataSourceSortOrder>
<DataSourceNullCollationOrder>4</DataSourceNullCollationOrder>
<CurrentDbCollationSameAsDefaultRemoteDbCollation>0</CurrentDbCollationSameAsDefaultRemoteDbCollation>
<UnicodeLiteralSupport>0</UnicodeLiteralSupport>
<UnicodeLiteralPrefix></UnicodeLiteralPrefix>
<UnicodeLiteralSuffix></UnicodeLiteralSuffix>
<DateLiteralPrefix></DateLiteralPrefix>
<DateLiteralSuffix></DateLiteralSuffix>
<ObjectNameConstructionFlags>54</ObjectNameConstructionFlags>
<SchemaSeparator>.</SchemaSeparator>
<CatalogSeparator>.</CatalogSeparator>
<QuoteSeparator>"</QuoteSeparator>
<BitRemoting>0</BitRemoting>
<UnicodeLiterals>0</UnicodeLiterals>
<ProviderOledbVersion>131072</ProviderOledbVersion>
<HalloweenProtectionNeeded>1</HalloweenProtectionNeeded>
<RowsetUsableAcrossThreads>0</RowsetUsableAcrossThreads>
<ObjectNameIsSinglePart>0</ObjectNameIsSinglePart>
<Cardinality>-1</Cardinality>
<BookmarkSupport>0</BookmarkSupport>
<BookmarksReusable>0</BookmarksReusable>
<TableFlags>0</TableFlags>
</ProviderCapabilitiesAndSettings>
and here the details of the column in the filter:
DBCOLUMNINFO>
<pwszName>z_PO</pwszName>
<pTypeInfo>0x0000000000000000</pTypeInfo>
<iOrdinal>53</iOrdinal>
<dwFlags>120</dwFlags>
<ulColumnSize>10</ulColumnSize>
<wType>129</wType>
<bPrecision>255</bPrecision>
<bScale>255</bScale>
<DBID>
<eKind>DBKIND_NAME</eKind>
<uName.pwszName>z_PO</uName.pwszName>
</DBID>
</DBCOLUMNINFO>
As a FYI, the context is wrapping the openquery, joined with local data, into a SQL Server view, which is the only thing users would see - from there, they can apply any filter (WHERE) within PowerQuery (XL) or PowerBI. A way to circumvent the lack of DirectQuery support through ODBC.
My database component has the following configuration
<db:insert config-ref="Oracle_Configuration" bulkMode="true" doc:name="Database">
<db:dynamic-query><![CDATA[#[flowVars.dbquery]]]></db:dynamic-query>
</db:insert>
I have declared the "dbquery" variable as follows
<set-variable variableName="dbquery" value="INSERT INTO WBUSER.EMP VALUES('#[payload.FullName]','#[payload.SerialNumber]')" doc:name="Variable"/>
On running the application the values inserted into the DB are "#[payload.FullName] and #[payload.SerialNumber].
But when my database component has the following configuration actual values of FullName and SerialNumber are getting inserted into the database.
<db:insert config-ref="Oracle_Configuration" bulkMode="true" doc:name="Database">
<db:dynamic-query><![CDATA[INSERT INTO WBUSER.EMP VALUES('#[payload.FullName]','#[payload.SerialNumber]')]]></db:dynamic-query>
</db:insert>
Here FullName and SerialNumber are not variables. They are column names of the list in the payload as [{FullName=yo, SerialNumber=129329}, {FullName=he, SerialNumber=129329}].
Can someone tell me the difference here. And is there a way i can achieve database insertion using just the variable as in the earlier case?
It caused by different approach to insert data. It works correctly for the configuration inside db-insert, because the payload is in form of List and Bulk Mode option selected.
To make it work for the first configuration (declare SQL query in a variable) then you have to do the following steps:
Iterate each payload value by utilizing: collection-splitter.
Deselect Bulk Mode from database connector.
The configuration should be:
<collection-splitter doc:name="Collection Splitter"/>
<set-variable variableName="dbquery" value="INSERT INTO WBUSER.EMP VALUES('#[payload.FullName]','#[payload.SerialNumber]')" doc:name="Variable"/>
<db:insert config-ref="MySQL_Configuration" doc:name="Database">
<db:dynamic-query><![CDATA[#[flowVars.dbquery]]]></db:dynamic-query>
</db:insert>
We have a stored procedure in place that creates a new SSIS execution and starts it:
Declare #execution_id bigint
EXEC [SSISDB].[catalog].[create_execution]
#package_name=N'00 XXXX.dtsx',
#execution_id=#execution_id OUTPUT,
#folder_name=N'XX',
#project_name=N'XXX';
EXEC [SSISDB].[catalog].[start_execution] #execution_id;
When we call this stored procedure logged on using SQL Server Management Studio, this works perfectly. However, when we execute this stored procedure from within BizTalk (as a BTS service account user) we receive this error:
System.Data.SqlClient.SqlException (0x80131904): Incorrect syntax near '-'.
Incorrect syntax near '-'.
Incorrect syntax near '-'.
Server stack trace:
at System.Runtime.AsyncResult.End[TAsyncResult](IAsyncResult result)
at System.ServiceModel.Channels.ServiceChannel.SendAsyncResult.End(SendAsyncResult result)
at System.ServiceModel.Channels.ServiceChannel.EndCall(String action, Object[] outs, IAsyncResult result)
at System.ServiceModel.Channels.ServiceChannel.EndRequest(IAsyncResult result)
Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at System.ServiceModel.Channels.IRequestChannel.EndRequest(IAsyncResult result)
at Microsoft.BizTalk.Adapter.Wcf.Runtime.WcfClient`2.RequestCallback(IAsyncResult result)
The schema we use from BTS to MS SQL:
<?xml version="1.0" encoding="utf-16"?>
<schema xmlns:b="http://schemas.microsoft.com/BizTalk/2003" xmlns:ns0="http://schemas.microsoft.com/Sql/2008/05/TypedProcedures/dbo" elementFormDefault="qualified" targetNamespace="http://schemas.microsoft.com/Sql/2008/05/TypedProcedures/dbo" version="1.0" xmlns="http://www.w3.org/2001/XMLSchema">
<annotation>
<appinfo>
<fileNameHint xmlns="http://schemas.microsoft.com/servicemodel/adapters/metadata/xsd">TypedProcedure.dbo</fileNameHint>
</appinfo>
</annotation>
<element name="StartBifImport">
<annotation>
<documentation>
<doc:action xmlns:doc="http://schemas.microsoft.com/servicemodel/adapters/metadata/documentation">TypedProcedure/dbo/StartBifImport</doc:action>
</documentation>
</annotation>
<complexType>
<sequence />
</complexType>
</element>
</schema>
The BTS service account has the db_owner role on the database where the stored procedure resides, and we have given it explicitly all 'grant' permissions on the SSIS folder and SSIS package.
We can see in the executions table that no execution is created, so something seems to go wrong on that call. When connecting with SQL Profiler we see the same error returning, but without much further context as to what is wrong; leaving is without much trace.
Any pointers/ideas on how the debug or resolve this issue?
I suggest not to use a "Strongly-Typed Procedure" schema, but a "Procedure" schema. The "Strongly-Typed Procedure" from the WCF-SQL adapter has a documented issue with generating the schema for stored procedures with temp tables in it.
The SQL adapter does not support generating metadata for strongly-typed stored procedures that contain temporary tables in their definition. Instead, you should generate metadata for the same procedure from under the Procedures node while using the Add Adapter Service Reference Plug-in or Consume Adapter Service Add-in.
More information can be found here: https://msdn.microsoft.com/en-us/library/dd788435(BTS.10).aspx#BKMK_SQLMetadataStronglyTyped
Create a simple one-way send port that has the same filter (and map, if applicable) as one you are busy debugging, that simply writes to a file. Use the output of the file to confirm the message sent to the WCF-SQL adaptor has the structure it is expecting.
Judging by the schema, it should produce a file with the following content:
<ns0:StartBifImport xmlns:ns0="http://schemas.microsoft.com/Sql/2008/05/TypedProcedures/dbo">
</ns0:StartBifImport>