Mule Database Connector Multiple Queries - database

I am running multiple select queries and I want them to run one after the other.
In this example, I select account numbers then use those numbers in the following query. Will the queries run consecutively one after the other i.e. next query only runs after the previous query has finished. Do I need to wrap them in a composite-source and wrap them in a transaction? What would that look like?
<flow name="PopulateAccount">
<db:select config-ref="dsConfig" doc:name="Get Account ID">
<db:paramterized-query><![CDATA[
SELECT ACC_NUM....
</db:select>
<custom-transformer class="com.vf.ListTransformer">
<set-session-variable variableName="messageID" value="#[payload]"
doc:name="Set Account IDs"/>
<!-- The next query depends on the Account IDS from
previous results in the session variable -->
<db:select config-ref="dsConfig" doc:name="Get Account Detail">
<db:paramterized-query><![CDATA[
SELECT ACC_NAME,....
</db:select>
<custom-transformer class="com.vf.AccountsTransformer">
<!-- More database operations --->
</flow>

Will the queries run consecutively one after the other
yes, as long as you do not take any measures to run them in parallel, like for example moving the database component in a separate flow and call it in a asynchronous way.
Do I need to wrap them in a composite-source
no, especially not if you use the result of the first query in the second query (like in your example)
and wrap them in a transaction
why? you are not inserting/updating anything in your example.
What would that look like?
just like in your example. the only thing i would change is the way how you store the result of your first query. although there is nothing wrong with using set-variable i prefer using a enricher to store result of a component in variable instead of changing the payload and setting the variable afterwards.
<flow name="testFlow">
<enricher target="#[flowVars.messageID]" doc:name="Message Enricher">
<db:select config-ref="MySQL_Configuration" doc:name="select ACC_NUM">
<db:parameterized-query><![CDATA[SELECT ACC_NUM ...]]></db:parameterized-query>
</db:select>
</enricher>
</flow>

Related

MuleSoft 4: ORA-01000: maximum open cursors exceeded

Getting 'ORA-01000: maximum open cursors exceeded' issue after processing few records from file inside try-catch scope. I've 3 select statements and 5 stored procedure calls(have insert statement inside store proc) in side try-catch scope.
Here is my pooling profile config:
<db:pooling-profile maxPoolSize="10" preparedStatementCacheSize="0" />
Using default configurations in Stored Procedure
<db:stored-procedure doc:name="insert into SPCHG_SERVICE_RENDERED" doc:id="d5b44d97-0f00-4377-a98d-b35a6b78df9e" config-ref="Database_Config" transactionalAction="ALWAYS_JOIN">
<reconnect count="3" />
<db:sql ><![CDATA[{call schema.ARRAY_INSERT_SERVICE(:serviceData,:error_num,:error_msg)}]]></db:sql>
<db:input-parameters ><![CDATA[#[{"serviceRenderedData" : payload}]]]></db:input-parameters>
<db:output-parameters >
<db:output-parameter key="error_num" type="INTEGER" />
<db:output-parameter key="error_msg" type="VARCHAR" />
</db:output-parameters>
</db:stored-procedure>
Runtime: 4.3
Any inputs to fix/avoid this issue.
Note: DBA team not going to increase cursor count, looking for solution from MuleSoft end.
Ensure that you are consuming the output of the stored procedure, even if not expecting a result set, by putting the operation in a separate flow inovked with a VM like the documentation suggest.
Alternatively put a foreach after the db operation.

Mule DB connector parameterized query

This is my first time experience with Mule & trying to define a flow with DB connector. I am defining a parameterized simple select query with 3 parameters but somehow not able to figure it out. Even I tried putting entire/partial query in Variable before passing the variable value to query but could not succeed. Any help is highly appreciated.
select * from employees where employee_country='UK' limit 1,10;
Here UK,1 & 10 are dynamic values which will come from UI as below, http://localhost:9090/employee?country=UK&start=1&limit=10
My try:
select * from employees where employee_country=#[message.inboundProperties.'http.query.params'.country] limit #[message.inboundProperties.'http.query.params'.start],#[message.inboundProperties.'http.query.params'.limit]
Thanx, -Swapnil
You are trying to create a dynamic query. For a parameterised query, you should separate the parameter values with placeholders. Parameterized won't replace expressions.
Here is the difference between the two options:
<db:select config-ref="DBConfig" doc:name="Database">
<db:parameterized-query><![CDATA[select * from employees where employee_country= :country limit 1,10;]]></db:parameterized-query>
<db:in-param name="country" type="VARCHAR" value="#[message.inboundProperties.'http.query.params'.country]" />
</db:select>
Dynamic:
<db:select config-ref="DBConfig" doc:name="Database">
<db:dynamic-query><![CDATA[select * from employees where employee_country=#[message.inboundProperties.'http.query.params'.country] limit #[message.inboundProperties.'http.query.params'.start],#[message.inboundProperties.'http.query.params'.limit]]]></db:dynamic-query>
</db:select>
Here is the documentation explaining the difference:
https://docs.mulesoft.com/mule-runtime/3.7/database-connector#query-types
Parameterized is the recommended approach as the disadvantage of using dynamic query statements is security as it leaves the statement open for SQL injection.
In real time you might need these values in the flow .To reuse them capture via flow variables.
When we have dynamic value that are coming in query params..capture it via variable name originalQueryParams
#[message.inboundProperties.'http.query.params']
Use groovy script to capture those variables and seperate them via set invocation properties ..
def inputRequest = message.getInvocationProperty('originalQueryParams');
message.setInvocationProperty('country',inputRequest.country);
return payload;
Now Country is you flow variable and pass it in the query .
select * from employees where employee_country = '#[flowVars.country]'
Hope this helps

To filter out the right records from a CSV file and hence to perform the required operation using Anypoint Studio

I am working on a flow that is supposed to take a CSV file from the system and insert the contained data into a database. Some of the records in the file have the wrong format required(Eg: wrong number of columns) and hence must me written into another file of text type for analysis.
I have created a flow that inserts all the good records into the database but it does not input the bad records into a file. I am currently a beginner and hence am not sure how to proceed with it.
The XML code:
<flow name="fileFlow">
<file:inbound-endpoint path="src/main/resources/Input" moveToPattern="#[message.inboundProperties.originalFilename].zip" moveToDirectory="src/main/resources/Output" responseTimeout="10000" metadata:id="b85f6b05-1679-4b60-8bbe-30e6d2c68df7" doc:name="File">
<file:filename-regex-filter pattern=".*csv" caseSensitive="true"/>
</file:inbound-endpoint>
<file:file-to-string-transformer doc:name="File to String"/>
<set-payload value="#[payload.replaceAll(",,", ", ,")]" doc:name="Set Payload"/>
<splitter expression="#[rows=StringUtils.split(message.payload,'\r\n');ArrayUtils.subarray(rows,1,rows.size())]" doc:name="Splitter"/>
<flow-ref name="fileFlow1" doc:name="fileFlow1"/>
<catch-exception-strategy doc:name="Insert the bad record into a file">
<byte-array-to-string-transformer doc:name="Byte Array to String"/>
<set-session-variable variableName="var" value="#[payload+'var']" doc:name="Session Variable"/>
<file:outbound-endpoint path="src/main/resources/Output" outputPattern="BadRecords.txt" responseTimeout="10000" doc:name="File"/>
<flow-ref name="fileFlow1" doc:name="fileFlow1"/>
</catch-exception-strategy>
</flow>
<flow name="fileFlow1">
<expression-transformer expression="#[StringUtils.split(message.payload,',')]" doc:name="Expression"/>
<db:insert config-ref="MySQL_Configuration" doc:name="Database">
<db:parameterized-query><![CDATA[insert into GoodRecords values(#[message.payload[0]], #[message.payload[1]], #[message.payload[2]], #[message.payload[3]], #[message.payload[4]], #[message.payload[5]], #[message.payload[6]], #[message.payload[7]], #[message.payload[8]], #[message.payload[9]], #[message.payload[10]], #[message.payload[11]], #[message.payload[12]], #[message.payload[13]], #[message.payload[14]], #[message.payload[15]], #[message.payload[16]], #[message.payload[17]], #[message.payload[18]], #[message.payload[19]], #[message.payload[20]])]]></db:parameterized-query>
</db:insert>
<logger message="#[payload] " level="INFO" doc:name="Logger"/>
</flow>
The flow structure:
Flow diagram
Personally I think the flow I've created is very inefficient and wrong.
How do I enter the bad records into the file(if the given flow is right)?
I wanted to use bulk mode for the given use case (as there are around 1000 odd records to work with) but am not sure how to proceed with that as well.
In your code you have used a private flow for inserting your record to database. So any exception occurred during insertion will not be catch by the parent flow exception strategy, you can have separate exception strategy for your private flow or you may use sub-flow.
There is another clean solution for that you can use batch processing to process these records and create a separate step to handle all failure records.

Solr deletions with custom full import

I'm trying to use the DataImportHandler to keep my index in sync with a SQL database (what I would think is a pretty vanilla thing to do). Since my database will be pretty large, I want to use incremental imports using this method http://wiki.apache.org/solr/DataImportHandlerDeltaQueryViaFullImport so the calls are of the form http://localhost:8983/solr/Items/dataimport?command=full-import&clean=false. This works perfectly well for adding items.
I have a separate DeletedItems table in my database which contains the primary keys of the items that have been deleted from the Items table, along with when they were deleted. As part of the DataImport call, I had hoped to be able to delete the relevant items from my index based on a query along the lines of
SELECT Id FROM DeletedItems WHERE WasDeletedOn > '${dataimporter.last_index_time}'
but I can't figure out how to do this. The link above alludes to it with the cryptic
In this case it means obviously that in case you also want to use deletedPkQuery then when running the delta-import command is still necessary.
but setting deletedPkQuery to the above SQL query doesn't seem to work. I then read that deletedPkQuery only works with delta-imports, so I am forced to make two requests to my solr server as part of the sync process? This doesn't seem right as the operations are parameterized by the dataimporter.last_index_time property, which changes. Both steps would need to be done in one "atomic" action, surely? Any ideas?
You must use the import handler special commands
https://wiki.apache.org/solr/DataImportHandler#Special_Commands
With these commands you can alter the boost or delete a document coming from the recordset of the full import query. Be aware that you must use the $skipDoc field to avoid that the document gets indexed again and that you must repeat the id in the $deleteDocById field.
You can use a union query
select
id,
text,
'false' as [$deleteDocById],
'false' as [$skipDoc]
from [rows to update or add]
Union Select
id,
'' as text,
id as [$deleteDocById],
true as [$skipDoc]
or a case when
select
id,
text,
CASE
when deleted = 1 then id
else 'false'
END as [$deleteDocById],
CASE
when deleted = 1 then 'true'
else 'false'
END as [$skipDoc]
Where updated > ${dih.last_index_time}
The deletedPkQuery is run as part of the regular call to delta-import, so you don't have to run anything twice (and when doing a full-import, there's no need to run deletedPkQuery, since the whole connection is cleared before importing anyway).
The deletedPkQuery should be configured on the same element as your main query. Be sure to match the field names exactly as well, and that the id produced by your deletedPkQuery matches the one provided by the main query.
There's a minimal example on solr.pl for importing and deleting fields using the same deleted_entries-table structure as you have here:
<entity
name="album"
query="SELECT * from albums"
deletedPkQuery="SELECT deleted_id as id FROM deletes WHERE deleted_at > '${dataimporter.last_index_time}'"
>
Also make sure that the format of the deleted_at-field is comparable against the value produced by last_index_time. The default is yyyy-MM-dd HH:mm:ss.
.. and lastly, remember that the last_index_time property isn't available before the second time the task is run, since there's no "previous index time" the first time an index is being populated (but the deletedPkQuery shouldn't run before that anyway).

Where is the data saved, from the Datatable-items, in ECM eRoom-database?

I am trying to retrieve data out of the ECM eRoom Database (which isn't documented, as I know of).
I have an eRoom with an custom "Database", some Fields.
When I query the objects table I find the "Database" row
select * from[dbo].[Objects] where internalId = 1234567
and the Rows for the entries
select top 10 * from[dbo].[Objects] where parentInternalId = 1234567
but I don't find any field with the values of the entries, only an Column with NonSearchableProperties., that is only full with Hex Data.
My Question(s),
how could i retrieve the values?
Is it possible to retrieve them with mere SQL?
What is the simplest way?
This is not the silver bullet, but it is okay for my usecase
After long goolging and alot of test scripts, i found some answers, but probably due to the fact the the system is soon end-of-life and that the documentation is not easy to read, here are my finding.
Is it possible to retrieve them with mere SQL?
As far as I could find out, no! (please correct me If I'm wrong)
how could i retrieve the values?
With the eRoom API(on the Server there are some Sample programms to query the data/objects <installation-path>\eRoom Server\Toolkit\Samples, with c++, vb, vbscript, ... all a bit to much overhead), or with the eRoom XML Query Language(exql) over soap calls.
What is the simplest way?
After alot of tests, searching in forums and many tests with soap ui. I found out that queries with exql seem to be the simplest way to retrieve Data, if you understand the structure.
Here some ressource that were helpful:
(very) Basic info of exql from the manufacturer https://eroom.asce.org/eRoomHelp/en/XML_Help/Introduction.htm
(disclaimer: I didn't find it helpful, but it shows at least some basics)
short 9 page Developer guide https://developer-content.emc.com/developer/downloads/eRoomXMLCapabilitiesUseCaseProgramDashboard.pdf (the last example on page 8, helped me understand how to setup the query, with alot of fantasy)
But for this to work, don't forget, to activate Allow XML queries and commands from external applications in the Site Settings
TIP 1:
you always can go go deeper you just need to know the right xml-element under. <Database>, <Cells> and <DBCell> can help you go deeper
TIP 2:
don't query to much data since this query likely run into timeouts
Update 1:
Just to save time for anyone who is looking, this "query" returns all rows (properties) for a Database(s) created in an eRoom Root.
(don't forget to set facility and room in the Url ex. http://server/eroomxml/facilities/TEST/Rooms/TestRoom, although it could be set in the query)
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:er="http://www.eroom.com/eRoomXML/2003/700">
<soapenv:Header/>
<soapenv:Body>
<er:ExecuteXMLCommand>
<er:eRoomXML>
<er:command er:select="HomePage/Items">
<er:getproperties>
<er:Item>
<Database>
<Rows>
<Item>
<Cells>
<DBCell>
<Content>
</Content>
</DBCell>
</Cells>
</Item>
</Rows>
</Database>
</er:Item>
</er:getproperties>
</er:command>
</er:eRoomXML>
</er:ExecuteXMLCommand>
</soapenv:Body>
</soapenv:Envelope>

Resources