BaseX XQuery is slow (again!) - basex

Years ago, we had problems with an XQuery. See here:
BaseX: Slow XQuery
We were able to solve the issue with the help of the community (string literal required for index to operate properly). But now (suddently) the very same query is slow again!
The query xquery /*:ARTICLE/*:ART[*:PRDNO='1104423'] takes ~3-30s (XML file is ~500MB), same with xquery count(/*:ARTICLE/*:ART[*:PRDNO='1104423']).
Result from info database is unsuspicious, there is a text index.
I set "QUERYPLAN", "XMLPLAN" and "COMPPLAN" to true.
This gives the following for the "count" query:
Query plan:
<QueryPlan compiled="true">
<FnCount name="count(items)">
<IterPath>
<Root/>
<IterStep axis="child" test="*:ARTICLE"/>
<IterStep axis="child" test="*:ART">
<CmpG op="=">
<CachedPath>
<IterStep axis="child" test="*:PRDNO"/>
</CachedPath>
<Str value="1104423" type="xs:string"/>
</CmpG>
</IterStep>
</IterPath>
</FnCount>
</QueryPlan>
Query:
count(/*:ARTICLE/*:ART[*:PRDNO='1104423'])
Parsing: 0.21 ms
Compiling: 0.27 ms
Evaluating: 28453.24 ms
Printing: 0.15 ms
Total Time: 28453.87 ms
Hit(s): 1 Item
Updated: 0 Items
Printed: 1 b
Read Locking: hospindex
Write Locking: (none)
Query executed in 28453.87 ms.
Similarly, this is what I get for the query without "count":
Query plan:
<QueryPlan compiled="true">
<IterPath>
<Root/>
<IterStep axis="child" test="*:ARTICLE"/>
<IterStep axis="child" test="*:ART">
<CmpG op="=">
<CachedPath>
<IterStep axis="child" test="*:PRDNO"/>
</CachedPath>
<Str value="1104423" type="xs:string"/>
</CmpG>
</IterStep>
</IterPath>
</QueryPlan>
Query:
/*:ARTICLE/*:ART[*:PRDNO='1104423']
Parsing: 0.02 ms
Compiling: 0.01 ms
Evaluating: 0.49 ms
Printing: 20093.88 ms
Total Time: 20094.41 ms
Hit(s): 1 Item
Updated: 0 Items
Printed: 1583 b
Read Locking: hospindex
Write Locking: (none)
Query executed in 20094.41 ms.
Interestingly, the "print" time is very high, but the result which is printed only has a few rows.
We have recreated and optimized the database.
The query plan looks as if the index was used, right?
Any advice? Thank you very much!

Related

More like this query parser not working SOLR 8.9 | Error completing MLT request. Could not fetch document with id [XXXXX]

I am trying to use a simple query in the SOLR cloud with 3 shards in my collection with basic authentication enabled. Everything is working fine in the cloud but when I am using more like this ( MLT ) query parser there is an issue.
The id used XXXXX is a valid id and present in the index
SOLR version : 8.9.0
Query:
http://localhost:8983/solr/my_collection/select?q={!mlt+qf%3Dtext}XXXXX&start=0&rows=7&fl=id,text
The query fails even on valid document id with the below error in the SOLR logs:
Error completing MLT request. Could not fetch document with id [XXXXX]
at org.apache.solr.search.mlt.CloudMLTQParser.parse(CloudMLTQParser.java:68)
at org.apache.solr.search.QParser.getQuery(QParser.java:174)
at org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:162)
at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:334)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:216)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2637)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:794)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:567)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:427)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:357)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191)
at org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:177)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:383)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:882)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1036)
at java.lang.Thread.run(Thread.java:748)

xQuery value returns null for valid XML tags

I have multiple XMLs in the following format:
<Report xmlns:rd="http://schemas.microsoft.com/SQLServer/reporting/reportdesigner" xmlns="http://schemas.microsoft.com/sqlserver/reporting/2008/01/reportdefinition">
<DataSources>
<DataSource Name="DataSource1">
<DataSourceReference>/DataSources/Infinite</DataSourceReference>
<rd:DataSourceID>RandomID</rd:DataSourceID>
</DataSource>
</DataSources>
<DataSets>
<DataSet Name="DataSet1">
<Query>
<DataSourceName>DataSource1</DataSourceName>
<CommandText>HCC.RPTUnderwriter</CommandText>
<rd:UseGenericDesigner>true</rd:UseGenericDesigner>
</Query>
</DataSet>
<DataSet Name="DataSet2">
<Query>
<DataSourceName>DataSource1</DataSourceName>
<CommandText>HCC.RptUnderwriterList</CommandText>
<rd:UseGenericDesigner>true</rd:UseGenericDesigner>
</Query>
</DataSet>
<rd:ReportID>randomReportIDx</rd:ReportID>
<rd:ReportServerUrl>http://sampleRS/ReportServer</rd:ReportServerUrl>
</Report>
Below is the SQL code I am using to convert the 'Cat.Content' into an XML and then extract the 'Datasource' name and 'CommandText'. The issue I am having is that for some XMLs, it returns the correct value for the 'Datasource' and 'CommandText' fields, however, there are some XMLs that return NULL even though there is a value in the 'Datasource' and 'CommandText' fields. I am trying to figure out why it works on some XMLs and not on others. The XMLs are mostly in the same format and order and there are a few that have an additional namespace.
WITH XMLNAMESPACES
( DEFAULT
'http://schemas.microsoft.com/sqlserver/reporting/2010/01/reportdefinition',
'http://schemas.microsoft.com/SQLServer/reporting/reportdesigner' AS RF
)
SELECT
CATDATA.Name AS ReportName
,CATDATA.Path AS ReportPathLocation,
catdata.reportXML as ReportXml
,xmlcolumn.value('(#Name)[1]', 'VARCHAR(250)') AS DataSetName
,xmlcolumn.value('(Query/DataSourceName)[1]','VARCHAR(250)') AS DataSourceName
,xmlcolumn.value('(Query/CommandText)[1]','VARCHAR(2500)') AS CommandText
from (
select
Cat.Name,
cat.Path,
convert (xml, convert (varbinary(max),cat.content)) as reportXML,
Cat.Content
from ReportServer.dbo.Catalog Cat
where cat.Content is not null
and cat.type = 2
) catdata
outer APPLY reportXML.nodes('/Report/DataSets/DataSet') xmltable ( xmlcolumn )
ORDER BY CATDATA.Name;
EDIT:
This is the XML that DOES NOT WORK with the current SQL query:
<Report
xmlns:rd="http://schemas.microsoft.com/SQLServer/reporting/reportdesigner" xmlns="http://schemas.microsoft.com/sqlserver/reporting/2008/01/reportdefinition">
<DataSources>
<DataSource Name="DataSource1">
<DataSourceReference>/DataSources/Infinite</DataSourceReference>
<rd:DataSourceID>RandomID</rd:DataSourceID>
</DataSource>
</DataSources>
<DataSets>
<DataSet Name="DataSet1">
<Query>
<DataSourceName>DataSource1</DataSourceName>
<CommandText>HCC.RPTUnderwriter</CommandText>
<rd:UseGenericDesigner>true</rd:UseGenericDesigner>
</Query>
</DataSet>
<DataSet Name="DataSet2">
<Query>
<DataSourceName>DataSource1</DataSourceName>
<CommandText>HCC.RptUnderwriterList</CommandText>
<rd:UseGenericDesigner>true</rd:UseGenericDesigner>
</Query>
</DataSet>
<DataSets>
<rd:ReportID>randomReportIDx</rd:ReportID>
<rd:ReportServerUrl>http://sampleRS/ReportServer</rd:ReportServerUrl>
</Report>
This is the XML that WORKS with the current query:
<Report xmlns:rd="http://schemas.microsoft.com/SQLServer/reporting/reportdesigner" xmlns:cl="http://schemas.microsoft.com/sqlserver/reporting/2010/01/componentdefinition" xmlns="http://schemas.microsoft.com/sqlserver/reporting/2010/01/reportdefinition">
<AutoRefresh>0</AutoRefresh>
<DataSources>
<DataSource Name="Infinite">
<DataSourceReference>/DataSources/Infinite</DataSourceReference>
<rd:DataSourceID>DSRAndomID</rd:DataSourceID>
</DataSource>
</DataSources>
<DataSets>
<DataSet Name="DataSet1">
<Query>
<DataSourceName>Infinite</DataSourceName>
<CommandText>HCC.RPTBrokerState</CommandText>
<rd:UseGenericDesigner>true</rd:UseGenericDesigner>
</Query>
</DataSet>
<DataSet Name="DataSet2">
<Query>
<DataSourceName>Infinite</DataSourceName>
<CommandText>HCC.RptStateList</CommandText>
<rd:UseGenericDesigner>true</rd:UseGenericDesigner>
</Query>
</DataSet>
</DataSets>
</Report>
however, there are some XMLs that return NULL even though there is a
value in the 'Datasource' and 'CommandText' fields. I am trying to
figure out why it works on some XMLs and not on others. The XMLs are
mostly in the same format and order and there are a few that have an
additional namespace.
Please try the following solution for the provided XML. You would need to add "not working" XML sample(s) to your question.
All elements in the XML are bound to a different default namespace in different rows. That's why I am using a namespace wildcard as #JeroenMostert already suggested.
Both XQuery methods .nodes() and .value() are using a namespace wildcard in the XPath expressions.
And the WITH XMLNAMESPACES clause is commented out.
SQL
-- DDL and sample data population, start
DECLARE #tbl TABLE (ID INT IDENTITY PRIMARY KEY, xmldata XML);
INSERT INTO #tbl (xmldata) VALUES
(N'<Report xmlns:rd="http://schemas.microsoft.com/SQLServer/reporting/reportdesigner"
xmlns="http://schemas.microsoft.com/sqlserver/reporting/2008/01/reportdefinition">
<DataSources>
<DataSource Name="DataSource1">
<DataSourceReference>/DataSources/Infinite</DataSourceReference>
<rd:DataSourceID>RandomID</rd:DataSourceID>
</DataSource>
</DataSources>
<DataSets>
<DataSet Name="DataSet1">
<Query>
<DataSourceName>DataSource1</DataSourceName>
<CommandText>HCC.RPTUnderwriter</CommandText>
<rd:UseGenericDesigner>true</rd:UseGenericDesigner>
</Query>
</DataSet>
<DataSet Name="DataSet2">
<Query>
<DataSourceName>DataSource1</DataSourceName>
<CommandText>HCC.RptUnderwriterList</CommandText>
<rd:UseGenericDesigner>true</rd:UseGenericDesigner>
</Query>
</DataSet>
</DataSets>
<rd:ReportID>randomReportIDx</rd:ReportID>
<rd:ReportServerUrl>http://sampleRS/ReportServer</rd:ReportServerUrl>
</Report>')
, (N'<Report xmlns:rd="http://schemas.microsoft.com/SQLServer/reporting/reportdesigner"
xmlns:cl="http://schemas.microsoft.com/sqlserver/reporting/2010/01/componentdefinition"
xmlns="http://schemas.microsoft.com/sqlserver/reporting/2010/01/reportdefinition">
<AutoRefresh>0</AutoRefresh>
<DataSources>
<DataSource Name="Infinite">
<DataSourceReference>/DataSources/Infinite</DataSourceReference>
<rd:DataSourceID>DSRAndomID</rd:DataSourceID>
</DataSource>
</DataSources>
<DataSets>
<DataSet Name="DataSet1">
<Query>
<DataSourceName>Infinite</DataSourceName>
<CommandText>HCC.RPTBrokerState</CommandText>
<rd:UseGenericDesigner>true</rd:UseGenericDesigner>
</Query>
</DataSet>
<DataSet Name="DataSet2">
<Query>
<DataSourceName>Infinite</DataSourceName>
<CommandText>HCC.RptStateList</CommandText>
<rd:UseGenericDesigner>true</rd:UseGenericDesigner>
</Query>
</DataSet>
</DataSets>
</Report>');
-- DDL and sample data population, end
--WITH XMLNAMESPACES (DEFAULT 'http://schemas.microsoft.com/sqlserver/reporting/2008/01/reportdefinition')
SELECT ID
, c.value('#Name', 'VARCHAR(50)') AS [Name]
, c.value('(*:Query/*:DataSourceName/text())[1]', 'VARCHAR(50)') AS DataSourceName
, c.value('(*:Query/*:CommandText/text())[1]', 'VARCHAR(MAX)') AS CommandText
FROM #tbl
CROSS APPLY xmldata.nodes('/*:Report/*:DataSets/*:DataSet') AS t(c);
Output
+----+----------+----------------+------------------------+
| ID | Name | DataSourceName | CommandText |
+----+----------+----------------+------------------------+
| 1 | DataSet1 | DataSource1 | HCC.RPTUnderwriter |
| 1 | DataSet2 | DataSource1 | HCC.RptUnderwriterList |
| 2 | DataSet1 | Infinite | HCC.RPTBrokerState |
| 2 | DataSet2 | Infinite | HCC.RptStateList |
+----+----------+----------------+------------------------+

How to handle apostrophe ( ' ) in XMl and SQL Server

I have an XML file which has details of employees and the problem I am try to figure it out is these XML values have apostrophe ( ' ) in random places all over the XML content, so it's getting hard to insert them into SQL Server tables.
I will be sending this entire XML content from MVC C# to a SQL Server stored procedure which will insert the data into various tables, but whenever there is an apostrophe ( ' ) in XML content, the error occurs. So these apostrophes should be either handled or replaced or removed. How can I do this?
This is some sample XML content:
<xml>
<Channel>
<Program id="1" category="A">
<name>Pra'Matino</name>
<Bin>
<Date>1/1/2020</Date>
<Date>1/1/2020</Date>
</Bin>
<Player>
<Pla>S'Rajesh</Pla>
<Pla>Su'man</Pla>
</Player>
<Television>
<HostDeails>2/9/2020</HostDeails>
<HostDeails>MALE</HostDeails>
<HostDeails>Colour</HostDeails>
</Television>
<addresses>
<address>
<address1>No 10</address1>
<city>Chennai</city>
<country>IN's</country>
<ProductName>Lavender's</ProductName>
</address>
<address>
<address1>N0 72</address1>
<city>Sanagoor Road</city>
<postalCode>641006</postalCode>
</address>
<address>
<address1>Old No 10/ New No 3</address1>
<city>Madurai</city>
<country>IN</country>
<ProductName>Lavender</ProductName>
</address>
<address>
<address1>N0 98</address1>
<city>BridhSanagoor Road</city>
<country>SriLanka</country>
<postalCode>641006</postalCode>
</address>
</addresses>
</Program>
<Program id="25" category="B">
<name>Rahman'G</name>
<Bin>
<Date>10/1/2020</Date>
<Date>1/12/1989</Date>
</Bin>
<Player>
<Pla>Paul'D</Pla>
<Pla>Right'F</Pla>
</Player>
<Television>
<HostDeails>5/7/2021</HostDeails>
<HostDeails>MALE</HostDeails>
<HostDeails>C'olour</HostDeails>
</Television>
<addresses>
<address>
<address1>S7</address1>
<city>Coimbatire</city>
<country>IN</country>
<ProductName>Lavender</ProductName>
</address>
<address>
<address1>Sai Akshya Appartment</address1>
<city>Sanagoor Road</city>
<postalCode>631009</postalCode>
</address>
<address>
<address1> No 3</address1>
<city>Thenkaasi</city>
<ProductName>Lavender</ProductName>
</address>
<address>
<address1>N0 98</address1>
<city>Bridh'Sanagoor Road</city>
<country>SriLanka</country>
<postalCode>641006</postalCode>
</address>
</addresses>
</Program>
</Channel>
</xml>
Thank you all.
Copied from comment:
I doing like this
SqlCommand.Parameters.Add("#XMLValue", SqlDbType.Xml).Value = xmlDetails.ToString();
If you use Parameterized Queries (Take a look at SqlParameters), you won't be having these issues.
SqlCommand cmd = new SqlCommand(query, connection);
SqlParameter param = new SqlParameter();
param.ParameterName = "#ParamName";
param.Value = valueVariable;
try not to use string concatenation and be aware of the risks and the implications.
EDIT:
xmlDetails.ToString() dont convert xmlDetails to string. Instead parse it
SqlXml newXml = new SqlXml(new XmlTextReader("MyTestStoreData.xml"));
SqlXml newXml = new SqlXml(xmlDetails); <-- xmlDetails is needed to be either Stream or XmlReader type

Writing large number of records to a CSV file - Not Working

Background: We are building an application MuleSoft and as part of the requirement we have to write a large number of records (approx. 30K) to a csv file. Before that we need to extract the data in the forms of XML, standalone data from DB2. Then we are applying some transformation/mapping rules and then finally we are writing the data to a csv file and FTP the csv file. I am attaching the XML.
Issue: The process is hanging somewhere after processing about 2500-2600 records only. It is not throwing any error. It just stays there, it doesn't do anything. We tried options like 1. Putting the flow as part of a mule batch flow. No difference observed 2. set max error count = -1, as we found this somewhere in the blog
Please if somebody can provide any suggestion, that will be really helpful. Is there any limit in number of records while writing to a file?
<?xml version="1.0" encoding="UTF-8"?>
<mule xmlns:batch="http://www.mulesoft.org/schema/mule/batch" xmlns:db="http://www.mulesoft.org/schema/mule/db"
xmlns:file="http://www.mulesoft.org/schema/mule/file"
xmlns:dw="http://www.mulesoft.org/schema/mule/ee/dw" xmlns:metadata="http://www.mulesoft.org/schema/mule/metadata"
xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation"
xmlns:spring="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.mulesoft.org/schema/mule/db http://www.mulesoft.org/schema/mule/db/current/mule-db.xsd
http://www.mulesoft.org/schema/mule/file http://www.mulesoft.org/schema/mule/file/current/mule-file.xsd
http://www.mulesoft.org/schema/mule/ee/dw http://www.mulesoft.org/schema/mule/ee/dw/current/dw.xsd
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-current.xsd
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/batch http://www.mulesoft.org/schema/mule/batch/current/mule-batch.xsd">
<db:generic-config name="Generic_Database_Configuration1" url="jdbc:db2://faadbcdd0017:60004/MATIUT:user=mat_adm;password=q1w2e3r4;" driverClassName="com.ibm.db2.jcc.DB2Driver" doc:name="Generic Database Configuration"/>
<file:connector name="File" outputPattern="Carfax.csv" writeToDirectory="C:\opt\CCM\Output\IUT" autoDelete="false" outputAppend="true" streaming="true" validateConnections="true" doc:name="File"/>
<file:connector name="File1" outputPattern="sample.txt" readFromDirectory="C:\opt\CCM" autoDelete="true" streaming="true" validateConnections="true" doc:name="File"/>
<batch:job name="batch2Batch">
<batch:input>
<logger message="Startr>>>>>>>>>>>>>>>>>>>>>>>>>>>>>" level="INFO" doc:name="Logger"/>
<foreach doc:name="For Each">
<db:select config-ref="Generic_Database_Configuration1" doc:name="Database">
<db:parameterized-query><![CDATA[select MSG_ID,TEMPL_ID,MSG_DATA,EMAIL_CHNL_IND,PUSH_CHNL_IND, INSERT_TMSP,UID FROM IUT.message_master WHERE INSERT_TMSP between
(CURRENT TIMESTAMP- HOUR (CURRENT TIMESTAMP) HOURS- MINUTE(CURRENT TIMESTAMP) MINUTES- SECOND(CURRENT TIMESTAMP) SECONDS
- MICROSECOND(CURRENT TIMESTAMP) MICROSECONDS) and ((CURRENT TIMESTAMP- HOUR (CURRENT TIMESTAMP) HOURS
- MINUTE(CURRENT TIMESTAMP) MINUTES- SECOND(CURRENT TIMESTAMP) SECONDS- MICROSECOND(CURRENT TIMESTAMP) MICROSECONDS) + 1 DAY)
and SOURCE_SYS='CSS' and ONLINE_BATCH_IND IN('Y','E') AND APPL_PROCESS_IND = 'N' with UR]]></db:parameterized-query>
</db:select>
</foreach>
<logger message="#[payload]" level="INFO" doc:name="Logger"/>
</batch:input>
<batch:process-records>
<batch:step name="Batch_Step">
<component class="com.mule.object.transformer.Mapper" doc:name="Java"/>
<dw:transform-message metadata:id="9bd2e755-065a-4208-95cf-1277f5643ee9" doc:name="Transform Message">
<dw:input-payload mimeType="application/java"/>
<dw:set-payload><![CDATA[%dw 1.0
%output application/csv separator = "|" , header = false , ignoreEmptyLine = true
---
[{
Timestamp: payload.timeStamp,
NotificationType: payload.notificationType,
UID: payload.UID,
Name: payload.messageData.firstName,
MiddleName: payload.messageData.middleName,
LastName: payload.messageData.lastName,
Email: payload.messageData.email,
HHNumber: payload.messageData.cssDataRequest.householdNumber,
PolicyNumber: payload.messageData.cssDataRequest.policyContractNumber,
SentDate: payload.messageData.cssDataRequest.sendDate,
PinNumber: payload.messageData.cssDataRequest.pin,
AOR: payload.messageData.cssDataRequest.agentOfRecord
}]]]></dw:set-payload>
</dw:transform-message>
<file:outbound-endpoint path="C:\opt\CCM\Output\IUT" connector-ref="File" responseTimeout="10000" doc:name="File"/>
</batch:step>
</batch:process-records>
<batch:on-complete>
<logger message="Batch2 Completed" level="INFO" doc:name="Logger"/>
</batch:on-complete>
</batch:job>
</mule>
Try to use Batch Processing. Inside the BatchStep keep a BatchCommit which can be used for accumulating all the records within batch. And set this attribute streaming="true" for Batch Commit block. And Your File connector should be inside Batch Commit. Let me know if this helped

SQL Server : estimated query cost does not correspond to real execution time

This is my story: I have a long running query, it takes about 10s to execute. In order to optimize it I've created an index that I thought might help - it's creation was recommended by the sys.dm_db_missing_index_group_stats system view as well.
But the server decided not to use the index. Out of curiosity, I tried to force it by hint - and the query finished in 0.5s!
I tried to find out why the server ignores the index. I updated statistics on all tables included in query, with fullscan to be sure. But no change.
Then I found surprising thing: according to execution plan, estimated subtree cost of the original query (executing 10s) was 0.675 but estimated subtree cost of query with forced index (executing 0.5s) was 3.28 - see included execution plan for detail.
If I understand it well, the server thinks that execution plan with index will run much slower than plan without it, but the reality is the other way around. And the difference is really significant. How is that possible?
To be sure about the real execution times I've set STATISTICS TIME ON and get these results:
Original query
SQL Server Execution Times:
CPU time = 10766 ms, elapsed time = 11046 ms.
Query with forced index:
SQL Server Execution Times:
CPU time = 203 ms, elapsed time = 433 ms.
I've googled it for a while but havent found any clue. I don't think parallelism could give any explanation (see Measuring Query Performance : "Execution Plan Query Cost" vs "Time Taken") because both queries run with Degree Of Parallelism = 1 and also, the "fast" query has not only short elapsed time but also low CPU time.
My environment: Windows Server 2012 R2 64bit, SQL Server 2012 (v11.0.5613.0), 64GB RAM, 16 cores. The biggest table t_object has around 25M rows.
Here is the execution plan of both queries run in a batch, first the original one, second the one with forced index:
Graphical execution plan
And here is "frame part" of execution plan in XML. I can provide a full one, after I find how to not exceed the limit of body size:
<?xml version="1.0" encoding="utf-16"?>
<BatchSequence>
<Batch>
<Statements>
<StmtSimple StatementCompId="3" StatementEstRows="241" StatementId="1" StatementOptmLevel="FULL" StatementOptmEarlyAbortReason="GoodEnoughPlanFound" StatementSubTreeCost="0.675167" StatementText="SELECT TOP 241 t_Object.c_id AS id, t_Object.c_objectDefName AS objectDefName, t_Object.c_id AS display, t_Object.c_owner AS owner, t_Object.c_ownerDefName AS ownerDefName
, t_KPDokBase.c_typ, a_KPCisTypDokumentu0.c_nazev, t_KPDokBase.c_kpid, t_KPDokBase.c_pojistnik, t_KPDokBase.c_kpdatum, t_KPDokBase.c_urgentni, t_KPDokBase.c_poznamka
FROM t_KPDokBase LEFT JOIN t_KPCiselnik a_KPCisTypDokumentu0 ON t_KPDokBase.c_typ=a_KPCisTypDokumentu0.c_Id , t_Document, t_Object
WHERE t_Document.c_Id = t_KPDokBase.c_Id AND t_Object.c_Id = t_Document.c_Id AND ((t_Object.c_createDT >= '20151021 10:16:04.875') AND (((t_KPDokBase.c_pravaTyp = '') OR (t_KPDokBase.c_pravaTyp <> 'zdravotni')))) AND t_Document.c_trashed=0 AND t_Document.c_versionType=1
--doba využití procesoru = 10531 ms, uplynulá doba = 10816 ms.
" StatementType="SELECT" QueryHash="0x5B65E1C5AEEE1DB3" QueryPlanHash="0x3AEC766C66B98CA1" RetrievedFromCache="true">
<QueryPlan DegreeOfParallelism="1" MemoryGrant="1376" CachedPlanSize="72" CompileTime="57" CompileCPU="57" CompileMemory="848">
<MemoryGrantInfo SerialRequiredMemory="1024" SerialDesiredMemory="1376" RequiredMemory="1024" DesiredMemory="1376" RequestedMemory="1376" GrantWaitTime="0" GrantedMemory="1376" MaxUsedMemory="640" />
<OptimizerHardwareDependentProperties EstimatedAvailableMemoryGrant="209569" EstimatedPagesCached="419139" EstimatedAvailableDegreeOfParallelism="8" />
<RelOp AvgRowSize="271" EstimateCPU="2.41E-05" EstimateIO="0" EstimateRebinds="0" EstimateRewinds="0" EstimatedExecutionMode="Row" EstimateRows="241" LogicalOp="Top" NodeId="0" Parallel="false" PhysicalOp="Top" EstimatedTotalSubtreeCost="0.675167">
<OutputList>
</OutputList>
<RunTimeInformation>
<RunTimeCountersPerThread Thread="0" ActualRows="241" ActualEndOfScans="1" ActualExecutions="1" />
</RunTimeInformation>
<Top RowCount="false" IsPercent="false" WithTies="false">
<TopExpression>
<ScalarOperator ScalarString="(241)">
<Const ConstValue="(241)" />
</ScalarOperator>
</TopExpression>
</Top>
</RelOp>
</QueryPlan>
</StmtSimple>
</Statements>
</Batch>
<Batch>
<Statements>
<StmtSimple StatementCompId="4" StatementEstRows="241" StatementId="2" StatementOptmLevel="FULL" StatementSubTreeCost="3.28305" StatementText="SELECT TOP 241 t_Object.c_id AS id, t_Object.c_objectDefName AS objectDefName, t_Object.c_id AS display, t_Object.c_owner AS owner, t_Object.c_ownerDefName AS ownerDefName
, t_KPDokBase.c_typ, a_KPCisTypDokumentu0.c_nazev, t_KPDokBase.c_kpid, t_KPDokBase.c_pojistnik, t_KPDokBase.c_kpdatum, t_KPDokBase.c_urgentni, t_KPDokBase.c_poznamka
FROM t_KPDokBase LEFT JOIN t_KPCiselnik a_KPCisTypDokumentu0 ON t_KPDokBase.c_typ=a_KPCisTypDokumentu0.c_Id , t_Document, t_Object WITH (INDEX(IX_cdt_obdn_ow_owdn))
WHERE t_Document.c_Id = t_KPDokBase.c_Id AND t_Object.c_Id = t_Document.c_Id AND ((t_Object.c_createDT <= '20151021 10:16:04.875') AND (((t_KPDokBase.c_pravaTyp = '') OR (t_KPDokBase.c_pravaTyp <> 'zdravotni')))) AND t_Document.c_trashed=0 AND t_Document.c_versionType=1
--doba využití procesoru = 218 ms, uplynulá doba = 669 ms.
" StatementType="SELECT" QueryHash="0xED6590A88AF84552" QueryPlanHash="0x61ED516F58A6CB68" RetrievedFromCache="true">
<StatementSetOptions ANSI_NULLS="true" ANSI_PADDING="true" ANSI_WARNINGS="true" ARITHABORT="true" CONCAT_NULL_YIELDS_NULL="true" NUMERIC_ROUNDABORT="false" QUOTED_IDENTIFIER="true" />
<QueryPlan DegreeOfParallelism="1" MemoryGrant="2451104" CachedPlanSize="80" CompileTime="26" CompileCPU="26" CompileMemory="800">
<MemoryGrantInfo SerialRequiredMemory="1536" SerialDesiredMemory="2451104" RequiredMemory="1536" DesiredMemory="2451104" RequestedMemory="2451104" GrantWaitTime="0" GrantedMemory="2451104" MaxUsedMemory="2632" />
<OptimizerHardwareDependentProperties EstimatedAvailableMemoryGrant="209569" EstimatedPagesCached="419139" EstimatedAvailableDegreeOfParallelism="8" />
</QueryPlan>
</StmtSimple>
</Statements>
</Batch>
</BatchSequence>
</ShowPlanXML>

Resources