I have camel route on ingress of files recieved, sometimes these files contain multiple can be thousands of empty rows or records. these occur at the end of the files.
help or advice on how to handle this situation.
2/3/20 0:25,12.0837099,22.07255971,51.15338002,52.76662495,52.34712651,51.12155216,45.7655507,49.96555147,54.47205637,50.66135512,54.43864717,54.31627797,112.11765,1305.89126,1318.734411,52.31780487,44.27374363,48.72548294,43.01383257,23.85434055,41.98898447,47.50916052,31.13055873,112.2747269,0.773642045,1.081464888,2.740194779,1.938788885,1.421660186,0.617588546,21.28219363,25.03362771,26.76627344,40.21132809,29.72854555,33.45911109
,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
The route goes to a splitter.
<route autoStartup="true" id="core.predix.accept.file.type.route">
<from id="_from3" uri="{{fileEntranceEndpoint}}"/>
<convertBodyTo id="_convertBodyTo1" type="java.lang.String"/>
<split id="_split1" strategyRef="csvAggregationStrategy" streaming="true" stopOnException="true">
<tokenize token="\n"/>
<process id="_process3" ref="toCsvFormat"/>
<!-- passthru only we do not allow embedded commas in numeric data -->
</split>
<log id="_log1" loggingLevel="INFO" message="CSV body: ${body}"/>
<choice id="_choice1">
<when id="_when1">
<simple>${header.CamelFileName} regex '^.*\.(csv|CSV|txt|gpg)$'</simple>
<log id="_log2" message="${file:name} accepted for processing..."/>
<choice id="_choice2">
<when id="_when2">
<simple>${header.CamelFileName} regex '^.*\.(CSV|txt|gpg)$'</simple>
<setHeader headerName="CamelFileName" id="_setHeader1">
<simple>${file:name.noext.single}.csv</simple>
</setHeader>
<log id="_log3" message="${file:name} changed file name."/>
</when>
</choice>
<split id="_split2" streaming="true">
<tokenize prop:group="noOfLines" token="\n"/>
<log id="_log4" message="Split Group Body: ${body}"/>
<to id="_to1" uri="bean:extractHeader"/>
<to id="acceptedFileType" ref="predixConsumer"/>
</split>
<to id="_to2" uri="bean:extractHeader?method=cleanHeader"/>
</when>
<otherwise id="_otherwise1">
<log id="_log5" loggingLevel="INFO" message="${file:name} is an unknown file type, sending to unhandled repo."/>
<to id="_to3" uri="{{unhandledArchive}}"/>
</otherwise>
</choice>
</route>
The simple aggregator
public class CsvAggregationStrategy implements AggregationStrategy {
private Logger log = LoggerFactory.getLogger(CsvAggregationStrategy.class.getName());
#Override
public Exchange aggregate(Exchange oldExchange, Exchange newExchange) {
//Theory
//-------------------------------------------------------------------------------------
// Arrived | oldExchange | newExchange | Description
//-------------------------------------------------------------------------------------
// A | NULL | A | first message arrives for the first group
// B | A | B | second message arrives for the first group
// F | NULL | F | first message arrives for the second group
// C | AB | C | third message arrives for the first group
//---------------------------------------------------------------------------------------
log.debug("Aggregation Strategy :: start");
if ( newExchange.getException() != null ) {
if ( oldExchange == null ) {
return newExchange;
} else {
oldExchange.setException(newExchange.getException());
return oldExchange;
}
}
if ( oldExchange == null ) { //This will set the 1st record with the Header
return newExchange;
}
String newBody = newExchange.getIn().getBody(String.class);
String oldBody = oldExchange.getIn().getBody(String.class);
String body = oldBody + newBody;
oldExchange.getIn().setBody( body );
log.debug("Aggregation Strategy :: finish");
return oldExchange;
} //Exchange process
} //class AggregationStrategy
I thought I would handle the empty rows in the class toCsvFormat
The class ToCsvFormat simply changes the inbound csv delimiter to a comma.
public class ToCsvFormat implements Processor {
private static final Logger LOG = LoggerFactory.getLogger(ToCsvFormat.class);
#Override
public void process(Exchange exchange) throws Exception {
String body = exchange.getIn().getBody(String.class);
body = body.replaceAll("\\t|;",",");
String bodyCheck = body.replaceAll(",","").trim();
LOG.info("BODY CHECK: " + bodyCheck);
if ( bodyCheck.isEmpty() || bodyCheck == null ) {
throw new IllegalArgumentException("Data record is Empty or NULL. Invalid Data!");
} else {
StringBuilder sb = new StringBuilder(body.trim());
LOG.debug("CSV Format Body In: " + sb.toString());
LOG.debug("sb length: " + sb.length());
if ( sb.toString().endsWith(",") ) {
sb.deleteCharAt(sb.lastIndexOf(",", sb.length()));
}
LOG.info("CSV Format Body Out: " + sb.toString());
sb.append(System.lineSeparator());
exchange.getIn().setBody(sb.toString());
}
}
}
*** the problem I'm having is I need the splitter to finish processing until it hits all the empty rows, or skip over or stop the splitter on empty records. but I need what was previously split or processed. Throwing and capture of exception stops the splitter I get nothing. I'm using the splitter stoponexception but like it says, it stops on the exception.
thank you
So you set up stopOnException=true and asked why your route stopped when exception wasn't catched =) ? As workaround forget about throwing exception and validate your body and if it has inappropriate data just set empty body and then sum them in your AggregationStrategy like in pseudo-route below. I haven't used the xml description for a very long time so i hope your will understand this example with Java DSL.
public class ExampleRoute extends RouteBuilder {
AggregationStrategy aggregationStrategy = new AggregationStrategy() {
#Override
public Exchange aggregate(final Exchange oldExchange, final Exchange newExchange) {
log.debug("Aggregation Strategy :: start");
if (oldExchange != null) {
newExchange.getIn().setBody(newExchange.getIn().getBody(String.class) + oldExchange.getIn().getBody(String.class));
}
log.debug("Aggregation Strategy :: finish");
return newExchange;
}
};
#Override
public void configure() throws Exception {
from("{{fileEntranceEndpoint}}")
.convertBodyTo(String.class)
.split(tokenize("\n"), aggregationStrategy).streaming().stopOnException()
.choice()
.when(body().regex(",+\\$"))
.setBody(constant(""))
.otherwise()
.process("toCsvFormat")
;
}
I recommend you use Java DSL. As you can see, many things are easy to use with it.
Thank you C0ld. Appreciate going easy. yeah, I get it. sometimes we do silly things, why another pair of eyes is a wonderful thing. I took your suggestion and it works like a charm. thank you very much for responding.
<split id="_split1"
strategyRef="emptyRecordAggregationStrategy" streaming="true">
<tokenize token="\n"/>
<choice id="_choice5">
<when id="_when5">
<simple>${body} regex '^,+$'</simple>
<setBody id="_setBody1">
<constant/>
</setBody>
</when>
<otherwise>
<process id="_processCSV" ref="toCsvFormat"/>
</otherwise>
</choice>
</split>
public class EmptyRecordAggregationStrategy implements AggregationStrategy {
private Logger log = LoggerFactory.getLogger(EmptyRecordAggregationStrategy.class.getName());
#Override
public Exchange aggregate(Exchange oldExchange, Exchange newExchange) {
if ( newExchange.getException() != null ) {
if ( oldExchange == null ) {
newExchange.getIn().setBody(newExchange.getIn().getBody(String.class) + System.lineSeparator());
return newExchange;
} else {
oldExchange.getIn().setBody(oldExchange.getIn().getBody(String.class) + System.lineSeparator());
return oldExchange;
}
}
if ( oldExchange == null ) {
newExchange.getIn().setBody(newExchange.getIn().getBody(String.class) + System.lineSeparator());
return newExchange;
}
if ( !newExchange.getIn().getBody(String.class).isEmpty() ) {
oldExchange.getIn().setBody(oldExchange.getIn().getBody(String.class) + newExchange.getIn().getBody(String.class) + System.lineSeparator());
}
return oldExchange;
}
}
Is there any way to select/update/delete dynamically using Ibatis/MyBatis?
When I say "dynamically" it means I don't want to create any POJO/DataMapper at all.
Any URL example would be welcomed.
Yes, just set the resultType attribute to map and the table data will be placed into a HashMap of column names to values. If the query returns more than 1 row, the mapped rows will be put into a List. If you want to select a single column, you can get just that value (as String, int, etc) or as a list.
<select id="test1" resultType="map">select * from user</select>
<select id="test2" resultType="map" parameterType="int">
select * from user where id=#{value}</select>
<select id="test3" resultType="string">select name from user</select>
...
// returns a list of maps
List test = sqlSession.selectList("test1");
// returns a single map
Object map = sqlSession.selectOne("test2", 0);
// returns a list of strings
List names = sqlSession.selectList("test3");
This applies to MyBatis 3; I think you can do something similar in iBatis 2.
The following approach can be useful.
Say, you have some generic select interface, like:
public interface IAutoRepository {
/**
* The automatically generated insertPKs sql statements.
* Parts of the query can be set manually in the sql (insert-select query).
*
* #param items the {#link WhereStmt} statements
* #return the inserted rows count
*/
#Transactional
<T extends WhereStmt> Integer insertPKs(#Param("items") List<T> items);
/**
* Returns the value based on the {#link Parameter} class
*
* #param param the {#link Parameter} instance
* #return the searched value in a {#link java.util.Map} form
*/
#MapKey("VAL")
<T extends Parameter> Map<String, Map<String, ?>> getDistinctValues(#Param("param") T param);
}
According to some external type (say, single column or date range or range whatsoever) you can define the following query in the template Common.xml:
<sql id="includeDistinctValues">
SELECT
<choose>
<when test='param.type.name() == "set"'>
DISTINCT ${param.column} AS val
</when>
<when test='param.type.name() == "date" or param.type.name() == "range"'>
<some uid> AS val,
MIN(${param.minColumn}) AS min,
MAX(${param.maxColumn}) AS max
</when>
</choose>
FROM ${entityTable}
</sql>
What you receive from mybatis is the java.util.Map. Then you can you use it some kind like:
public enum StmtType {
set((valMap) -> {
final Set<String> distinctValues = valMap
.values()
.stream()
.map(val -> (String) val.get("VAL"))
//use in date/range case
//final Date minDate = (Date) val.get("MIN");
//final Date maxDate = (Date) val.get("MAX");
.collect(Collectors.toSet());
return distinctValues;
},
(values, params) -> {
final SetParameter parameter = (SetParameter) params.getParams();
return new WhereSetStmt<>(parameter.getColumn(), values, params.getIncludeEmptyValues());
});
#Getter
private Function<Map<String, Map<String, ?>>, ? extends Iterable> toValue;
#Getter
private BiFunction<Collection, DataParam, ? extends WhereStmt> toWhereStmt;
StmtType(
Function<Map<String, Map<String, ?>>, ? extends Iterable> toValue,
BiFunction<Collection, DataParam, ? extends WhereStmt> toWhereStmt
) {
this.toValue = toValue;
this.toWhereStmt = toWhereStmt;
}
}
where the SetParameter can be represented as the following:
#Getter
public class SetParameter extends Parameter {
/**
* Column in sql query,
*/
private final String column;
public SetParameter(String column) {
super(StmtType.set);
this.column = column;
}
}
Moreover, you can define some WhereStmt like:
public abstract class WhereStmt {
/**
* Type of the statement
*/
private final StmtType type;
/**
* Shall empty values be included.
*/
private final boolean includeEmptyValues;
}
#Getter
public class WhereSetStmt<T> extends WhereStmt {
/**
* The column for `column` IN (...) statement
*/
private String column;
/**
* Values for `column` IN (...) statement
*/
private Collection<T> values;
public WhereSetStmt(String column, Collection<T> values, boolean includeEmptyValues) {
super(StmtType.set, includeEmptyValues);
this.column = column;
this.values = values;
}
}
#Getter
#AllArgsConstructor
public final class DataParam<P extends Parameter> {
/**
* Whether to include nullable values.
*/
private final Boolean includeEmptyValues;
/**
* Represents database required information for later processing and sql statements generation.
*/
private final P params;
}
Finally, in mybatis generic Common.xml you can use it like:
<sql id="includeInsertPkSelect">
SELECT DISTINCT(${id})
FROM ${entityTable}
</sql>
<sql id="includeInsertPkWhere">
<if test="items != null and items.size() > 0">
AND
<foreach collection="items" item="item" index="i" separator="AND">
<choose>
<when test='item.type.name() == "set" and ( item.values != null and item.values.size() > 0 or item.includeEmptyValues )'>
(
<if test="item.values != null and item.values.size() > 0">
${item.column} IN
<foreach item="value" collection="item.values" separator="," open="("
close=")">
#{value}
</foreach>
<if test="item.includeEmptyValues">
OR
</if>
</if>
<if test="item.includeEmptyValues">
${item.column} IS null
</if>
)
</when>
<when test='item.type.name() == "date"'>
(
COALESCE(${item.column}, SYSDATE + 1000000)
BETWEEN #{item.from} AND #{item.to}
<if test="item.includeEmptyValues">
OR ${item.column} IS NULL
</if>
)
</when>
<when test='item.type.name() == "range"'>
(
COALESCE(${item.column}, 1000000000)
BETWEEN #{item.from} AND #{item.to}
<if test="item.includeEmptyValues">
OR ${item.column} IS NULL
</if>
)
</when>
</choose>
</foreach>
</if>
</sql>
And combine sql statements as the template, say:
<insert id='insertPKs'>
INSERT INTO requiredTable
<include refid="Common.includeInsertPkSelect">
<property name="id" value="entityId"/>
<property name="entityTable" value="entityTable"/>
</include>
<include refid="Common.includeInsertPkWhere">
<property name="id" value="entityId"/>
<property name="entityTable" value="entityTable"/>
</include>
</insert>
Yes, it should be possible to build the mapping in runtime through an API, and to use Maps instead of entity classes.
Yes, you can do select/update/delete without using POJO.
In myBatis you can write your query in Mapper XML and set the resultType that will be returned from SQL statement and store that in objects.
For Example,
<mapper namespace = "data">
<select id = "fetch" resultType = "java.util.HashMap">
select * from table_name
</select>
</mapper>
You can use List of Maps to store those results.
List<Map<String,Object>> results = session.selectList("data.fetch");