How to transform a CSV file data in Apache camel - apache-camel

Question

I want to transform some field's data in specific rows in csv file.I tried the following .
1).Using csv marshaling and unmarshaling I achieved it ,but the output CSV is not coming in proper order even though I sent list of maps (i.e List) .
following is my program
from("file:E://camelinput//?noop=true")
.unmarshal(csv)
.convertBodyTo(List.class)
.process(new Processor() {
#Override
public void process(Exchange msg) throws Exception {
List<List<String>> data = (List<List<String>>) msg.getIn().getBody();
List<Map<String,Object>> newdata=new ArrayList<Map<String,Object>>();
Map<String,Object> map=null;
for (List<String> line : data) {
System.out.println(line.size());
map=new HashMap<String,Object>();
if("1502873".equals(line.get(3))){
line.set(18, "Y");
}
// newdata.add(line);
int count=0;
for(Object field:line){
// System.out.println("line.get(count) "+line.get(count));
map.put(String.valueOf(count),field);
count++;
}
newdata.add(map);
}
msg.getIn().setBody(newdata);
}
})
.marshal().csv().convertBodyTo(List.class)
.to("file:E://camelout").end();
2)And again I tried Using .split(body()) and trying to process each row(i.e with out using Marshaling I am trying),but it is taking very huge time and getting terminated with some Interrupted exception.
following is the code
from("file:E://camelinput//?noop=true")
.unmarshal(csv)
.convertBodyTo(List.class)
.split(body())
.process(new Processor() {
#Override
public void process(Exchange msg) throws Exception {
List<String> rec= new ArrayList<String>();
if("1502873".equals(rec.get(3))){
rec.set(18, "Y");
}
String dt=rec.toString().trim().replace("[","").replace("]", "");
msg.getIn().setBody(dt, String.class);
}
})
.to("file:E://camelout").end();
following is my sample Csv
25 STANDARD N 1435635 415 1087 15904 7 null 36 Cross Mechanical Pencil, Lead and Eraser, 0.5 mm 2 23162 116599 7/7/2015 15:45 N 828
25 STANDARD N 1435635 415 1087 15905 8 null 36 Jumbo Ballpoint and Selectip Refill, Medium, Black 4 23163 116599 7/7/2015 15:45 N 829
25 STANDARD N 1435635 415 1087 15906 1 3487 null 598 Copier Toner, Cannon 220 23164 116599 7/7/2015 15:45 N 830
25 STANDARD N 1435635 415 1087 15907 2 3495 null 823 Envelopes 27 23165 116599 7/7/2015 15:45 N 831
25 STANDARD N 1435635 415 1087 15908 3 3513 null 789 Legal Pads, 8 1/2 x 11 3/4" White" 30 23166 116599 N 832
25 STANDARD N 1435635 415 1087 15909 4 3577 null 791 Paper Clips 5 23167 116599 7/7/2015 15:45 N 833
31 STANDARD N 1574437 415 1087 15910 5 null 36 Clic Stic Pen, Fine, Black 0.72 23168 116599 7/7/2015 15:45 N 834
31 STANDARD N 1574437 557 1233 15911 6 null 36 Laser Cards, 50 Note Cards/Envelopes, 4-1/4 inch x 5-1/2 inch, White 21.58 23169 116599 7/7/2015 15:45 N 835
31 STANDARD N 1574437 578 1275 15912 1 201 null 32 Keyboard - 101 Key 20.82 23170 116599 7/7/2015 15:45 N 836
25 STANDARD N 1574437 147 2033 15913 1 225 null 30 Monitor - 19" 225.39 23171 116599 7/7/2015 15:45 N 837
1314 STANDARD N 1502873 22 2199 16287 1 628 null 1 Envoy Deluxe Laptop 822.87 23545 116599 7/7/2015 15:45 N 838
1314 STANDARD N 1502873 22 2199 16288 1 151 null 91 Envoy Standard Laptop 1283.44 23546 116599 7/7/2015 15:45 N 839
7653 STANDARD N 1502873 22 2199 16289 2 606 null 1 Battery - Extended Life 28 23547 116599 7/7/2015 15:45 N 840
7652 STANDARD N 1502873 21 459 16290 1 2157 null 1 Envoy Laptop - Rugged 1525.02 23548 116599 7/7/2015 15:45 N 841
1314 STANDARD N 1502873 3 1594 16291 1 251 null 32 RAM - 256MB 51.25 23549 116599 7/7/2015 15:45 N 842
7654 STANDARD N 1502873 22 2199 16292 1 606 null 1 Battery - Extended Life 28 23550 116599 7/7/2015 15:45 N 843
7652 STANDARD N 1502873 21 459 16293 1 247 null 30 Monitor - 17" 225.39 23551 116599 7/7/2015 15:45 N 844
1704 STANDARD N 1502873 41 2200 16294 2 225 null 30 Monitor - 19" 225.39 23552 116599 7/7/2015 15:45 N 845
7658 STANDARD N 1502873 21 460 16295 1 201 null 32 Keyboard - 101 Key 20.82 23553 116599 7/7/2015 15:45 N 846
I have large Csv files which contains hundreds of thousands of rows.

Answer 1

I think your solution 1 might be overly complex if you only want to alter values in csv and output it it back in the same order. Just edit fields in the original List and marshall it back to file.
I've made here assumption that your data was actually delimited by tabs rather than random amount of spaces in your example but I've included the CsvDataFormat that I used. Code uses camel-core and camel-csv version 2.15.3.
public void configure() {
CsvDataFormat csv = new CsvDataFormat();
csv.setDelimiter('\t'); // Tabs
csv.setQuoteDisabled(true); // Otherwise single quotes will be doubled.
from("file://src/data?fileName=data.csv&noop=true&delay=15m")
.unmarshal(csv)
.convertBodyTo(List.class)
.process(new Processor() {
#Override
public void process(Exchange msg) throws Exception {
List<List<String>> data = (List<List<String>>) msg.getIn().getBody();
for (List<String> line : data) {
// Checks if column two contains text STANDARD
// and alters its value to DELUXE.
if ("STANDARD".equals(line.get(1))) {
System.out.println("Original:" + line);
line.set(1, "DELUXE");
System.out.println("After: " + line);
}
}
}
}).marshal(csv).to("file://src/data?fileName=out.csv")
.log("done.").end();
}

Answer 2

The problem is that you are processing single line in single thread. If parallel processing correct for you, try to use ThreadPool.
<camel:camelContext id="camelContext">
.....
<camel:threadPoolProfile id="customThreadPoolProfile"
defaultProfile="true"
poolSize="{{split.thread.pool.size}}"
maxPoolSize="{{split.thread.max.pool.size}}"
maxQueueSize="{{split.thread.max.queue.size}}">
</camel:threadPoolProfile>
</camel:camelContext>
And upgrade split
.split(body().tokenize("\n"))
.streaming()
.parallelProcessing()
.executorServiceRef("customThreadPoolProfile")
.....
.end()