Camel consuming large file encoded with base64 - file

Question

I'm trying to read a large file with apache camel. The content of the file is base64 encoded and I want to decode this. Reading a large ASCII-based text file is no problem, but this one...
In this case I cannot split it at the end of a line, because the base64 line end is not the line end of my data (the data is csv). Later I want do manipulate the data.
With following camel route it works, but so the file have to exist and is not locked, when the route starts, else I get an IllegalArgumentException. But I don't know, when the file will exist.
EDIT: I forgot: camel-version: 2.14.1 | java: 1.7
I need a solution for my promblen, can someone help me?
from("stream:file?fileName=/path/to/file&scanStream=true")
.routeId(ROUTE_ID_BASE64)
.unmarshal().base64()
.setHeader("foo", constant("foo"))
.aggregate(header("foo"), new StringBodyAggregator())
.completionSize(1000)
.completionTimeout(1500)
.to("file:" + this.tempPfad + "?fileName=" + this.tempDateiname + "&charset=utf-8&fileExist=Append");
I tried also:
from("file:" + this.eingabePfad + "?filter=#dateinamenFilter&maxDepth=1&readLock=changed&noop=true&charset=utf-8")
.routeId(ROUTE_ID_BASE64)
.unmarshal().base64()
.setHeader("foo", constant("foo"))
.aggregate(header("foo"), new StringBodyAggregator())
.completionSize(1000)
.completionTimeout(1500)
.to("file:" + this.tempPfad + "?fileName=" + this.tempDateiname + "&charset=utf-8&fileExist=Append");
But then I get following error:
ERROR Failed delivery for (MessageId: ID-smith-52872-1421933301634-0-1 on ExchangeId: ID-smith-52872-1421933301634-0-3). Exhausted after delivery attempt: 2 caught: org.apache.camel.component.file.GenericFileOperationFailedException: Cannot store file: /path/to/temp/tempDatei.csv. Processed by failure processor: FatalFallbackErrorHandler[Channel[DelegateSync[yy.xxxxx.myproject.routes.DateiPolling$1#5734ea]]]
Message History
---------------------------------------------------------------------------------------------------------------------------------------
RouteId ProcessorId Processor Elapsed (ms)
[routeBase64 ] [routeBase64 ] [file:///path/to/?charset=utf-8&filter=%23datein] [ 1998]
[routeBase64 ] [unmarshal2 ] [unmarshal[org.apache.camel.model.dataformat.Base64DataFormat#153097e] ] [ 5]
[routeBase64 ] [setHeader1 ] [setHeader[foo] ] [ 0]
[routeBase64 ] [aggregate1 ] [aggregate[header(foo)] ] [ 0]
[routeBase64 ] [to1 ] [file:/path/to/temp/?fileName=tempDatei.csv&char] [ 24]
[ ] [process1 ] [yy.xxxxx.myproject.routes.DateiPolling$1#5734ea ] [ 15]
Exchange
---------------------------------------------------------------------------------------------------------------------------------------
Exchange[
Id ID-smith-52872-1421933301634-0-3
ExchangePattern InOnly
Headers {breadcrumbId=ID-smith-52872-1421933301634-0-1, CamelFileAbsolute=true, CamelFileAbsolutePath=/path/to/base64File.txt, CamelFileLastModified=1417777182000, CamelFileLength=267439971, CamelFileName=base64File.txt, CamelFileNameConsumed=base64File.txt, CamelFileNameOnly=base64File.txt, CamelFileParent=/path/to, CamelFilePath=/path/to/base64File.txt, CamelFileRelativePath=base64File.txt, CamelRedelivered=true, CamelRedeliveryCounter=1, CamelRedeliveryMaxCounter=1, foo=foo}
BodyType org.apache.commons.codec.binary.Base64InputStream
Body [Body is instance of java.io.InputStream]
]
Stacktrace
---------------------------------------------------------------------------------------------------------------------------------------
org.apache.camel.component.file.GenericFileOperationFailedException: Cannot store file: /path/to/temp/tempDatei.csv
at org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:276)
at org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:277)
at org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:165)
at org.apache.camel.component.file.GenericFileProducer.process(GenericFileProducer.java:79)
at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61)
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:120)
at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:416)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:105)
at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:87)
at org.apache.camel.processor.aggregate.AggregateProcessor$1.run(AggregateProcessor.java:548)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at org.apache.camel.util.concurrent.SynchronousExecutorService.execute(SynchronousExecutorService.java:62)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:110)
at org.apache.camel.processor.aggregate.AggregateProcessor.onSubmitCompletion(AggregateProcessor.java:540)
at org.apache.camel.processor.aggregate.AggregateProcessor.access$900(AggregateProcessor.java:82)
at org.apache.camel.processor.aggregate.AggregateProcessor$AggregationTimeoutMap.onEviction(AggregateProcessor.java:853)
at org.apache.camel.processor.aggregate.AggregateProcessor$AggregationTimeoutMap.onEviction(AggregateProcessor.java:814)
at org.apache.camel.support.DefaultTimeoutMap.purge(DefaultTimeoutMap.java:212)
at org.apache.camel.processor.aggregate.AggregateProcessor$AggregationTimeoutMap.purge(AggregateProcessor.java:826)
at org.apache.camel.support.DefaultTimeoutMap.run(DefaultTimeoutMap.java:162)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Stream closed
at java.io.BufferedReader.ensureOpen(BufferedReader.java:115)
at java.io.BufferedReader.read(BufferedReader.java:172)
at org.apache.camel.converter.IOConverter$1.read(IOConverter.java:83)
at java.io.InputStream.read(InputStream.java:170)
at java.io.InputStream.read(InputStream.java:101)
at org.apache.commons.codec.binary.BaseNCodecInputStream.read(BaseNCodecInputStream.java:158)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at java.io.BufferedReader.read1(BufferedReader.java:203)
at java.io.BufferedReader.read(BufferedReader.java:279)
at java.io.Reader.read(Reader.java:140)
at org.apache.camel.util.IOHelper.copy(IOHelper.java:224)
at org.apache.camel.component.file.FileOperations.writeFileByReaderWithCharset(FileOperations.java:402)
at org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:266)
... 29 more
2015-01-22 14:28:26,113 [eTimeoutChecker] AggregateProcessor WARN Error processing aggregated exchange. Exchange[base64File.txt]. Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot store file: /path/to/temp/tempDatei.csv]
org.apache.camel.component.file.GenericFileOperationFailedException: Cannot store file: /path/to/temp/tempDatei.csv
at org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:276)
at org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:277)
at org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:165)
at org.apache.camel.component.file.GenericFileProducer.process(GenericFileProducer.java:79)
at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61)
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:120)
at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:416)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:105)
at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:87)
at org.apache.camel.processor.aggregate.AggregateProcessor$1.run(AggregateProcessor.java:548)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at org.apache.camel.util.concurrent.SynchronousExecutorService.execute(SynchronousExecutorService.java:62)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:110)
at org.apache.camel.processor.aggregate.AggregateProcessor.onSubmitCompletion(AggregateProcessor.java:540)
at org.apache.camel.processor.aggregate.AggregateProcessor.access$900(AggregateProcessor.java:82)
at org.apache.camel.processor.aggregate.AggregateProcessor$AggregationTimeoutMap.onEviction(AggregateProcessor.java:853)
at org.apache.camel.processor.aggregate.AggregateProcessor$AggregationTimeoutMap.onEviction(AggregateProcessor.java:814)
at org.apache.camel.support.DefaultTimeoutMap.purge(DefaultTimeoutMap.java:212)
at org.apache.camel.processor.aggregate.AggregateProcessor$AggregationTimeoutMap.purge(AggregateProcessor.java:826)
at org.apache.camel.support.DefaultTimeoutMap.run(DefaultTimeoutMap.java:162)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Stream closed
at java.io.BufferedReader.ensureOpen(BufferedReader.java:115)
at java.io.BufferedReader.read(BufferedReader.java:172)
at org.apache.camel.converter.IOConverter$1.read(IOConverter.java:83)
at java.io.InputStream.read(InputStream.java:170)
at java.io.InputStream.read(InputStream.java:101)
at org.apache.commons.codec.binary.BaseNCodecInputStream.read(BaseNCodecInputStream.java:158)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at java.io.BufferedReader.read1(BufferedReader.java:203)
at java.io.BufferedReader.read(BufferedReader.java:279)
at java.io.Reader.read(Reader.java:140)
at org.apache.camel.util.IOHelper.copy(IOHelper.java:224)
at org.apache.camel.component.file.FileOperations.writeFileByReaderWithCharset(FileOperations.java:402)
at org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:266)
... 29 more

Answer 1

With adding the line .split().tokenize(LINE_SEPARATOR).streaming() it works now. Important is .end() at the end of the route, and not direct after the unmarshalling.
I changed also the Aggregatorstrategy from String to byte[], to prevent errors with UTF-8 encoding.
The complete route:
from("file:" + this.eingabePfad + "?filter=#dateinamenFilter&maxDepth=1&readLock=changed&noop=true")
.routeId(ROUTE_ID_BASE64)
.split().tokenize(LINE_SEPARATOR).streaming()
.unmarshal().base64()
.setHeader("foo", constant("foo"))
.aggregate(header("foo"), new ByteArrayBodyAggregator())
.completionSize(1000)
.completionTimeout(1500)
.to("file:" + this.tempPfad + "?fileName=" + this.tempDateiname + "&charset=utf-8&fileExist=Append")
.end();