Does camel have support for streaming splitting of large JSON data? I know that the support is there for xml. I tried splitting large json files with jsonpathexpression. It worked fine with small files but gives OOM exception with large files.
JsonPathExpression jsonPathExpression = new JsonPathExpression("$.");
from("stream:hxxp://dummy:5984/test/a123/p.txt")
.split(jsonPathExpression).streaming().parallelProcessing()
.process(new Processor() {
#Override
public void process(Exchange exchange) throws Exception {
String s = exchange.getIn().getBody(String.class);
}
})});
Related
I have the following route:
from("file:/home/tmp/test?move=.done")
.routeId("file")
.split(body().tokenize("\n"),new GroupedBodyAggregationStrategy())
.process(new Processor() {
#Override
public void process(Exchange exchange) throws Exception {
exchange.getMessage().setHeader("Test", "test");
System.out.println(exchange.getIn().getBody());
}
})
and the file that is being consumed is:
1234,56676,2345
234,213,412
124,423,5236
I would expect only one exchange to reach the processor after split and that its body contains all the above lines, however it behaves like I do not use an aggregation strategy at all. 3 exchanges reach the processor each with the body that corresponds to a single line. Also, it does not matter if I use GroupedBodyAggregationStrategy or a custom one, it is always the same. What am I doing wrong?
Give a look here enter link description here
What you look is :
from("file:/home/tmp/test?move=.done")
.routeId("file")
.split(body().tokenize("\n"),new GroupedBodyAggregationStrategy())
// instructions on each message from the split (.ie each row here)
.end()
// instructions on aggregated result
.process(new Processor() {
#Override
public void process(Exchange exchange) throws Exception {
exchange.getMessage().setHeader("Test", "test");
System.out.println(exchange.getIn().getBody());
}
})
The goal is to produce a report every hour by comparing two CSV files with
use of Camel 3.0.0. One is located on a FTP server, the other on disk. How to use poll enrich pattern in combination with unmarshalling the CSV on disk with Bindy Dataformat?
Example code (for simplicity the FTP endpoint is replaced by a file endpoint):
#Component
public class EnricherRoute extends RouteBuilder {
#Override
public void configure() {
from("file://data?fileName=part_1.csv&scheduler=quartz2&scheduler.cron=0+0+0/1+*+*+?")
.unmarshal().bindy(BindyType.Csv, Record.class)
.pollEnrich("file://data?fileName=part_2.csv", new ReportAggregationStrategy())
.marshal().bindy(BindyType.Csv, Record.class)
.to("file://reports?fileName=report_${date:now:yyyyMMdd}.csv");
}
}
The problem in this example is that in the ReportAggregationStrategy the resource (coming from data/part_2.csv, see below) is not unmarshalled. How to unmarshal data/part_2.csv as well?
public class ReportAggregationStrategy implements AggregationStrategy {
#Override
public Exchange aggregate(Exchange original, Exchange resource) {
final List<Record> originalRecords = original.getIn().getBody(List.class);
final List<Record> resourceRecords = resource.getIn().getBody(List.class); // Results in errors!
...
}
}
You can wrap enrichment with direct endpoint and do unmarshaling there.
from("file://data?fileName=part_1.csv&scheduler=quartz2&scheduler.cron=0+0+0/1+*+*+?")
.unmarshal().bindy(BindyType.Csv, Record.class)
.enrich("direct:enrich_record", new ReportAggregationStrategy())
.marshal().bindy(BindyType.Csv, Record.class)
.to("file://reports?fileName=report_${date:now:yyyyMMdd}.csv");
from("direct:enrich_record")
.pollEnrich("file://data?fileName=part_2.csv")
.unmarshal().bindy(BindyType.Csv, Record.class);
I'm using camel-hystrix-eip in my project, I want a processor before my fallback, but it's not working
from("direct:sample").id("id:direct:sample").process(requestProcessor)
.hystrix()
.to(endPoint)
.onFallbackViaNetwork()
.to(fallback_endPoint)
I want to alter my fallback_endpoint using a processor, but seems like after onFallbackViaNetwork() we have to immediately provide to().
please suggest if there any way to do so.
I tried something like below, but it's not working.
from("direct:sample").id("id:direct:sample").process(requestProcessor)
.hystrix()
.to(endPoint)
.onFallbackViaNetwork()
.process(fallbackProcessor)
.to(fallback_endPoint)
Actually, I'm using requestProcessor to override the actual endpoint, and in case of a fallback, fallback_endPoint is also getting overridden, is there any way to avoid this.
You can have a processor after onFallbackViaNetwork(). You can also use the toD EIP to send the message to a dynamic endpoint.
Based on your code, you could set a Header MyEndpoint which contains your new endpoint string, and then reference it using .toD("${header.MyEndpoint}"). Repeat this pattern whenever you need to set a dynamic endpoint.
For example:
from("direct:sample")
.process(new Processor() {
#Override
public void process(Exchange exchange) throws Exception {
// do something
exchange.getIn().setHeader("EndpointHystrix", "mock:hystrix");
}
})
.hystrix()
.toD("${header.EndpointHystrix}")
.onFallbackViaNetwork()
.process(new Processor() {
#Override
public void process(Exchange exchange) throws Exception {
// do something more
exchange.getIn().setHeader("EndpointFallback", "mock:fallback");
}
})
.toD("${header.EndpointFallback}")
.end()
.to("...");
I've tested this in Camel 2.20.0 and 2.21.0.
I am newcomer in Apache Camel. Please have a look to my code bellow:
I have a service which exposed as cxf webservice:
interface CxfService{
public OutputType hello(InputType input);
}
This is my route:
from("cxf:/test?serviceClass=" + CxfService.class.getName())
.to("log:cxfLog1")
.recipientList(simple("direct:${header.operationName}"));
from("direct:hello")
.process(new Processor(){
public void process(Exchange exchange) throws Exception {
InputType file = exchange.getIn().getBody(InputType.class);
exchange.getOut().setBody(new OutputType());
}
});
The code works as expected, it consume InputType and produce OutputType.
I want to borrow my body to do another stuffs, so i rewrite that like this:
from("cxf:/test?serviceClass=" + CxfService.class.getName())
.to("log:cxfLog1")
.recipientList(simple("direct:${header.operationName}"));
from("direct:hello")
.process(new Processor(){
public void process(Exchange exchange) throws Exception {
InputType file = exchange.getIn().getBody(InputType.class);
exchange.getOut().setHeader("header.temporary", new OutputType());
}
})
.to("some endpoint")
.setBody(simple("${header.temporary}"));
This webservice consume InputType and produce nothing. What wrong with that?
In your second piece of code, when setting the header.temporary, you should change two things:
setHeader("temporary", new OutputType()) - the 'header' prefix isn't
needed - you're addressing headers directly via the method call.
Use getIn() instead of getOut(). The input will get copied to the
output. You may want to do some research into the procedure for
Camel building the out message for details - I'm not 100% sure of
this one.
Change
exchange.getOut().setHeader("header.temporary", new OutputType());
To
exchange.getIn().setHeader("temporary"), new OutputType());
.setHeader() is when you use the simple language. In 99% of the cases getIn() is sufficient.
In my Camel router:
from(<SourceURI>)
.process(new Processor() {
#Override
public void process(Exchange exchange) throws Exception {
// I want to extract the file object from the exchange
}
.to(<targetURI>).
How can I achieve this?
I tried e.g. exchange.getIn().getHeader(Exchange.FILE_NAME, String.class) which gives me the file name.
I am searching for something Exchange.FILE which gives me the actual file object. My Ultimate goal is to extract the file in the processor as the routed exchange is an archive file.
Get the file from the body. Camel uses a 'org.apache.camel.component.file.GenericFile' to store as the file body. But you can use Camel's type converters to get the file in a type you want.
For example you can get the content in different types, such as:
String text = exchange.getIn().getBody(String.class);
byte[] bytes = exchange.getIn().getBody(byte[].class);
InputStream is = exchange.getIn().getBody(InputStream.class);
For those who have a from("file:...") the following works:
File in = exchange.getIn().getBody(File.class);