I'm using the Camel's ProducerTemplate to make http calls like below.
#Produce
private ProducerTemplate producer;
producer.requestBodyAndHeaders(url, requestString, headers,
String.class);
In my application, I'm using this to call another time taking rest api multiple times for a batch job. But from the access logs of the other server, I found that the server is getting only 2 requests at a time.
Once, it processes these two, it was getting another 2 requests.
Looks like something to do with the connection pooling of Camel's producerTemplate. Is it defaulting to 2 connections?
How can I change this configuration?
Related
I am using camel to orchestrate web service calls to multiple rest endpoint. I am using multicast to send the request params and creating headers for each webservice in different routes and using recepientList to call dynamic url. But looks like the whole process is taking too long.
There is minimal processing of service responses so that doesnt take more than 100ms.
Here is the first multicast route:
from("direct:getCustomer")
.multicast().shareUnitOfWork().parallelProcessing().streaming()
.aggregationStrategy(txnAggregator)
.executorService(exeServiceForRest)
.to("direct:RouteA", "direct:RouteB", "direct:RouteC");
There are 8 routes where this msg will be multicasted.
2 of those routes are making another multicast call to another web service based on the response.
from("direct:routeA")
.process(orderReqBuilder)
.streamCaching()
.recipientList(header("url"))
.multicast().shareUnitOfWork().parallelProcessing().streaming()
.executorService(exeServiceForRest)
.aggregationStrategy(txnAggregator)
.to("direct:subRouteA", "direct:subRouteB");
The max response time of all 8 routes is 400ms. And the max response time of these subRoute web services is 300 ms
So the overall response time should be 400(level 1 service calls) + 300(level2 servicer calls) + some computation time(~100 ms) = 900 ms
but it is taking around 2 sec to complete.
Moreover, in the logs I found :
WMPLTFMLOG420420 1501782127187 2017-08-03 10:42:07.187 when the msg comes to header builder class
WMPLTFMLOG420420 1501782127362 2017-08-03 10:42:07.362 applog.cls=org.apache.camel.component.http4.HttpComponent,applog.mthd=createConnectionManager,applog.line=322,applog.msg=Created ClientConnectionManager org.apache.http.impl.conn.PoolingHttpClientConnectionManager#77809ca2
So there is almost 200ms spent in creating the url and sending the request, I guess. Why is it happening?
Overall, how can I improve the performance in this case. I test this on 4 core machine.
Camel version 2.18.3 and I am using http4 component.
I am using Camel as an Orchestration Engine.
clients sends HTTP request <-> CAMEL code <---- HTTP Req----- > external
server(s)
I am using HTTP4 Component (with default settings) for making HTTP Requests
to external server. I have quite a few http backends.
Right now the way we are making http calls to our backend is as follow:-
// The producer is created during app initialisation. This is actually done
via blueprint.xml
ProducerTemplate producer = camelContext.createProducerTemplate();
// Whenever I need to make a http call I am executing the below code with
URL set as something like:- "http4://order-api:8099/orders/v1/ordersearch/"
Exchange exchange = producer.request(URL, new Processor() {
#Override
public void process(Exchange exchange) throws Exception {
log.info("Executing the HTTP request : URL - " + URL + " Headers -
" + headers + " Body : " + body);
exchange.getIn().setHeaders(headers);
exchange.getIn().setBody(body);
}
});
The query I am having is:-
Does HTTP4 in the default setting camel uses some http connection
pooling while making a call to the external servers?
If yes Is there a way I can configure the connection pooling from
blueprint.xml?
I am using Camel 2.16.1 and the application is deployed in Karaf 3.0.5.
The http4 component use Apache HttpClient, which support pooling with the use of a HttpClientConnectionManager.
By default, camel uses a PoolingHttpClientConnectionManager which is configured with the properties connectionsPerRoute and maxTotalConnections.
If you want to have more control over this clientConnectionManager, you can provide your own implementation of org.apache.http.conn.HttpClientConnectionManager
See HttpClient connection manager
I have 1,000,000 IDs, and need to call a REST API once per day for each user. How can I use Camel to send batch HTTP requests?
My current solution is to create an HTTP component for every ID, using Quartz. But It will create 1,000,000 Quartz jobs, the performance is bad.
I have two ideas:
If Camel can provide a batch API, I could use that to send 1,000,000
requests to the same URL but with different parameters. That is one
Quartz job. Does Camel support that API?
Every HTTP request only visit one time, and use spring schedule to
control. But for each request Camel will poll the server. How can I
make Camel only call the REST API once?
The codes here:
// ids is a List, contains 1,000,000 id
for (final String id : ids) {
context.addRoutes(new RouteBuilder() {
#Override
public void configure() throws Exception {
from(String.format("quartz://%s/?cron=0+0+12+*+*+?", id))
.to(String.format("http:xxx.com?id=%s", id))
.to("file:to");
}
});
}
It could more heave to create million routes for sending the request with different id.
with the help of bean, you can define the loop of sending the request in the bean method.
BTW, you can change the id with message header "CamelHttpQuery".
I'm trying to consume a webservice from camel using the cxf component, like this:
<cxf:cxfEndpoint id="webservice"
address="http://webservice.url.com/webservice"
serviceClass="com.url.webservice.MyWebService"/>
<camel:camelContext>
<camel:route>
<camel:from uri="direct:a"/>
<camel:inOnly uri="cxf:bean:webservice?defaultOperationName=sendMessage"/>
</camel:route>
</camel:camelContext>
The sendMessage method has no response, hence the inOnly rather than to (although I have the same problem when I try to instead). The problem is that apparently camel still expects a response, and the route hangs while waiting for one. I suppose if I let it try long enough, it would eventually time out.
To be clear, I'm running a test method:
/* ... */
#Produce(uri = "direct:a")
protected ProducerTemplate directA;
#Test
public void sendMessage() throws Exception {
directA.sendBody(new String[] {"client id", "message"});
directB.sendBody(new String[] {"client id", "message 2"});
}
And I'm seeing the effect of the first call (that is, the message arrives at the server), but not the second, and the method doesn't finish running (again, I'm assuming it'll timeout at some point... if so, the timeout's pretty long: I ran the test as I started writing this, and it's still running).
Am I missing something? Is it a bug? Is it just bad practice to have webservice methods with no response?
By the way, when testing methods which have a response, it works fine.
I think Willem Jiang recently fixed some issue with one-way CXF in Camel recently. Maybe try Camel 2.6-SNAPSHOT from trunk.
I personally prefer two-way with web services, just returning some ACK back in case there is no data to return back. The client most often want some confirmation the server has received and acknowledged the data.
camel-cxf producer decides if it will handle the response message by checking if the operation is oneway. Even the sendMessage operation is oneway invocation, your server should send Http states 202 message to the client according the HTTP protocol, otherwise camel-cxf producer will still wait for the response.
BTW, I think the issue[1] that Claus said was related to async invocation of camel-cxf producer with one way message, not sure if it's the same issue as you mentioned(I didn't know which version of camel are you using).
[1]https://issues.apache.org/jira/browse/CAMEL-3426
Willem
I'd like to start multiple HTTP requests rapidly after each other, without having to wait on the previous response to arrive. I've tried this using WebClient.UploadStringAsync, but this doesn't work. I'd like to efficiently implement the following scenario:
Send request 1
Send request 2
Send request 3
And in another thread:
Receive response 1
Receive response 2
Receive response 3
Can this be done in Silverlight?
I'd like to start multiple HTTP requests rapidly after each other, without having to wait on the previous response to arrive
That's called HTTP Pipelining (assuming you hope to use the same socket) and it's not supported by many proxies and gateway devices. I would be surprised if Silverlight tried to support it.
Yes it can be done. What leads you to believe that UploadStringAsync isn't working?
Here is my guess you are posting to ASP.NET with Sessions turned on (the default) right?
The requests will be queued at the server end because ASP.NET will only process one request for a specific Session at a time.