I have 1,000,000 IDs, and need to call a REST API once per day for each user. How can I use Camel to send batch HTTP requests?
My current solution is to create an HTTP component for every ID, using Quartz. But It will create 1,000,000 Quartz jobs, the performance is bad.
I have two ideas:
If Camel can provide a batch API, I could use that to send 1,000,000
requests to the same URL but with different parameters. That is one
Quartz job. Does Camel support that API?
Every HTTP request only visit one time, and use spring schedule to
control. But for each request Camel will poll the server. How can I
make Camel only call the REST API once?
The codes here:
// ids is a List, contains 1,000,000 id
for (final String id : ids) {
context.addRoutes(new RouteBuilder() {
#Override
public void configure() throws Exception {
from(String.format("quartz://%s/?cron=0+0+12+*+*+?", id))
.to(String.format("http:xxx.com?id=%s", id))
.to("file:to");
}
});
}
It could more heave to create million routes for sending the request with different id.
with the help of bean, you can define the loop of sending the request in the bean method.
BTW, you can change the id with message header "CamelHttpQuery".
Related
I am using camel 2.18.0 version.I am facing a issue while using restlet component in SEDA flow. Please find the route details below
from("timer://foo?repeatCount=1")
.to("http4://localhost:8080/pages")
.split(body())
.to("seda:pageConsumer");
from("seda:pageConsumer?concurrentConsumer=5")
.toD("restlet:${in.body}")
.process(enrich());
Details on route:
"http4://localhost:8080/pages" this rest endpoint returns list of url
url= http://localhost:8080/data?page=1&size=5
seda consumer is used for consuming each page parallel
url "http://localhost:8080/data?page=1&size=5" streams the data (This is a rest endpoint and endpont does not send a list of data instead endpoint streams the data)
restlet endpoint invokes the page url and streams the data.
Issue:
when seda endpoint receives 1st URL all is good i.e. "http://localhost:8080/data?page=1&size=5" rest endpoint is invoked by restlet and expected data is processed in route.
when seda endpoint receives 2nd URL i.e.
"http://localhost:8080/data?page=2&size=5" , this where issue starts instead of invoking rest endpoint with "page=2&size=5" as query param, restlet use query param from 1st URL i.e. "page=1&size=5". and issue continues for rest of the urls.
upon debugging i found out that camel caches producer in ProducerCache
and producer is cached in a hashmap with endpoint uri as the key.
questions why does camel does not honour query param during caching? is there why to avoid the caching?
Please note i got the code working by changing the url to include page details which is ugly. Currently url looks like
http://localhost:8080/data/page/2/size/5.
If the restlet endpoint is calling the same http url (eg host:port) then it can be slightly better to use a plain to with this host:port as static, and then set a header with the dynamic part.
.setHeader(Exchange.HTTP_QUERY, constant("page1=&size=5"))
.to("http:xxxx")
Then the same endpoint/producer is reused and the header then includes the dynamic part with the query parameters.
There is also a header where you can set the context-path as well on Exchange.
If you keep using toD then you can adjust the size of the cache with the cacheSize option - its the same as on recipient list: http://camel.apache.org/recipient-list.html
I'm using the Camel's ProducerTemplate to make http calls like below.
#Produce
private ProducerTemplate producer;
producer.requestBodyAndHeaders(url, requestString, headers,
String.class);
In my application, I'm using this to call another time taking rest api multiple times for a batch job. But from the access logs of the other server, I found that the server is getting only 2 requests at a time.
Once, it processes these two, it was getting another 2 requests.
Looks like something to do with the connection pooling of Camel's producerTemplate. Is it defaulting to 2 connections?
How can I change this configuration?
I have an Apache camel application which talks with a web service. This is purely a integration(mediation) application. We take REST request and transform it to a SOAP message (using VM templates) and call the web service. Receive response from web service, transform it to JSON and send back to our client.
REST ---->transform to SOAP req (Velocity template) ---->call WS ---->receive response---->transform into JSON---->return response to caller.
We are using servlet endpoint to receive request from our client. We can obtain HttpSession object from exchange before calling web service as follows :
HttpServletRequest req = exchange.getIn().getBody(HttpServletRequest.class);
HttpSession session = req.getSession();
However, the problem is that I cannot obtain HTTPSession from exchange after receiving response from web service. If I check the Exchange object in debug mode, the Exchange.getIn() is of type DefaultMessage rather than HttpMessage. That is the reason I think I loose Request and response objects.
I tried setting the exchange pattern to InOut but that doesn’t help.
The only solution I could find is to store the original body of the in message in a header or a property and retrieve it at the end. But I think there must be a better solution that this.
Could anybody help please?
Note: We need HttpSession so that we can store corresponding session information like session id created on WS for the session created by our request. We cannot pass session information created on WS to our callers, and need a place on our application to hold this mapping info. HttpSession serves this requirement. Is there any better way?
You can store the http session as an exchange property, camel copy these properties across the exchanges, so you can access it in the route when you need.
I have to create a Fuse service which would in-turn invoke a REST service exposed by an external service provider. Fuse service will be receiving request in XML format and converting to a query string before invoking the REST service.
Sample request XML for Fuse service -
<CustomerDetails>
<CustomerName>ABC</CustomerName>
<CustomerAge>28</CustomerAge>
<CustomerName>DEF</CustomerName>
<CustomerAge>54</CustomerAge>
<CustomerDetails>
The REST service consumes request in key value params and responds back in XML format.
sample URL:
https://www.customer.com/cust/api/v1/store/abc.xml?Customername=ABC&Customerage=28&Customername=DEF&customerage=54)
I have tried searching a lot but couldn't find any tutorial in the net.
Can someone please provide suggestions on how to implement the fuse service using cxf-rs components (preferably Spring DSL camel configuration )
Thanks in advance..
If you just want to turn the XML request to the url parameter, you can just use jaxb data format to unmarshal the request and use a bean object to setup the URI request parameters. You don't need to use camel-cxf component.
from("direct:start").unmarshal(jaxb).process(new Processor() {
public void process(Exchange exchange) throws Exception {
// get the request object
CustomerDetail request = exchange.getIn().getBody();
// Just mapping the request object into a query parameters.
String query = requestToParameter(request);
exchange.getIn().setHeader(Exchange.HTTP_QUERY, query);
// to remove the body, so http endpoint can send the request with Get Method
exchange.getIn().setBody(null);
}).to("https://www.customer.com/cust/api/v1/store/abc.xml");
I'm having difficulty understanding some aspects of Google AppEngine Push Task Queue for java, namely, when a task is executed, where does the response go?
If I add something to the queue like this:
Queue queue = QueueFactory.getDefaultQueue();
queue.add(withUrl("/worker").param("key", key));
Then sometime later the task is executed, with something like this:
public class SomeServlet extends HttpServlet {
#Override
protected void doGet(HttpServletRequest req, HttpServletResponse resp)
throws ServletException, IOException {
...
resp.getWriter().println("something"); //-- where does this response go???
}
}
Sorry for the newbie question, but where does this response go? My app didn't set up a normal http request so it's not waiting for a normal http response. I must be missing something fundamental about the task queue.
The response doesn't go anywhere. In a task you would typically write to the datastore, add new tasks to task queue, send xmpp, call external URLs.
If the response code is 200 OK, the task is removed from the queue, if it's an error code, it is retried (depending on retry settings).
There is no document indicated where GAE stored these responses. I would not surprise GAE just
don't care the response after it trigger the request url successfully.
By the way, I don't see a point to care about the response.
If you want to log what happen during the task execution, you should use Log instead.
https://developers.google.com/appengine/docs/java/runtime#Logging