I have an issue consuming a Chunked HTTP RSVP Stream using Camel.
The stream is here and you can find more information about it at the end of this page
I have created a simple route such:
from("http4://stream.meetup.com/2/rsvps").log(org.apache.camel.LoggingLevel.INFO, "MeetupElasticSearch.Log", "JSON RSVP ${body}")
But nothing happens no message are consumed. I tried by adding a camel timer before because I am not sure you can use http4 component directly in the from but result is the same.
Can you help me please?
You cannot use the http4 component in a from() directive.
However, there is several ways to call an URL and to retrieve the result:
One is to create a timer and call the http4 component from a to() like this:
from("quartz2://HttpPoll/myTimer?trigger.repeatCount=0")
.to("http4://stream.meetup.com/2/rsvps")
.log("Body is: ${body}")
.to(mockDestination);
Another way is to use the content enricher pattern if you want to for example aggegates results into one another stucture.
Note as well, if your URL is dynamic you must use recipientList() instead to().
UPDATE:
Another way to consume a stream from the Internet is to use another component than http4: the camel-stream.
This one you can declare it in a from()directive:
// Read stream from the given URL... an publish in a queue
from("stream:url?url=http://stream.meetup.com/2/rsvps&scanStream=true&retry=true").
to("seda:processingQueue");
// Read records from the processing queue
// and do something with them.
from("seda:processingQueue").to("log:out")
You can see more sample in this site.
Related
I want to write a camel route which will take input from multiple file destination and process them after aggregating.
is it possible to take input from multiple files for a single route?
Yes you can use poll-enrich to call consumer-endpoints like file to enrich the message. This works for many other consumer-endpoints as well like SFTP or message queues.
If you need to read same file multiple times it can get trickier as you'll likely have to set noop=true and possibly use something like dummy idempotent repository to get around camels default behavior.
Note that calling pollEnrich seems to clear headers / create new message so use exchange properties to persist data between pollEnrich calls.
from("file:someDirectory")
.setProperty("file1").body()
.pollEnrich("file:otherDirectory", 3000)
.setProperty("file2").body()
.pollEnrich("file:yetAnotherDirectory", 3000)
.setProperty("file3").body();
I have this simple route in my RouteBuilder.
from("amq:MyQueue").routeId(routeId).log(LoggingLevel.DEBUG, "Log: ${in.headers} - ${in.body}")
As stated in the doc for HTTP-component:
Camel will store the HTTP response from the external server on the OUT body. All headers from the IN message will be copied to the OUT message, ...
I would like to know if this concept also applies to amq-component, routeId, and log? Is it the default behaviour, that IN always gets copied to OUT?
Thank you,
Hadi
First of all: The concept of IN and OUT messages is deprecated in Camel 3.x.
This is mentioned in the Camel 3 migration guide and also annotated on the getOut method of the Camel Exchange.
However, it is not (yet) removed, but what you can take from it: don't care about the OUT message. Use the getMessage method and don't use getIn and getOut anymore.
To answer your question:
Yes, most components behave like this
Every step in the route takes the (IN) message and processes it
The body is typically overwritten with the new processing result
The headers typically stay, new headers can be added
So while the Camel Exchange traverses the route, typically the body is continuously updated and the header list grows.
However, some components like aggregator create new messages based on an AggregationStrategy. In such cases nothing is copied automatically and you have to implement the strategy to your needs.
Currently I'm receiving messages over tcp in a self written component that allows me to use Netty as a TCP server producer.
the messages i receive are formatted in XML style, for example:
<customheader>
<someattribute></someattribute>
</customheader>
<custombody>
</custombody>
The messages I receive are stored in a byte[] and to send it to another endpoint I create a new exchange via:
Exchange exchange = new DefaultExchange(endpoint);
exchange.getContext().createProducerTemplate().sendBody("someendpointuri", receivedbytes);
now my questions:
Is my approach with the new exchange correct?
If want to for example get rid of the header or use other camel components, do I need to convert the receivedbytes from byte[] to a different datatype or is byte[] okay?
If i want to remove the custom header can i use the remove header component from camel?
Thanks for your help
You're creating an new Exchange, but you're not actually using it. Instead, you only use it to access CamelContext. The method sendBody is creating a new Exchange for you and it is this Exchange that is actually being sent to endpoint specified by someendpointuri. Note that you should not create a new producer template every time you want to send a message.
When you say that you store messages as byte[], I assume you're storing it inside Message body. In this case you store both customheaders and custombody as byte[] and Camel treats them both as Message body, not headers.
If you want to use Camel header-related components or language constructs, you need to parse your customheaders and then set them on the Message as headers by using Exchange.getIn().setHeaders() (see API with a note about this). If you do that, you would probably only want to set content of custombody in the Exchange.getIn().setBody().
If you do these changes in your custom component, your component will now be handling this specific XML format only. If you want to keep your component generic, you can instead implement a custom DataFormat and call marshal() and unmarshal() in your routes. I think SOAP DataFormat does something very similar.
I have 2 routes. The first route uses poll enrich to check if a file is present. The second route uses a poll enrich on the same uri to read and process the file. The first route invokes the second via a SEDA queue, like so:
public void configure() throws Exception {
String myFile = "file://myDir?fileName=MyFile.zip&delete=false&readLock=none";
from("direct:test")
.pollEnrich(myFile, 10000)
.to("seda:myQueue")
;
from("seda:myQueue")
.pollEnrich(myFile, 10000)
.log("Do something with the body")
;
}
As it stands, if I execute the first route, the poll enrich finds a file, but when the poll enrich in the second route executes, it returns a body of null. If I just execute the second route on its own, it retrieves the file correctly.
Why does the second poll enrich return null, is the file locked? (I was hoping using a combination of noop,readLock, and delete=false would prevent any locking)
Does camel consider the second poll enrich as a duplicate, therefore filtering it out? (I have tried implementing my own IdempotentRepository to return false on contains(), but the second pollEnrich still returns null)
You may wonder why I'm trying to enrich from 2 routes, the first route has to check if a number of files exist, only when all files are present (i.e., pollEnrich doesn't return null) can the second route start processing them.
Is there an alternative to pollEnrich that I can use? I'm thinking that perhaps I'll need to create a bean that retrieves a file by URI and returns it as the body.
I'm using camel 2.11.0
I realize this is now an old topic, but I just had a similar problem.
I suggest you try the options:
noop=true
which you already have, and
idempotent=false
To tell Camel it is OK to process the same file twice.
Update after testing:
I actually tested this with both settings as suggested above, it works some times, but under moderate load, it fails, i.e. returns null body for some exchanges, although not all.
The documentation indicates that setting noop=true automatically sets idempotent=true, so I am not sure the idempotent setting is being honoured in this case.
Is there any specific reason why you are not using just one route?
I don't understand why you are using two routes for this. File component can check if the file is there and if it is, pull it. If you are worried about remembering the files so you don't get duplicates, you can use an idempotent repository. At least, based on your question, I don't think you need to complicate the logic using two routes and the content enricher EIP.
the second route returns NULL because the file was already consumed in the first route...if you are just looking for a signal message when all files are present, then use a file consumer along with an aggregator and possibly a claim check to avoid carrying around large payloads in memory, etc...
As you've probably learned this does not work as one might expect
noop=true&idempotent=false
my guess is that Camel ignores idempotent=false and as documented uses instance of MemoryMessageIdRepository. To work around this, one can configure file endpoint to use custom idempotent repo:
noop=true&idempotentRepository=#myRepo
and register custom repository in the registry or spring context:
#Component("myRepo")
public class MyRepo implements IdempotentRepository {
#Override
public boolean contains(Object o) {
return false;
}
...
}
Try pollEnrich with strategyMethodAllowNull="true". By default , this value is false. When it is false, the aggregation strategy looks for the existing Exchange body, to aggregate the content returned from file.
When we make strategyMethodAllowNull="true", the existing body is considered as null. So every time , the content of the file is set into the current exchange body
What I want to be able to do is following:
from(...)
.replyWith()
.from(...)
.end()
So that the response to my producer is taken from the consumption of another endpoint, an example would be something like a REST endpoint for a queue.
Is there an idiomatic way in Camel to be able to do something like the above without grabbing an Endpoint from a CamelContext instance and manually retrieving the contents and setting them into the Exchange?
Can you explain a bit more?
You dont want to just do
from A
to B
from B
to C
So that a message send to A will be send to B. And B is processed in another route, and the response from this will be send back to the first route, which will be used as reply to any client invoking A in the first place.
Also if you want something with dynamic endpoints, then you can use the Recipient List EIP pattern
http://camel.apache.org/recipient-list.html
from("http://0.0.0.0:9001/getResultsFromQueue")
.pollEnrich("activemq:queue:myQueue")
.to("log:test?level=DEBUG");