The need is to have a Camel (Mina/Netty) based TCP server running on a specific port, which allows multiple TCP clients to connect to it. The content to be streamed is available in files and the TCP server has to send each line in the text files to one of the connected clients (round robin).
Can someone help me with the outline of a camel route to achieve this?
Also it is possible to throttle the streaming speed for example 100 msg/sec per connected client?
Thanks in advance.
MK
I have done something like this. Well not quite exactly to what you want but I can make some suggestions on how to get started.
Lets use the MINA component for this route as I had some issues with the Netty component see this link Exception thrown from Apache Camel Netty Consumer When more than one client sends data. Apparently this has been fixed but as my project got cancelled I never tested it again.
So from your description this will be a text line based protocol a simple route for this would look like this in the DSL
<route>
<from uri="mina2:tcp://localhost:5555?textline=true"/>
<to uri="bean:fileProcessing"/>
</route>
This route will open a listening socket on the localhost on port 5555. The route is also setup to use a textline codec. A textline codec is essentially a line of text terminated end of line character i.e. \n. If you are going to be using some other protocol you will need to look into the following items:
ProtocolEncoder—The ProtocolEncoder handles the task of taking an input
payload and putting bytes onto the TCP channel.
ProtocolDecoder—The ProtocolDecoder interprets the custom
binary protocol message to something your application can understand.
ProtocolCodecFactory— This creates the encoder and decoder.
You could implement some logic in the fileProcessing bean to send the replies back. There is a catch however the clients will have to request when they are ready to get a new line. From my understanding this follows a request reply scenario. However from what I have seen the route is only active when there is a message coming in from the client to the server.
There might be a way to initiate the sending from the server but you will need to experiment with that by yourself on getting it done I have not done anything like that myself.
Critical reading would be the following.
Mina Documentation
Blog Entry On Using MINA
My suggestion is to start with a basic route like this and then expand your logic and then come back with problems you might encounter.
UPDATE:
So I did a little research on this and it does not appear possible to send events from the server to the client without following one of the following patterns InOnly and InOut.
Try just using MINA or Netty then.
Related
I want to listen on ActiveMQ topic based on the hostname of system and some other logic. I planned to use pollEnrich for it so I evaluate my logic and provide topic name in pollEnrich but as per document:
pollEnrich or enrich does not access any data from the current Exchange which means when polling it cannot use any of the existing headers you may have set on the Exchange. For example you cannot set a filename in the Exchange.FILE_NAME header and use pollEnrich to consume only that file. For that you must set the filename in the endpoint URI.
How i can figure out this
from("timer://ipc?repeatCount=1")
.. some logic..
.setHeader("topic_no",simple("{{env:HOSTNAME}}"))
.pollEnrich("mqtt:foo?host=tcp://0.0.0.0:1883&subscribeTopicNames=${header.topic_no}/status&clientId=ipc")
.to("log:my?showAll=true&multiline=true");
Please don't suggest to use hostname directly in URI. As I highlighted I have to compute other logic too.
What other option or way I can use?
Will pollEnrich kept listening on topic or it will listen once and end the route?
Update1:
I figured out we can use simple expression with for dynamic URI, But one issue with pollEnrich it only pick one message how i can make sure it kept on listening as consumer? I want that before pollEnrich part get execute once and TopicListener kept listening till application is up.
Will pollEnrich kept listening on topic or it will listen once and end the route?
Same as the fact you have figure out, Camel pollEnrich component will listen on topic and consume at most one message per call.
What other option or way I can use?
Repeat pollEnrich by loop
Create new route at run-time by routeBuilder
Option 1 is naive, but simple in concept. pollEnrich will do once and loop will repeat it. However, this method need to handle more scenario than you might expected.
Option 2 is a better approach. You create a route at run-time and the consumer endpoint URI is pass by variable. That said, you can create the consumer route dynamically after your computation logic.
Example for routeBuilder
For an IoT project I am working on I am researching a next, enhanced, version of our “Socket Handler” which is over 5 years of age and has evolved in a big beast doing apart from handling socket connections with IoT devices also in thread processing that has become a real pain to manage.
For my total rewrite I am looking into Apache Camel as a routing and transformation toolkit and understand how this can help us split processing steps into micro-services, loosely coupled through message queues.
One thing that I have trouble understanding however is how I can implement the following logic “the Apache Camel way”:
An IoT device sends an initial message which contains its id, some extra headers and a message payload.
Apart from extracting the message payload and routing it to a channel, I also need to use the device Id to check a message queue, named after the device id, for any commands that have to go to the device over the same socket connection that received the initial message.
Although it seems that Netty4, which is included in Camel, can deal with synchronous duplex comms, I cannot see how the above logic can be implemented in the Camel Netty4 component. Camel Routing seems to be one way only.
Is there a correct way to do this or should I forget about using camel for this and just use Netty4 bare?
It is possible to reproduce a full duplex communication by using two Camel routes :
thanks by using reuseChannel property
As this full duplex communication will be realised thanks to two different Camel routes, sync property must be set to false.
Here the first route:
from("netty4:tcp://{{tcpAddress}}:{{tcpPort}}?decoders=#length-decoder,#string-decoder&encoders=#length-encoder,#bytearray-encoder&sync=false&reuseChannel=true") .bean("myMessageService", "receiveFromTCP").to("jms:queue:<name>")
This first route will create a TCP/IP consumer thanks to a server socket (it is also possible to use a client socket thanks to property clientMode)
As we want to reuse the just created connection, it is important to initialize decoders but also encoders that will be used later thanks to a bean (see further). This bean will be responsible of sending data by using the Channel created in this first route (Netty Channel contains a pipeline for decoding / encoding messages before receiving from / sending to TCP/IP.
Now, we want to send back some data to the parter connected to the consumer (from) endpoint of the first route. Since we are not able to do it thanks to classical producer endpoint (to), we use a bean object :
from("jsm:queue:<name>").bean("myMessageService", "sendToTCP");
Here the bean code :
public class MessageService {
private Channel openedChannel;
public void sendToTCP(final Exchange exchange) {
// opened channel will use encoders before writing on the socket already
// created in the first route
openedChannel.writeAndFlush(exchange.getIn().getBody());
}
public void receiveFromTCP(final Exchange exchange) {
// record the channel created in the first route.
this.openedChannel = exchange.getProperty(NettyConstants.NETTY_CHANNEL, Channel.class);
}
}
Of course,
the same bean instance is used in the two routes, you need to use a registry to do so :
SimpleRegistry simpleRegistry = new SimpleRegistry();
simpleRegistry.put("myMessageService", new MessageService());
Since the bean is used in two different asynchronous routes, you will have to deal with some unexected situations, by for example protecting the access of openedChannel member, or dealing with unexpected deconnexion.
This post helped me a lot to find this solution :
how-to-send-a-response-back-over-a-established-tcp-connection-in-async-mode-usin
reuseChannel property documentation
After end of camel route, exchange's body and headers will back to requester as response.
I currently have a camel route that exposes a cxf endpoint. When a messages comes through the endpoint I would first enrich that message with some information from another webservice and then do more processing afterwards. However, I want make the first half of this route synchronous so I can send back a response to whomever called my exposed cxf endpoint.
The route looks something like this:
from(cxf:CxfEndpoint)
.process(someProcessing)
.to(cxf:ExternalCxfEndpoint)
.to(activemq:queue:somequeue)
//return a response back to caller here
from(activemq:queue:somequeue)
... //additional processing here
...
The reason for this is because when a message comes via my exposed cxf endpoint I don't know if it's a valid message. I need to first validate it with the message enrichment. Once the message is enriched, I want let whomever sent the message know that their message is accepted but don't want them to wait for the message to make it through the whole route as that could take hours.
Does anyone know how this would work?
Thanks in advance!
I believe all you need to do is set exchangePattern to InOnly a.k.a. make it an Event Message. This should have your route not wait for a reply from ActiveMQ. Camel exchange will default to InOut when it's originating from a web service, as in your case.
A related question with an answer from a Camel dev here.
Also see this one for some details on the behavior when your broker is down.
Yes definitely , 100% possible. A simple example would be this :
From cxf endpoint
Store your request in a camel property or header
To xslt - generate xslt for cxf endpoint - Synchronous flow
Reset your original payload using set body.
Wiretap Endpoint - to any endpoint downstream or even a route , this becomes asynchronous . This won't take part in the above sync response .
Note- step 2 & 4 may not be required, it depends on your use case .
There are whole lots of things you can do, I just gave a very simple example . It doesn't need to be wiretap as well, but wiretap helps us not to write any additional custom exceptional handling.
Does Camel provide anything out of the box which tells if it is able to connect all endpoints?
These endpoints could be MQ, webservice etc.
If not then I have to write a servlet which will send test request to all the endpoints. I will be using multicast or splitter for this implementation.
From my experience Camel will only provide warning logs if a from() endpoint is not available since it is constantly trying to read from them. Every other endpoint won't be accessed until the exchange tries to use that endpoint. If your goal is to test if various resources are alive I believe you would need to create your own testing program. I don't think this will be implemented as a feature because typically applications build in error handling if a resource is down and definte appropriate behaviors.
If we're talking about producers, then no. If your route is sending messages to an amq or http4 endpoint for instance, camel with not automatically send TCP-packets on these connections for monitoring purposes. A common way to handle failure of external endpoints is by using "circuit breakers". Take a look at https://camel.apache.org/load-balancer.html. A more robust alternative, imho, is Netflix's Hystrix.
If you have a polling consumer, say a from:ftp://.. then the polling consumer will poll messages every n-th millisecond, and you'll get an error if the connection is broken.
What's the best strategy to send SMS via SMPP with Camel ? Should I use the ProducerTemplate ?
I'm new to camel so I'm not confident if my strategy is the best.
In my application upon reception of an SMS, I have to send back an other SMS with some computed content.
I created a
route smsIn that looks like this
from "uri=smpp ..."
unmarshal ref="bindyDataFormat"
to "uri=bean:myBean
and a route smsOut with
from "uri=direct:smsOut"
to "uri=smpp ..."
The smsIn route, receives the sms, transforms its conent (csv data) in a pojo and send that pojo to myBean.
In myBean I do some processing and then call a ProducerTemplate which send my computed message to the endpoint "direct:smsOut".
The reason I use the producerTemplate is that I have to set some info from my pojo in the header (CamelSmppDestAddr) and the body of the Exchange.
I have tested with the logica SMSC simulator, this seems to work fine, but would like to have your opinion about this solution ?
What about reliability , transaction ?
Should I store my message before trying to send it to the SMSC ?
Should I store it in a database, post it to a queue ?
I'm not sure why you have a producer template, you could just build up the route instead (given that you return something from your bean or takes an Exchange as paramter).
<from uri="smpp: ..."/>
<bean ref="bean:myBean"/>
<to uri="jms:queue:myQueue"/>
then not use direct, but use a JMS queue that is transactional and persistent. Say your smpp call fails, the message would have been gone. Using a queue like this and make sure its transactional, you can make sure not to lose data in this stage of the route.
<from uri="jms:queue:myQueue"/>
<transactional/>
<to uri="smpp.."/>
I suggest using Apache ActiveMQ as JMS middleware. Actually, if you download ActiveMQ, you get camel bundled, so you could actually run your Camel routes from ActiveMQ.
You might want to tweak how retries and error handling occurs dependent on what you want to happen (retry every second forever?, retry five times, then put to error queue? etc).
Read this page: Transaction Error handling in Camel
For deeper info and more tweaks, you might also want to read this:
Transactional Client