I have created a camel (2.20.1) route using groovy DSL. I need to use multicast for 2 endpoints viz. ftps and file. If the order of routes is ftps and file then file is property written on ftp server but on file system empty file is written (with size 0 bytes). If i reverse the order i.e. file and then ftps then file is written on file system properly and empty file is written on ftp server.
It is working fine on Apache Mina FTP server, but with client ftp server it is working as mentioned above.
I have tried both multicast options:
.to("ftps:....").to("file:...")
as well as
.to("ftps:...").to("file:..."))
Also tried parallelProcessing(), but still the same result.
camelContext.addRoutes(new RouteBuilder() {
def void configure() {
from("file:///home/xyz/?fileName=file.txt&charset=utf-8&noop=true")
.multicast()
.to("ftps://localhost:21/files?username=anonymous&password=anonymous&binary=true&fileName=file.txt&passiveMode=true&fileExist=Fail")
.to("file://${directory}?fileName=\${file:name}-\${date:now:yyyyMMddHHmmssSSS}")
}
})
I am expecting that multicast should write the same content to both endpoints without data loss.
See Why is my message empty?. File consumer returns InputStream, which is consumed by first endpoint and thus is empty for the second endpoint. You need to enable Stream Caching, or convert body to some reusable object (e.g. String) before multicasting.
Enable stream caching:
from("file:///home/xyz/?fileName=file.txt&charset=utf-8&noop=true")
.streamCaching()
.multicast()
...
Convert body:
from("file:///home/xyz/?fileName=file.txt&charset=utf-8&noop=true")
.convertBodyTo(String.class)
.multicast()
...
Related
For an IoT project I am working on I am researching a next, enhanced, version of our “Socket Handler” which is over 5 years of age and has evolved in a big beast doing apart from handling socket connections with IoT devices also in thread processing that has become a real pain to manage.
For my total rewrite I am looking into Apache Camel as a routing and transformation toolkit and understand how this can help us split processing steps into micro-services, loosely coupled through message queues.
One thing that I have trouble understanding however is how I can implement the following logic “the Apache Camel way”:
An IoT device sends an initial message which contains its id, some extra headers and a message payload.
Apart from extracting the message payload and routing it to a channel, I also need to use the device Id to check a message queue, named after the device id, for any commands that have to go to the device over the same socket connection that received the initial message.
Although it seems that Netty4, which is included in Camel, can deal with synchronous duplex comms, I cannot see how the above logic can be implemented in the Camel Netty4 component. Camel Routing seems to be one way only.
Is there a correct way to do this or should I forget about using camel for this and just use Netty4 bare?
It is possible to reproduce a full duplex communication by using two Camel routes :
thanks by using reuseChannel property
As this full duplex communication will be realised thanks to two different Camel routes, sync property must be set to false.
Here the first route:
from("netty4:tcp://{{tcpAddress}}:{{tcpPort}}?decoders=#length-decoder,#string-decoder&encoders=#length-encoder,#bytearray-encoder&sync=false&reuseChannel=true") .bean("myMessageService", "receiveFromTCP").to("jms:queue:<name>")
This first route will create a TCP/IP consumer thanks to a server socket (it is also possible to use a client socket thanks to property clientMode)
As we want to reuse the just created connection, it is important to initialize decoders but also encoders that will be used later thanks to a bean (see further). This bean will be responsible of sending data by using the Channel created in this first route (Netty Channel contains a pipeline for decoding / encoding messages before receiving from / sending to TCP/IP.
Now, we want to send back some data to the parter connected to the consumer (from) endpoint of the first route. Since we are not able to do it thanks to classical producer endpoint (to), we use a bean object :
from("jsm:queue:<name>").bean("myMessageService", "sendToTCP");
Here the bean code :
public class MessageService {
private Channel openedChannel;
public void sendToTCP(final Exchange exchange) {
// opened channel will use encoders before writing on the socket already
// created in the first route
openedChannel.writeAndFlush(exchange.getIn().getBody());
}
public void receiveFromTCP(final Exchange exchange) {
// record the channel created in the first route.
this.openedChannel = exchange.getProperty(NettyConstants.NETTY_CHANNEL, Channel.class);
}
}
Of course,
the same bean instance is used in the two routes, you need to use a registry to do so :
SimpleRegistry simpleRegistry = new SimpleRegistry();
simpleRegistry.put("myMessageService", new MessageService());
Since the bean is used in two different asynchronous routes, you will have to deal with some unexected situations, by for example protecting the access of openedChannel member, or dealing with unexpected deconnexion.
This post helped me a lot to find this solution :
how-to-send-a-response-back-over-a-established-tcp-connection-in-async-mode-usin
reuseChannel property documentation
After end of camel route, exchange's body and headers will back to requester as response.
I need to somehow enable XML file transfer in the way that some machine, which generates a XML file sends the XML file through HTTP to another client. This other client would be based on C, receive the XML file and process it.
Is this possible in any way? I only found results for sending XML files to a HTTP server using some URL. I guess I'd have to implement my own HTTP server in my C application? Any ideas?
maybe you can use libcurl
here is a http POST example: http://curl.haxx.se/libcurl/c/http-post.html
on the receiving computer has to be a listening socket. here is a socket example: http://www.codeproject.com/Articles/586000/Networking-and-Socket-programming-tutorial-in-C
this socket takes the incoming data for later processing...
The need is to have a Camel (Mina/Netty) based TCP server running on a specific port, which allows multiple TCP clients to connect to it. The content to be streamed is available in files and the TCP server has to send each line in the text files to one of the connected clients (round robin).
Can someone help me with the outline of a camel route to achieve this?
Also it is possible to throttle the streaming speed for example 100 msg/sec per connected client?
Thanks in advance.
MK
I have done something like this. Well not quite exactly to what you want but I can make some suggestions on how to get started.
Lets use the MINA component for this route as I had some issues with the Netty component see this link Exception thrown from Apache Camel Netty Consumer When more than one client sends data. Apparently this has been fixed but as my project got cancelled I never tested it again.
So from your description this will be a text line based protocol a simple route for this would look like this in the DSL
<route>
<from uri="mina2:tcp://localhost:5555?textline=true"/>
<to uri="bean:fileProcessing"/>
</route>
This route will open a listening socket on the localhost on port 5555. The route is also setup to use a textline codec. A textline codec is essentially a line of text terminated end of line character i.e. \n. If you are going to be using some other protocol you will need to look into the following items:
ProtocolEncoder—The ProtocolEncoder handles the task of taking an input
payload and putting bytes onto the TCP channel.
ProtocolDecoder—The ProtocolDecoder interprets the custom
binary protocol message to something your application can understand.
ProtocolCodecFactory— This creates the encoder and decoder.
You could implement some logic in the fileProcessing bean to send the replies back. There is a catch however the clients will have to request when they are ready to get a new line. From my understanding this follows a request reply scenario. However from what I have seen the route is only active when there is a message coming in from the client to the server.
There might be a way to initiate the sending from the server but you will need to experiment with that by yourself on getting it done I have not done anything like that myself.
Critical reading would be the following.
Mina Documentation
Blog Entry On Using MINA
My suggestion is to start with a basic route like this and then expand your logic and then come back with problems you might encounter.
UPDATE:
So I did a little research on this and it does not appear possible to send events from the server to the client without following one of the following patterns InOnly and InOut.
Try just using MINA or Netty then.
I've defined the following camel route:
RouteBuilder rb = new RouteBuilder() {
#Override
public void configure() throws Exception {
from("sftp://myhost//path/to/files/")
.to("log:loggingCategory?level=INFO")
.to("file:///tmp/");
}
};
When I start the context with this route camel does connect and it downloads the files. My problem is that camel repeats downloading the same files until the context is shut down. Why does the FTP2 component do this and how can I stop it?
I've included version 2.10.4 of camel-core and camel-ftp via maven.
The Indempotent Consumer does the trick. Docs of the FTP2 component refer to the File2 component as "as all the options there also applies for this component". There is a parameter "indempotent=true" that activates usage of an LRUCache:
Option to use the Idempotent Consumer EIP pattern to let Camel skip
already processed files. Will by default use a memory based LRUCache
that holds 1000 entries. If noop=true then idempotent will be enabled
as well to avoid consuming the same files over and over again.
My complete source definition now looks like this:
from("sftp://myhost//path/to/files/?username=user&password=secret&idempotent=true")
From the camel ftp2 documentation:
The FTP consumer will by default leave the consumed files untouched on
the remote FTP server. You have to configure it explicitly if you want
it to delete the files or move them to another location. For example
you can use delete=true to delete the files, or use move=.done to move
the files into a hidden done sub directory.
To delete the file, change the route to
from("sftp://myhost//path/to/files?delete=true")
Ensure that the connected user has required permissions.
I have a method exposed in a MBean in server. Now, i have to call this method from the client and transfer a file to the server. How do i do it?
Regards,
Al Niyas
I'm not clear on what this remote method looks like, but what I would recommend for this operation is to have the JMX Server expose a method like:
void transferFile(String fileName, byte[] content)
Your local client can read the local file, read in all the bytes into an array and then pass the file name and the byte array content as arguments to the JMX Server method.