ChannelHandler handling content of unknown size - apache-camel

Question

I am still struggling with Camel (2.16.1) and Netty (4.0.33) to have them both receive tcp content of freely chosen length. Because of the unknown size of the tcp content received I was not yet able to create a working decoder for.
Let me describe my problem with an example. Lets say I have a file with a length of 3129 byte. When I nc that file to my route the size is not known until the last byte is read:
cat file.bin | nc localhost 10001
My route is defined like this:
from( "netty4:tcp://127.0.0.1:10001?sync=false&allowDefaultCodec=false&
decoder=#factory&receiveBufferSize=1000000")
.to("file:/temp/in");
The factory looks like this because I need to make sure that each ChannelHandler is used only once:
public class Factory implements ChannelHandlerFactory {
#Override
public ChannelHandler newChannelHandler() {
return new RawPrinterDecoder();
}
}
In my decoder I have this code:
public class RawPrinterDecoder extends ReplayingDecoder<Void> {
#Override
protected void decode(ChannelHandlerContext ctx, ByteBuf in,
List<Object> out) throws Exception {
while (in.isReadable()) {
byte readByte = in.readByte();
job.addContent(readByte);
}
in.discardReadBytes();
}
public void channelInactive(ChannelHandlerContext ctx) throws Exception {
System.out.println("Bytes in job: " + job.getSize() );
}
}
The problem with this is that instead of 3129 byte I receive 9273. The reason for this is that the file is split into 3 segments of 1024 byte and 1 with 57 byte. Those are passed repeatedly to my decoder and although I try to invalidate the segments after they are first processed with in.discardReadBytes() they are processed again so instead of ...
segment1
segment2
segment3
segment4
... my decoder sees them like this
segment1
segment1+segment2
segment1+segment2+segment3
segment1+segment2+segment3+segment4
I tried so solve my problem by using checkpoint() but the segments were still called repeatedly.
How can I make sure that each segment is only processed once and in the correct order ? If this can be done more efficiently instead of reading single bytes recommendations are welcome (readableBytes() always return 2 GB so I can not use this to get the number of bytes).

Answer 1

It gets seperated into segments because of the ByteBuffAllocator that your server is using. You can change that this way:
#Bean
public ChannelInitializer<SocketChannel> channelInitializer() {
return new ChannelInitializer<SocketChannel>() {
#Override
public void initChannel(SocketChannel ch) throws Exception {
ch.config().setRecvByteBufAllocator(new FixedRecvByteBufAllocator(2048));
}
};
}
You can read all of the available bytes at once using:
ByteBuf buffer = in.readBytes(in);
or
ByteBuf buffer = in.readSlice(in.readableBytes());