I have a question about communication between a client and a server. I would like to send data over a TCP unix socket (I know how to do this), and I don't know what is the best practises to test if the message sent is ready to be read entirely (not block per block).
Thus, I'm thinking of this
The client send the data printf(3) formatted, the message is written in a string and sent.
The server receive the message, but how to be sure the message if full ? Do I need to loop until the message is complete ?
So my idea is to use a code (or checksum maybe ?) that will be prepended and appended to the message like this :
[verification code] my_long_data_formatted [verification code]
And then, the server tries to read the data until the second verification code is read and succesfully checked.
Is this a proper solution for client / server communication ? If yes, what do you advise me for the verfications boundaries ?
TCP already has a built-in checksum/verification. So if the message was received, it was received correctly.
Typically then, the only thing you have to worry about is figuring out how long the message is. This is done either by sending the message length at the beginning or by putting a termination character or sequence at the end.
To make sure both the sender and receiver are "on the same page" so to speak, the receiver will typically send back a response after the message was received, even if that response only says "OK".
Examples of this technique include HTTP, SMTP, POP3, IMAP, and many others.
There are several ways to do this, so I don't think there is a "proper" solution. Your solution will probably work. The one note of caution is that you need to make sure the valiation code you chose is not sent as part of the data in the message. If that is the case, you will detect that the message is complete, even though it is really not the end of the message. If that is not possible to know what the data will look like, you may need to try a different technique.
From your description, it sounds like your messages are variable length. Another way would be to make all the messages the same length, that way you know how much data is to be read each time to get a complete message.
Another way would be to send over the length of the message first (such as a binary 32 bit number) which indicates the number of bytes to be read until the end of the message. You read this first to get the amount of data, and then read that amount from the socket.
If you had a set number of messages where the length was the same each time, you could assign a number to each message and send that number first, which you could then read. With that information you can determine how much data is to be read based on the assigned number to the message.
What you select to use for a solution will probably be based on factors like if messages are varaible or fixed in length and/or do you need to send additional information with the data. In this case, you might have a mixture where you send a fixed length header, which contains the information about the data which follows; either in length or the type of data which follows.
You need to establish an application-level protocol that would tell you somehow where application messages start and end in the byte stream provided to you by TCP (and then maybe how connected parties proceed with the conversation).
Popular choices are:
Fixed-length messages. Works well with binary data and is very simple.
Variable-size messages with fixed format or size header that tells exact size (and maybe type) of the rest of the message. Works with both binary and text data.
Delimited messages - some character(s) like new-line or \x1 are special and denote message boundaries. Best with text data.
Self-describing messages like XML, or S-expressions, or ASN.1.
Related
What I want to achieve:
I am coding a Java program that uses IMAP to connect to some gmail accounts every 5 minutes and extract information from some messages.
I want to check all the messages (incoming and outgoing) and take only the ones I have not processed. By "processed" I do not mean only "read" or "seen" messages. My application does not care whether or not another user has accessed that account and read a message. My application needs to keep track of which was the last message it processed and, the next time it goes through the messages, start with the first non-processed message.
I do not want to change anything in the messages. I do not want to mark them as seen or read.
What I have implemented:
Establish IMAP connection.
Open and access all messages in "[Gmail]/All Mail" folder.
What I have tried:
I have been reading about UID and message number, but I am not sure if any of them could help me achieve what I want. Maybe UID could, but: how do I retrieve it with JavaMail?
I found Folder.getMessages(int start, int end), but I think it refers to the index of the message in a folder, which I believe can easily change.
Can anyone provide some guidance at what is the best approach to take here?
Thanks!
IMAP UIDs are relative to the folder containing the message. I don't know how Gmail handles UIDs for messages in the "[Gmail]/All Mail" folder, but if it does the right thing you could use the UIDFolder interface to get the UIDs. And as described, once you've processed a certain UID, all the new messages will have larger UIDs, which can make processing more efficient.
The alternative is to use Message-IDs, which has a different set of problems...
I have a scenario where I get as input Message A. Message A must then be split into 3 different types of message, and forwarded to other routes. It is important that the messages arrive in a precise order, Ie. A-1 must be sent before A-2, which must be sent before A-3.
To do this I have done the following (outline):
from("activemq:queue:somequeue-local")
.multicast().to("direct:a1","direct:a2","direct:a3");
from("direct:a1)
//split incoming message and prepare output document for A-1
.to("activemq:queue:otherqueue")
.from("direct:a2)
//split incoming message and prepare output document for A-2
.to("activemq:queue:otherqueue")
.from("direct:a3)
//split incoming message and prepare output document for A-3
.to("activemq:queue:otherqueue")
And in another context, responsible for sending out the info to the external system, I have
.from("activemq:queue:otherqueue?maxMessagesPerTask=1&concurrentConsumers=1&maxConcurrentConsumers=1")
// do different stuff based on which type we are called with then end with
.beanref("somebean","writeToFileAndCallImportbat");
Now, my problem is, that when I get to the receiver, I get the messages in random order. Sometimes A-1,A-3,A-2, sometimes right, A-1,A-2,A-3.
I have tried adding JMSXGroupID and JMSXGroupSeq to the messages, but without any luck.
I have also tried skipping the MQ part entirely, and use direct-vm: to call the shared receiver, but then it looks like I have three simultanious invocations of the receiver at once, and still in random execution order.
I was under the impression that multicast would run sequential, unless otherwise prompted to?
Is there something fundamentally wrong with the approach taken?
I am using Camel version 2.12.
Or, said more plainly:
I would like a route that creates three different output messages, and executes a batch file on them, in order. How do I go about that?
If you use the Splitter pattern, have you checked to see if the streaming property is set to false.
If enabled then Camel will split in a streaming fashion, which means it will split the input message in chunks. This reduces the memory overhead. For example if you split big messages its recommended to enable streaming. If streaming is enabled then the sub-message replies will be aggregated out-of-order, eg in the order they come back. If disabled, Camel will process sub-message replies in the same order as they where splitted.
So, it turned out to not be a problem with multicast after all.
Rather, in each of my sub-routes, I did this:
.split(..stax(SpecialClass)).streaming()
.beanRef("transformationBean","somefunction")
.aggregate(constant("1"), new MyAggregator())
.completionTimeout(5000)
.completionSize(1000)
.to(writeToFileAndRunBat)
Which, I assumed meant "Process all elements in the split, and if you aren't finished in 5 seconds or after 1000 elements, break out".
I changed it to
.split(..stax(SpecialClass), , new MyAggregator()).streaming()
.beanRef("transformationBean","somefunction")
.end()
.to(writeToFileAndRunBat)
Coming to think of it, it makes perfect sense, as the first version couldn't really know when we were done, while the last (I assume) just iterate over all elements in the split and calls the Aggregator for each.
Also, I had to .end() in the first version. So I guess the whole thing was just acting random.
I'd like to get raw messages for a particilar thread via "Users.threads: get" request. Is it possible to use format=raw parameter in this request somehow? I know it is possible for message request but I really need to do that for this one.
It might be useful to specify the format (raw or minimal) when requesting a thread, unfortunately this is not currently possible. The format can only be set for a message request. Requesting a thread in raw format however would not be a good idea since a thread could contain many large messages.
It is possible to set the format, but for some reason google doesn't allow the raw format when getting threads.
See the docs here
everyone, I am porting the WinPcap from NDIS6 protocol to NDIS6 filter. It is nearly finished, but I still have a bit of questions:
The comment of ndislwf said "A filter that doesn't provide a FilerSendNetBufferList handler can not originate a send on its own." Does it mean if I used the NdisFSendNetBufferLists function, I have to provide the FilerSendNetBufferList handler? My driver will send self-constructed packets by NdisFSendNetBufferLists, but I don't want to filter other programs' sent packets.
The same as the FilterReturnNetBufferLists, it said "A filter that doesn't provide a FilterReturnNetBufferLists handler cannot originate a receive indication on its own.". What does "originate a receive indication" mean? NdisFIndicateReceiveNetBufferLists or NdisFReturnNetBufferLists or both? Also, for my driver, I only want to capture received packets instead of the returned packets. So if possible, I don't want to provide the FilterReturnNetBufferLists function for performance purpose.
Another ressembled case is FilterOidRequestComplete and NdisFOidRequest, in fact my filter driver only want to send Oid requests itself by NdisFOidRequest instead of filtering Oid requests sent by others. Can I leave the FilterOidRequest, FilterCancelOidRequest and FilterOidRequestComplete to NULL? Or which one is a must to use NdisFOidRequest?
Thx.
Send and Receive
A LWF can either be:
completely excluded from the send path, unable to see other protocols' send traffic, and unable to send any of its own traffic; or
integrated into the send path, able to see and filter other protocols' send and send-complete traffic, and able to inject its own traffic
It's an all-or-nothing model. Since you want to send your own self-constructed packets, you must install a FilterSendNetBufferLists handler and a FilterSendNetBufferListsComplete handler. If you're not interested in other protocols' traffic, then your send handler can be as simple as the sample's send handler — just dump everything into NdisFSendNetBufferLists without looking at it.
The FilterSendNetBufferListsComplete handler needs to be a little more careful. Iterate over all the completed NBLs and pick out the ones that you sent. You can identify the packets you sent by looking at NET_BUFFER_LIST::SourceHandle. Remove those from the stream (possibly reusing them, or just NdisFreeNetBufferList them). All the other packets then go up the stack via NdisFSendNetBufferListsComplete.
The above discussion also applies to the receive path. The only difference between send and receive is that on the receive path, you must pay close attention to the NDIS_RECEIVE_FLAGS_RESOURCES flag.
OID requests
Like the datapath, if you want to participate in OID requests at all (either filtering or issuing your own), you must be integrated into the entire OID stack. That means that you provide FilterOidRequest, FilterOidRequestComplete, and FilterCancelOidRequest handlers. You don't need to do anything special in these handlers beyond what the sample does, except again detecting OID requests that your filter originated in the oid-complete handler, and removing those from the stream (call NdisFreeCloneOidRequest on them).
Performance
Do not worry about performance here. The first step is to get it working. Even though the sample filter inserts itself into the send, receive, and OID paths; it's almost impossible to come up with any sort of benchmark that can detect the presence of the sample filter. It's extremely cheap to have do-nothing handlers in a filter.
If you feel very strongly about this, you can selectively remove your filter from the datapath with calls to NdisFRestartFilter and NdisSetOptionalHandlers(NDIS_FILTER_PARTIAL_CHARACTERISTICS). But I absolutely don't think you need the complexity. If you're coming from an NDIS 5 protocol that was capturing in promiscuous mode, you've already gotten a big perf improvement by switching to the native networking data structures (NDIS_PACKET->NBL) and eliminating the loopback path. You can leave additional fine-tuning to the next version.
When we send a large amount of data to the client,its ReceiveAsync event is being called more than one time and in each time we get a few piece of the packet.
What shall we do to get C# Silverlight Tcp Packet in one piece and through one event?
Thank you in advance.
You can't. The very nature of TCP is that data gets broken up into packets. Keep receiving data until you've got the whole message (whatever that will be). Some options for this:
First send the size of the message before the message itself.
Close the connection when the message has been sent (so the client can basically read until the connection is closed)
Add a delimiter to indicate the end of the message
I generally dislike the final option, as it means "understanding" the message as you're reading it, which can be tricky - and may mean you need to add escape sequences etc if your delimiter can naturally occur within the message.