If a C socket's recvmsg() has a queue, how can I find out how many items are backlogged in the queue?
My problem is that the speed of the code after I receive something from recvmsg() is sometimes slower than the rate of data sent to recvmsg(), which would logically result in a queue? What happens if the queue becomes too big?
For example if this is my recv() code:
while (recvmsg(SocketA,...) > 0)
{
...
...something that takes 1.5 seconds to execute/complete...
...
}
and if the following function somewhere else gets called every 1 second:
// gets called every 1 second
int send_to_sock()
{
...
send(SocketA, ...);
...
}
I checked the reference pages for recvmsg() but none of them mention anything about a queue, but it seems like there is a queue because I am observing delays that incrementally add up. But the code stops working if the queue gets too long, so I want to know if there is a way to check length of queue.
It's not a queue, it's a buffer associated with most of I/O devices, not only sockets. For example, when you read from stdin with something like scanf, data goes to your program when you press enter. Where do you think all keystrokes are stored in the meantime?
The specific details of these buffers are implementation defined, but usually what happens with sockets is that new packets are discarded if the buffer is full. You can find more information about querying the state of the buffer in How to find the socket buffer size of linux and How can I tell if a socket buffer is full?.
Yes. There is a socket receive buffer. Its maximum size can be got and set via getsockopt() and setsockopt() using the SO_RCVBUF option. When it fills, in the case of TCP the sender is told to stop sending; in the case of UDP further incoming data is discarded.
Related
How to flush Input Buffer (if such thing exists at all) of an UDP Socket in C ?
I'm working on an embedded Linux environment and using C to create some native application. There are several of these embedded machines on the same network, and when an event occurs on one of them (lets call it the WHISTLE-BLOWER), WHISTLE-BLOWER should send a network message to the network broadcast address, so that all machines on the network (including the WHISTLE-BLOWER) knows about the event and executes some actions according to it. I'm using UDP socket by the way...
Here's the pseudo-code for it:
main
{
startNetworkListenerThread( networkListenerFunction );
while( not received any SIGTERM or such )
{
localEventInfo = checkIfTheLocalEventOccured();
broadcastOnNetwork( localEventInfo );
}
}
networkListenerFunction
{
bindSocket;
while( not SIGTERM )
{
// THIS IS WHERE I WANT TO FLUSH THE RECV BUFFER...
recv_data = recvfrom( socket );
if( validate recv data )
{
startExecuteLocalAction;
sleep( 5 );
stopExecuteLocalAction;
}
}
}
The way I expect and want to work this code is:
1. LOCAL_EVENT occured
2. Broadcasted LOCAL_EVENT_INFO on network
3. All machines received EVENT_INFO, including the original broadcaster
4. All machines started executing the local action, including the original broadcaster
5. All machines' network listener(thread)s are sleeping
6. Another LOCAL_EVENT2 occured
7. Since all machines' listener are sleeping, LOCAL_EVENT2 is ignored
8. All machines' network listener(thread)s are now active again
9. GO BACK TO 1 / RESTART CYCLE
RESULT = TOTAL 2 EVENTS, 1 IGNORED
The way it actually works is:
1. LOCAL_EVENT occured
2. Broadcasted LOCAL_EVENT_INFO on network
3. All machines received EVENT_INFO, including the original broadcaster
4. All machines started executing the local action, including the original broadcaster
5. All machines' network listener(thread)s are sleeping
6. Another LOCAL_EVENT2 occured
7. Eventhough all machines' listener are sleeping; LOCAL_EVENT2 is queued SOMEHOW
8. All machines' network listener(thread)s are now active again
9. All machines received EVENT_INFO2 and executed local actions again, slept and reactivated
10. GO BACK TO 1 / RESTART CYCLE
RESULT = TOTAL 2 EVENTS, 0 IGNORED
tl,dr: The packets/messages/UDP Broadcasts sent to an already binded socket, whoose parent thread is sleeping at the delivery-moment; are somehow queued/buffered and delivered at the next 'recvfrom' call on the said socket.
I want those UDP broadcasts to be ignored so I was thinking of flushing the receive buffer (obviously not the one i'm giving as parameter to the recvfrom method) if it exists before calling recvfrom. How can I do that? or what path should I follow?
Please note that the notion of "flushing" only applies to output. A flush empties the buffer and ensures everything in it was sent to its destination. Regarding an input buffer, the data is already at its destination. Input buffers can be read from or cleared out, but not "flushed".
If you just want to make sure you have read everything in the input buffer, what you are looking for is a non-blocking read operation. If you try it and there's no input, it should return an error.
A socket has a single receive buffer inside the TCP/IP stack. It's essentially a FIFO of the datagrams received. TCP and UDP handle that queue differently though. When you call recv(2) on a UDP socket you dequeue a single datagram from that buffer. TCP arranges the datagrams into a stream of bytes according to the sequence numbers. When receive buffer overflows the datagram is dropped by the stack. TCP tries to resend in this case. UDP doesn't. There's no explicit "flush" function for the receive buffer other then reading the socket or closing it.
Edit:
You have an inherent race condition in your application and it looks like you are trying to solve it with a wrong tool (TCP/IP stack). What I think you should be doing is defining a clean state machine for the app. Handle the events that make sense at current state, ignore events that don't.
One other thing to look at is using multicast instead of broadcast. It's a bit more involved but you would have more control of the "subscriptions" by joining/leaving multicast groups.
I wrote a simple C socket program that sends an INIT package to the server to indicate to prepare a text transfer. The server does not sends any data back at that time.
After sending the INIT package the client sends a GET package and waits for chunks of data from the server.
So every time the server receives a GET package it will send a chunk of data to the client.
So far so good. The buffer has a size of 512 bytes, a chunk is 100 Bytes plus a little overhead big.
But my problem is that the client does not receive the second message.
So my guess is that read() will blpck until the buffer is full. Is that right or what might be the reason for that?
It depends. For TCP sockets read may return before the buffer is full, and you may need to receive in a loop to get a whole message. For UDP sockets the size you read is typically the size of a single packet (datagram) and then read may block until it has read all the requested data.
The answer is no: read() on a tcp/ip socket will not block until the buffer has the amount of data you requested. read() will return immediately in all cases if any data is available, even if your socket is blocking and you've requested more data than is available.
Keep in mind that TCP/IP is a byte stream protocol and you must treat it as such. The interface is under no obligation to transmit your data together in a single packet, as long as it is presented to you in the order you placed it in the socket.
The answer is no , read is not blocking call , You can refer below points to guess the error
Several Checkpoints you can find :
Find out what read is returning at the second time .
memset the buffer every time in while before recv
use fflush(stdout) if not able to output.
Make sure all three are present . if problem not solved yet .please post source code here
My program contains a thread that waits for UDP messages and when a message is received it runs some functions before it goes back to listening. I am worried about missing a message, so my question is something along the line of, how long is it possible to read a message after it has been sent? For example, if the message was sent when the thread was running the functions, could it still be possible to read it if the functions are short enough? I am looking for guidelines here, but an answer in microseconds would also be appreciated.
When your computer receives a UDP packet (and there is at least one program listening on the UDP port specified in that packet), the TCP stack will add that packet's data into a fixed-size buffer that is associated with that socket and kept in the kernel's memory space. The packet's data will stay in that buffer until your program calls recv() to retrieve it.
The gotcha is that if your computer receives the UDP packet and there isn't enough free space left inside the buffer to fit the new UDP packet's data, the computer will simply throw the UDP packet away -- it's allowed to do that, since UDP doesn't make any guarantees that a packet will arrive.
So the amount of time your program has to call recv() before packets start getting thrown away will depend on the size of the socket's in-kernel packet buffer, the size of the packets, and the rate at which the packets are being received.
Note that you can ask the kernel to make its receive-buffer size larger by calling something like this:
size_t bufSize = 64*1024; // Dear kernel: I'd like the buffer to be 64kB please!
setsockopt(mySock, SOL_SOCKET, SO_RCVBUF, &bufSize, sizeof(bufSize));
… and that might help you avoid dropped packets. If that's not sufficient, you'll need to either make sure your program goes back to recv() quickly, or possibly do your network I/O in a separate thread that doesn't get held off by processing.
I'm using a TCP socket to send data from a client. The data is prepared, in advance, so I always have more data to send. Moreover, I can't change the size of the writes, otherwise the server will complain. I do the following:
while (1) {
poll(for POLLOUT condition);
write(to TCP socket);
if (no more data)
break;
}
The problem is that the POLL takes a very long time. I assume this is the time that the socket is actually being written to (or responded to). Is there anyway that I can reduce the amount of time spent in the poll? It is currently a major bottleneck.
Socket being busy causes delays in poll()
Of course it does. That's what poll() is for. To delay until a socket becomes readable or writable.
Your writer is faster than your reader. Look for a solution at the reading end. Your writing end is behaving correctly.
However calling it every time at the head of that loop is pointless. Only call it when you need to know the socket has become writable. It is normally writable all the time, except when your socket send buffer is full, so calling it every time is a waste of time.
Just keep writing until you get EAGAIN/EWOULDBLOCK. Then is the time to call poll(), to tell you when there is space in the socket send buffer. Then just resume writing again as before.
Poll will raise a POLLOUT event when there's enough buffer space to enqueue further data. (Look at this link - Man (7) socket )
If it doesn't, it means write buffer is full and it means you're writing faster than the read capabilities of the other peer. Or simply the network is slower than you expect.
First, a little bit of context to explain why I am on the "UDP sampling" route:
I would like to sample data produced at a fast rate for an unknown period of time. The data I want to sample is on another machine than the one consuming the data. I have a dedicated Ethernet connection between the two so bandwidth is not an issue. The problem I have is that the machine consuming the data is much slower than the one producing it. An added constraint is that while it's ok that I don't get all the samples (they are just samples), it is mandatory that I get the last one.
My 1st solution was to make the data producer send a UDP datagram for each produced sample and let the data consumer try to get the samples it could and let the others be discarded by the socket layer when the UDP socket is full. The problem with this solution is that when new UDP datagrams arrive and the socket is full, it is the new datagrams that get discarded and not the old ones. Therefore I am not guarantueed to have the last one!
My question is: is there a way to make a UDP socket replace old datagrams when new arrive?
The receiver is currently a Linux machine, but that could change in favor of another unix-like OS in the future (windows may be possible as it implements BSD sockets, but less likely)
The ideal solution would use widespread mecanisms (like setsockopt()s) to work.
PS: I thought of other solutions but they are more complex (involve heavy modification of the sender), therefore I would like first to have a definite status on the feasability of what I ask! :)
Updates:
- I know that the OS on the receiving machine can handle the network load + reassembly of the traffic generated by the sender. It's just that its default behaviour is to discard new datagrams when the socket buffer is full. And because of the processing times in the receiving process, I know it will become full whatever I do (wasting half of the memory on a socket buffer is not an option :)).
- I really would like to avoid having an helper process doing what the OS could have done at packet-dispatching time and waste resource just copying messages in a SHM.
- The problem I see with modifying the sender is that the code which I have access to is just a PleaseSendThisData() function, it has no knowledge that it can be the last time it is called before a long time, so I don't see any doable tricks at that end... but I'm open to suggestions! :)
If there are really no way to change the UDP receiving behaviour in a BSD socket, then well... just tell me, I am prepared to accept this terrible truth and will start working on the "helper process" solution when I go back to it :)
Just set the socket to non-blocking, and loop on recv() until it returns < 0 with errno == EAGAIN. Then process the last packet you got, rinse and repeat.
I agree with "caf".
Set the socket to a non-blocking mode.
Whenever you receive something on the socket - read in a loop until nothing more is left. Then handle the last read datagram.
Only one note: you should set a large system receive buffer for the socket
int nRcvBufSize = 5*1024*1024; // or whatever you think is ok
setsockopt(sock, SOL_SOCKET, SO_RCVBUF, (char*) &nRcvBufSize, sizeof(nRcvBufSize));
This will be difficult to get completely right just on the listener side since it could actually miss the last packet in the Network Interface Chip, which will keep your program from ever having had a chance at seeing it.
The operating system's UDP code would be the best place to try to deal with this since it will get new packets even if it decides to discard them because it already has too many queued up. Then it could make the decision of dropping an old one or dropping a new one, but I don't know how to go about telling it that this is what you would want it to do.
You can try to deal with this on the receiver by having one program or thread that always tries to read in the newest packet and another that always tries to get that newest packet. How to do this would differ based on if you did it as two separate programs or as two threads.
As threads you would need a mutex (semaphore or something like it) to protect a pointer (or reference) to a structure used to hold 1 UDP payload and whatever else you wanted in there (size, sender IP, sender port, timestamp, etc).
The thread that actually read packets from the socket would store the packet's data in a struct, acquire the mutex protecting that pointer, swap out the current pointer for a pointer to the struct it just made, release the mutex, signal the processor thread that it has something to do, and then clear out the structure that it just got a pointer to and use it to hold the next packet that comes in.
The thread that actually processed packet payloads should wait on the signal from the other thread and/or periodically (500 ms or so is probably a good starting point for this, but you decide) and aquire the mutex, swap its pointer to a UDP payload structure with the one that is there, release the mutex, and then if the structure has any packet data it should process it and then wait on the next signal. If it did not have any data it should just go ahead and wait on the next signal.
The processor thread should probably run at a lower priority than the UDP listener so that the listener is less likely to ever miss a packet. When processing the last packet (the one you really care about) the processor will not be interrupted because there are no new packets for the listener to hear.
You could extend this by using a queue rather than just a single pointer as the swapping place for the two threads. The single pointer is just a queue of length 1 and is very easy to process.
You could also extend this by attempting to have the listener thread detect if there are multiple packets waiting and only actually putting the last of those into the queue for the processor thread. How you do this will differ by platform, but if you are using a *nix then this should return 0 for sockets with nothing waiting:
while (keep_doing_this()) {
ssize_t len = read(udp_socket_fd, my_udp_packet->buf, my_udp_packet->buf_len);
// this could have been recv or recvfrom
if (len < 0) {
error();
}
int sz;
int rc = ioctl(udp_socket_fd, FIONREAD, &sz);
if (rc < 0) {
error();
}
if (!sz) {
// There aren't any more packets ready, so queue up the one we got
my_udp_packet->current_len = len;
my_udp_packet = swap_udp_packet(my_ucp_packet);
/* swap_udp_packet is code you would have to write to implement what I talked
about above. */
tgkill(this_group, procesor_thread_tid, SIGUSR1);
} else if (sz > my_udp_packet->buf_len) {
/* You could resize the buffer for the packet payload here if it is too small.*/
}
}
A udp_packet would have to be allocated for each thread as well as 1 for the swapping pointer. If you use a queue for swapping then you must have enough udp_packets for each position in the queue -- since the pointer is just a queue of length 1 it only needs 1.
If you are using a POSIX system then consider not using a real time signal for the signaling because they queue up. Using a regular signal will allow you to treat being signaled many times the same as being signaled just once until the signal is handled, while real time signals queue up. Waking up periodically to check the queue also allows you to handle the possibility of the last signal arriving just after you have checked to see if you had any new packets but before you call pause to wait on a signal.
Another idea is to have a dedicated reader process that does nothing but loops on the socket and reads incoming packets into circular buffer in shared memory (you'll have to worry about proper write ordering). Something like kfifo. Non-blocking is fine here too. New data overrides old data. Then other process(es) would always have access to latest block at the head of the queue and all the previous chunks not yet overwritten.
Might be too complicated for a simple one-way reader, just an option.
I'm pretty sure that this is a provably insoluble problem closely related to the Two Army Problem.
I can think of a dirty solution: establish a TCP "control" sideband connection which carries the last packet which is also a "end transmission" indication. Otherwise you need to use one of the more general pragmatic means noted in Engineering Approaches.
This is an old question, but you are basically wanting to turn the socket queue (FIFO) into a stack (LIFO). It's not possible, unless you want to fiddle with the kernel.
You'll need to move the datagrams from kernel space to user space and then process. Easiest approach would be a loop like this...
Block until there is data on the socket (see select, poll, epoll)
Drain the socket, storing datagrams per your own selection policy
Process the stored datagrams
Repeat