I'm trying to get to know how many bytes there are readable at my TCP socket. I am calling ioctl with the Flag "FIONREAD" which should actually give me this value.
When I call the function I get as return val 0 ( so no Error ) but also my integer argument gets the value 0. That would be no problem but when I call the recv() method I actually read some Bytes out of the socket. What am I doing wrong?
// here some Code:
char recBuffer[BUFFERLENGTH] = {0};
int bytesAv = 0;
int bytesRead = 0;
int flags = 0;
if ( ioctl (m_Socket,FIONREAD,&bytesAv) < 0 )
{
// Error
}
if ( bytesAv < 1 )
{
// No Data Available
}
bytesRead = recv(m_Socket,recBuffer,BUFFERLENGTH,flags);
When I call the recv function i acutally read some valid Data ( which I expected )
It's happening very quickly, that's why you don't see anything. What you're doing:
ioctl: Is there data for me ? No, nothing yet
recv: Block until there is data for me. Some (short) time later: Here is your data
So if you really want to see FIONREAD, just wait for it.
/* Try FIONREAD until we get *something* or ioctl fails. */
while (!bytesAv && ioctl (m_Socket,FIONREAD,&bytesAv) >= 0)
sleep(1);
The real answer here is to use select(2) like cnicutar said. Toby, what you aren't understanding is that you have a race condition. First you look at the socket and ask how many bytes are there. Then, while your code is processing the "no data here" block, bytes are being received by the hardware & OS asynchronous to your application. So, by the time that the recv() function is called, the answer of "no bytes are available" is no longer true...
if ( ioctl (m_Socket,FIONREAD,&bytesAv) < 0 )
{ // Error
}
// BYTES MIGHT BE RECEIVED BY HARDWARE/OS HERE!
if ( bytesAv < 1 ) // AND HERE!
{
// No Data Available
// BUT BYTES MIGHT BE RECEIVED BY HARDWARE/OS HERE!
}
// AND MORE BYTES MIGHT BE RECEIVED BY HARDWARE/OS HERE!
bytesRead = recv(m_Socket,recBuffer,BUFFERLENGTH,flags);
// AND NOW bytesRead IS NOT EQUAL TO 0!
Sure, a small sleep probably fixed your program two years ago, but it also taught you terrible coding practice and you lost out on an opportunity to learn how to use sockets correctly by using select().
Further, as Karoly Horvath said, you can tell recv to not read more bytes than you can store in the buffer that the user passed in. Then your function interface becomes "This fn will return as many bytes as are available on the socket, but not more than [buffer size you passed in]".
This means that this function doesn't need to worry about clearing the buffer any more. The caller can call your function as many times as necessary to clear all of the bytes out of it (or you can provide a separate fn that discards the data wholesale and not tie up that functionality in any specific data gather function). Your function is more flexible by not doing too many things. You can then create a wrapper function that is smart to your data transfer needs of a particular application, and that fn calls the get_data fn and the clear_socket fn as needed for that specific app. Now you are building a library you can carry around from project to project, and maybe job to job if you're so lucky as to have an employer that lets you take code with you.
Use select() then ioctl(FIONREAD) then recv()
You're doing nothing wrong, if you are using blocking I/O recv() will block untill the data is available.
Related
This must be a stupid question because this should be a very common and simple problem, but I haven't been able to find an answer anywhere, so I'll bite the bullet and ask.
How on earth should I go about reading from the standard input when there is no way of determining the size of the data? Obviously if the data ends in some kind of terminator like a NUL or EOF then this is quite trivial, but my data does not. This is simple IPC: the two programs need to talk back and forth and ending the file streams with EOF would break everything.
I thought this should be fairly simple. Clearly programs talk to each other over pipes all the time without needing any arcane tricks, so I hope there is a simple answer that I'm too stupid to have thought of. Nothing I've tried has worked.
Something obvious like (ignoring necessary realloc's for brevity):
int size = 0, max = 8192;
unsigned char *buf = malloc(max);
while (fread((buf + size), 1, 1, stdin) == 1)
++size;
won't work since fread() blocks and waits for data, so this loop won't terminate. As far as I know nothing in stdio allows nonblocking input, so I didn't even try any such function. Something like this is the best I could come up with:
struct mydata {
unsigned char *data;
int slen; /* size of data */
int mlen; /* maximum allocated size */
};
...
struct mydata *buf = xmalloc(sizeof *buf);
buf->data = xmalloc((buf->mlen = 8192));
buf->slen = 0;
int nread = read(0, buf->data, 1);
if (nread == (-1))
err(1, "read error");
buf->slen += nread;
fcntl(0, F_SETFL, oflags | O_NONBLOCK);
do {
if (buf->slen >= (buf->mlen - 32))
buf->data = xrealloc(buf->data, (buf->mlen *= 2));
nread = read(0, (buf->data + buf->slen), 1);
if (nread > 0)
buf->slen += nread;
} while (nread == 1);
fcntl(0, F_SETFL, oflags);
where oflags is a global variable containing the original flags for stdin (cached at the start of the program, just in case). This dumb way of doing it works as long as all of the data is present immediately, but fails otherwise. Because this sets read() to be non-blocking, it just returns -1 if there is no data. The program communicating with mine generally sends responses whenever it feels like it, and not all at once, so if the data is at all large this exits too early and fails.
How on earth should I go about reading from the standard input when there is no way of determining the size of the data?
There always has to be a way to determinate the size. Otherwise, the program would require infinite memory, and thus impossible to run on a physical computer.
Think about it this way: even in the case of a never-ending stream of data, there must be some chunks or points where you have to process it. For instance, a live-streamed video has to decode a portion of it (e.g. a frame). Or a video game which processes messages one by one, even if the game has undetermined length.
This holds true regardless of the type of I/O you decide to use (blocking/non-blocking, synchronous/asynchronous...). For instance, if you want to use typical blocking synchronous I/O, what you have to do is process the data in a loop: each iteration, you read as much data as is available, and process as much as you can. Whatever you can not process (because you have not received enough yet), you keep for the next iteration. Then, the rest of the loop is the rest of the logic of the program.
In the end, regardless of what you do, you (or someone else, e.g. a library, the operating system, the hardware buffers...) have to buffer incoming data until it can be processed.
Basically, you have two choices -- synchronous or asynchronous -- and both have their advantages and disadvantages.
For synchronous, you need either delimeters or a length field embedded in the record (or fixed length records, but that is pretty inflexible). This works best for synchronous protocols like synchronous rpc or simplex client-server interactions where only one side talks at a time while the other side waits. For ASCII/text based protocols, it is common to use a control-character delimiter like NL/EOL or NUL or CTX to mark the end of messages. Binary protocols more commonly use an embedded length field -- the receiver first reads the length and then reads the full amount of (expected) data.
For asynchronous, you use non-blocking mode. It IS possible to use non-blocking mode with stdio streams, it just requires some care. out-of-data conditions show up to stdio like error conditions, so you need to use ferror and clearerr on the FILE * as appropriate.
It's possible for both to be used -- for example in client-server interactions, the clients may use synchronous (they send a request and wait for a reply) while the server uses asynchronous (to be be robust in the presence of misbehaving clients).
The read api on Linux or the ReadFile Api on windows will immediately return and not wait for the specified number of bytes to fill the buffer (when reading a pipe or socket). Read then reurns the number of bytes read.
This means, when reading from a pipe, you set a buffersize, read as much as returned and the process it. You then read the next bit. The only time you are blocked is if there is no data available at all.
This differs from fread which only returns once the desired number of bytes are returned or the stream determines doing so is impossible (like eof).
I got a snippet from internet for send data through a socket .
Here is the code .
u32_t nLength = 0;
u32_t nOffset = 0;
do {
nLength = nFullLength - nOffset;
status = Socket->Send(((u8_t*) buff) + nOffset, &nLength);
if (status != ERROR_SUCCESS) {
break;
}
nOffset += nLength;
} while (nOffset < nFullLength);
My doubts are :
When send(sock_fd, buf+bytes, buflen-bytes, flags); function running , it will send the entire data ?
Let's assume i have a buff with 45 byte length . So it will send like
send(buf+0, 45-0) = send(buf+0, 45);
So it will send complete data with length 45 ? what is the use of length here ? initially it will 45 . Isn't ?
Well, no. There's no guarantee that it will send all the data you ask it to send, that's why the code looks the way it does.
The manual page for send() states this pretty clearly:
Return Value
On success, these calls return the number of characters sent. On error, -1
is returned, and errno is set appropriately.
The same is true for e.g. a regular write() to a local file, by the way. It might never happen, but the way the interface is designed you're supposed to handle partial sends (and writes) if they do happen.
TCP is a streaming transport. There is no guarantee that a given send() operation will accept all of the bytes given to it at one time. It depends on the available kernel buffer space, the I/O mode of the socket (blocking vs non-blocking), etc. send() returns the number of bytes it actually accepted and put into the kernel buffer for subsequent transmission.
In the code example shown, it appears that Socket->Send() expects nLength to be initially set to the total number of bytes to sent, and it will then update nLength with the number of bytes actually sent. The code is adjusting its nOffset variable accordingly, looping just in case send() returns fewer bytes than requested, so it can call send() as many times as it takes to send the full number of bytes.
So, for example, lets assume the kernel accepts up to 20 bytes at a time. The loop would call send() 3 times:
send(buf+0, 45-0) // returns 20
send(buf+20, 45-20) // returns 20
send(buf+40, 45-40) // returns 5
// done
This is typical coding practice for TCP programming, given the streaming nature of TCP.
I'm very new to C++, but I'm trying to learn some basics of TCP socket coding. Anyway, I've been able to send and receive messages, but I want to prefix my packets with the length of the packet (like I did in C# apps I made in the past) so when my window gets the FD_READ command, I have the following code to read just the first two bytes of the packet to use as a short int.
char lengthBuffer[2];
int rec = recv(sck, lengthBuffer, sizeof(lengthBuffer), 0);
short unsigned int toRec = lengthBuffer[1] << 8 | lengthBuffer[0];
What's confusing me is that after a packet comes in the 'rec' variable, which says how many bytes were read is one, not two, and if I make the lengthBuffer three chars instead of two, it reads three bytes, but if it's four, it also reads three (only odd numbers). I can't tell if I'm making some really stupid mistake here, or fundamentally misunderstanding some part of the language or the API. I'm aware that recv doesn't guarantee any number of bytes will be read, but if it's just two, it shouldn't take multiple reads.
Because you cannot assume how much data will be available, you'll need to continuously read from the socket until you have the amount you want. Something like this should work:
ssize_t rec = 0;
do {
int result = recv(sck, &lengthBuffer[rec], sizeof(lengthBuffer) - rec, 0);
if (result == -1) {
// Handle error ...
break;
}
else if (result == 0) {
// Handle disconnect ...
break;
}
else {
rec += result;
}
}
while (rec < sizeof(lengthBuffer));
Streamed sockets:
The sockets are generally used in a streamed way: you'll receive all the data sent, but not necessarily all at once. You may as well receive pieces of data.
Your approach of sending the length is hence valid: once you've received the length, you cann then load a buffer, if needed accross successive reads, until you got everything that you expected. So you have to loop on receives, and define a strategy on how to ahandle extra bytes received.
Datagramme (packet oriented) sockets:
If your application is really packet oriented, you may consider to create a datagramme socket, by requesting linux or windows socket(), the SOCK_DGRAM, or better SOCK_SEQPACKET socket type.
Risk with your binary size data:
Be aware that the way you send and receive your size data appers to be assymetric. You have hence a major risk if the sending and receiving between machine with CPU/architectures that do not use the same endian-ness. You can find here some hints on how to ame your code platform/endian-independent.
TCP socket is a stream based, not packet (I assume you use TCP, as to send length of packet in data does not make any sense in UDP). Amount of bytes you receive at once does not have to much amount was sent. For example you may send 10 bytes, but receiver may receive 1 + 2 + 1 + 7 or whatever combination. Your code has to handle that, be able to receive data partially and react when you get enough data (that's why you send data packet length for example).
I have a problem when I want to execute two consecutive times the read function Address to retrieve all the data sent by the server.
I read once then read create an infinity loop.
Code :
char buff[50] = {0};
nbytes = 1;
while (nbytes > 0)
{
nbytes = read(m_socket, buff, sizeof(buff));
}
why read create infinity loop ? is not the "while" the problem.
thank you for your answers.
socket(2) gives you a blocking file descriptor, meaning a system call like read(2) blocks until there's enough some data available to fulfill your request, or end of stream (for TCP) happens (return value 0), or some error happens (return value -1).
This means you never get out of your while loop until you hit an error, or the other side closes the connection.
Edit 0:
#EJP is thankfully here to correct me, as usual - read(2) blocks until any data is available (not the whole thing you requested, as I initially stated above), including an end-of-stream or an error.
I have two programs that use socket programming to communicate. Initially I will specify the no. of hops as to how many time they have to exchange messages between each other. Each time it receives a message, it will append its id to it. Hence the string grows in size every time. My program is working fine for 8000 hops, but after it crosses 8000, although program p1 sends a string of length 16388, p2 identifies that there are only 16385 in the socket ready to be read. I use ioctl() to determine the amount of characters ready to recv() in the socket, and then recv it in a char * variable...
Is it because there is a delay in the send () in p1 and recv() in p2 , that p2 identifies only 16385 characters in the socket ?
For ex:
If P1 sends length(16388)
P2 receives only the following length(16385)
Say I'm trying to send you 8 pumpkins. I put 6 of them on the table. You think, "I'm expecting 8 pumpkins, not 6. I'll wait until he puts the last two on the table." I think, "I don't want too many pumpkins 'in flight' at once. I'll wait until he takes 2 of these 6 before I put the last 2 on the table." We're stuck. We're each waiting for the other. We'll wait forever.
You are not permitted to wait until more bytes are received before accepting the bytes that have already been received. The reason for this is simple: No network protocol can allow each side to wait for the other. Since TCP permits the sending side to wait in this context, it cannot permit the receiving side to wait as well.
So accept the bytes as they are received. Don't wait for the other side to send all of them before accepting any of them. Otherwise, what happens if the other side is waiting for you to accept the first one before it sends any more?
You're probably hitting a kernel buffer limit. You can probably increase SO_RCVBUF on the receiver and it will work as you expect: SIOCINQ will eventually return the full size of the unread data.
But you shouldn't do that to ensure proper Functioning. messing with buffers should only be done when you want to tweak performance.
You should restructure the code so that you never have to ask the kernel how many bytes are available. Just read up to a reasonable limit(like 4096) and deal with one application-level message being broken up in multiple pieces. If you need message lengths/boundaries then you MUST implement them yourself on top of TCP.
Here's some silly code to read a message with a length header:
int ret, len = 0, have_read;
have_read = 0;
while (have_read < sizeof(len)) {
// This will likely always return sizeof(len) the first time.
ret = read(fd, ((char*)&len) + have_read, sizeof(len) - have_read);
if (ret <= 0) {
// Handle error.
}
have_read += ret;
}
char* buf = malloc(len);
if (!buf) {
// Handle error.
}
have_read = 0;
while (have_read < len) {
ret = read(fd, buf + have_read, len - have_read);
if (ret <= 0) {
// Handle error.
}
have_read += ret;
}
// Handle message in buf.