I have written a function to read a buffer which contains the details of hardware used. Now the size of buffer depends on the hardware. Is there any way to check before reading the buffer the size of buffer so that i do not access any illegal location and avoid segmentation fault? Or before reading any location is it possible to see if I am accessing valid location?
Methode 1:
If you don't know the buffer size, first ask the size of buffer every time. Then allocate the size using malloc. After using it free it.
Methode 2:
At least you have an idea about maximum size how much it will take. allocate MAX_BUFF_SIZE statically.
If you don't like above two procedures the do like below.
Methode 3:
Sender know how much size he is sending. Fix a limited size to transfer at a time, and send one by one.
Eg: Fixed transfer size = 10bytes. //both sender and receiver have an idea
message size to send = 50bytes. //sender have idea but receiver don't know.
struct my_struct {
int transaction_no;
char buf[SIZE];
int total_messages;
int current_message_no;
};
so sender splits his data into 5 packets(each one 10bytes). send each packet one by one.
so receiver checks the incoming packets and again reconstruct the data.
I think first two methods are better than this.
Related
I wish to dynamically allocate enough memory on the server and clients receive part when I send a package. I.e. if I send a package of 512 bytes to client from server, I want the clients sides char* receive_data to allocate corresponding amount of memory of the package.
Here's some pseudo code:
I was thinking of having an counter which loops 2 times, on the first loop it'll send the client/server the length of the package and on the second run it'll send the package itself.
int write_all(int socket, void *buffer, size_t *length){
int ret;
int counter = 0;
while (counter != 2) {
if (counter == 0) {
// Write() the length to client/server.
} else {
ret = write(socket, buffer, length);
if(ret < 1) break;
}
counter++;
}
return ret == -1 ? -1 : 0;
Is this a good or bad way to do this? I might have tunnel vision, if you have a more suitable way of doing this, please share.
Edit: http://beej.us/guide/bgnet/output/html/singlepage/bgnet.html#sendall I've read this part, and thought that perhaps be another way to do it. But then I have to statically allocate a char
(char receive_data[512];). Which might work as well, but is it as flexible as the way I'm trying to do it?
Edit: This first part of the answer deals with the suitability of alternatingly sending message size and message proper as a communication protocol
It will work, unless you can get out of sync. (I.e. could the receiver possibly miss the size packet but receive the data packet?)
You should also consider security implications: Can an attacker cause a denial of service by sending you an absurdly large data size, causing your server process to either swap massively or die.
Usually your problem is addressed by logically splitting network packets into a header and a body part, and some field in the header specifies the packet length. And the maximum length is limited to a reasonable value. This is very similar with what you do.
Edit: This second part of the answer deals with short reads/writes
After reading your latest comment, I guess your problem is:
Your communication is over TCP. This is your first time you do something like this. You've read about short reads and short writes, i.e. read() reading fewer bytes than you told it to, or write writing fewer bytes than you had available. You've read about how people usually deal with this (bookkeeping), but you wonder if there might be an easier solution: What if you just allocated enough memory, so that there would be no need for the read and write system calls to return without reading/writing everything you told them to.
Is that about about correct?
Then my response is: No, this is not a valid solution for the short read / short write problem. The limiting factor is not the memory that you provide to these system calls. All of your reads and writes may still end up shorter than you told them to. There is no easy way out of the bookkeeping.
So I'm sending data on a TCP socket, prefixed with the size of data, as so:
write(socket, &length, sizeof(length));
write(socket, data, length);
(Note: I have wrapper writen functions as described in the Unix Network Programming book, and am checking for errors, etc. The above is just for the simplicity of this question).
Now, my experience is that breaking up data into multiple writes can cause significant slowdown. I have had success speeding things up by creating my own buffer, then sending out one big chunk.
However, in the above case data may be incredibly large (lets say 1 Gig). I don't want to create a buffer 1 Gig large + 4 bytes, just to be able to have one write() call. Is there any way of doing something akin to:
write(socket, &length, data, sizeof(length) + length)
without paying the price of a large memory allocation ahead of time? I suppose I could just pre-allocate a chunk the size of write's buffer, and continuously send that (the below code has errors, namely, should be sending &chunk + 4 in some instances, but this is just the idea):
length += 4;
char chunk[buffer_size];
var total = 0;
while (total < length)
{
if (total < 4)
{
memcpy(&chunk, &length, 4);
total += 4;
}
memcpy(&chunk, data + total, min(buffer_size, length - total));
write(sock, &chunk, min(buffer_size, length - total));
total += min(buffer_size, length - total);
}
But in that case I don't know what write's buffer size actually is (is there an API to get it?) I also don't know if this is an appropriate solution.
There is an option to do this already. It will inform your network layer that you are going to send more data and you want to buffer rather than send it as soon as possible.
setsockopt(sock_descriptor, IPPROTO_TCP, TCP_CORK, (char *)&val, sizeof(val));
val is an int, and should be 0 or 1, with the "cork" on, your network layer will buffer things as much as possible, to only send full packets, you might want to "pop the cork" and "cork" again to handle the next batch of transmissions that you need to make on the socket.
Your idea is correct, this just saves you the trouble of implementing it, since it's already done in the network stack.
I suggest having a look at writev() (see man writev for full details).
This allows you to send multiple buffers in one go, with just one call. As a simple example, to send out two chunks in one go (one for length, one for data):
struct iovec bits[2];
/* First chunk is the length */
bits[0].iov_base = &length;
bits[0].iov_len = sizeof(length);
/* Second chunk is the payload */
bits[1].iov_base = data;
bits[1].iov_base = length;
/* Send two chunks at once */
writev(socket, bits, 2);
It can get more complicated if you need to use a variable number of chunks (you may need to allocate the array of struct iov dynamically), but the advantage is that, if your chunks are large, you can avoid copying them, and just manipulate pointer/length pairs, which are much smaller.
I think you are on the right track with your spooled solution presented. I think buffer_size should be larger than that used internally by the network stack. This way, you minimize the amount of per-write overhead without having to allocate a giant buffer. In other words, by giving the underlying network subsystem more data than it can handle at once, it is free to run at its fastest speed, spending most of its time moving data, rather than waiting for more data to be provided.
The optimal buffer_size value might vary from system to system. I would start with 1MB and do some experiments up and down from there to see what works best. There might also be values you can extract and adjust with a sysctl call for the current internal buffer size used on your system. Read this for a suggested technique. You might also use something like getsockopt(..., SO_MAX_MSG_SIZE, ...), as explained here.
Ethernet packets can range up to about 64K in size, so perhaps anything larger than 64K is sufficient. Read about maximum transmission unit (MTU) sizes to get a sense of what the lowest layers of the network stack are doing, and don't forget that the MTU varies with the network interface, not the process or kernel.
Beware that the MTU can vary along the route from your server to the data's destination. You can use ifconfig or traceroute/tracepath to discover it. With networking, every link in the chain is weak. ;)
I am testing a small issue with a daemon here (written in linux). I want to know whether what is done is right or not.
The daemon loads a shared object file (.so) using dlopen call. The the shared object receives some buffers from clients over the network. It uses the following call to read the buffer:
read_buffer(something, length of buffer read, buffer contents);
The read_buffer function copies the buffer of length specified in the second parameter, to another location using memcpy
On the client side, the following is done:
write_buffer(something, length of buffer, buffer contents);
The problem is that if we send an invalid length parameter (not matching the real length of what is copied in the third parameter), from the client side, there is a segfault in the server side in the memcpy location
I am not sure how to input-validate the parameters that are passed to memcpy function
Request you to please help me out understand what is possible solution
To check C/C++ code for errors in memory allocation / access, use Valgrind. The server side has no way (that I know of) to determine if the parmateres passed are valid or not. That's the C/C++ credo: Know what you're doing, or you die. There is no safety net.
You can append the length of the buffer at the start of the buffer.
-------------------------------------------------------------
| FIXED LENGTH OF BUFFER - n bytes | BUFFER
-------------------------------------------------------------
Now each time you read it on the server side, first read "n" bytes(reserved for storing the length) which contain the length. When the data arrives, you can compare length of buffer and first n bytes for validation.
Hope this helps.
I'm sending a C struct over UDP
struct packet{
int numInt;
int* intList; //malloc'ed as (sizeof(int)*numInt)
}
It will be serialized as [numInt][intList[0]]...[intList[numInt-1]].
My understanding is that calling recvfrom on UDP will read the entire packet, even if the buffer doesn't hold that many bytes. Is using a really large buffer the only option I have?
You could pass MSG_PEEK to recvfrom to find out exactly how big the buffer needs to be. So just recvfrom a few bytes with MSG_PEEK to find numInt and then recvfrom the real thing (this time without MSG_PEEK).
The standard says something about MSG_PEEK, but kernel.org spells it better:
MSG_PEEK
This flag causes the receive operation to return data from the
beginning of the receive queue without removing that data from the
queue. Thus, a subsequent receive call will return the same data.
Obviously at some point you will start wondering if doubling the number of system calls to save memory is worth it. I think it isn't.
UDP packets are sent and received as a whole. if you receive it, the size is right. The only thing you have to do is to supply a big enough buffer on read() or recv() or recfrom(). The length field inside the payload is redundant, since the read() will tell you the correct size. It is also dangerous, since it relies on the sender and reciever having the same byte order.
You could try using a small buffer, just large enough to get numInt, with the MSG_PEEK flag set. Then you can find out the size you actually need, and receive again without MSG_PEEK to get the whole thing.
I'm pretty sure recvfrom will read up to as many bytes as is told to it by its 3rd argument, len. If there are fewer bytes available, it will return what is there. If there are more, it will return up to len bytes. You may have to make additional calls to obtain all the data your are expecting.
This is in reference to a previous question of mine which was already answered:
C: Using a select call, when I am reading, how do I keep track of the data?
My protocol actually sends how much data it is about send initially before it sends any other data. But yesterday, when testing out this code of mine using a browser, I had this one question:
A lot of people suggested that I check for message length but take the scenario of a browser. The browser's HTTP Request does not have a size when it is first sent to the server. Now, assuming that I used a 256 byte buffer, how am I supposed to manage the data structure of this client if I keep receiving partial headers between each multiplexing action? Keep using realloc as I keep getting more data and then when I encounter a termination sequence ('\r\n'), assume that all data has been received?
What I mean is, have something like this:
typedef struct {
int fd;
char *data;
} CLIENT;
And then keep using realloc on data? I was told to allocate a buffer size of the max protocol header but is that the only approach?
reallocing the buffer in sizes is fine. I'd pick a bit larger buffer than 256 - 1024 atleast or 4096. Allocating a max buffer size of a protocol like HTTP is not really feasible. You could also build an abstraction on top of your buffer that are able to read lines - much like fgets does on a FILE*
Keep in mind that reading a chunk might read the last part of the HTTP headers and a piece of the HTTP body if there's any , so you need to make sure you slice the buffer properly as you likely want to seperate the header from the body. You'll also have to look for the Content-Length: header, parse that, so you know how long the body will be.