my task is get from server data of an file with HTTP protocol
but problem is that i dont know the size of the content so i iterate through read func to get data from socked but it iterates every time only once i dont know really why
int res, len, total, boolk = 0, p=0;
while ((res = read(client_socket,bufferOut,4095)) > 0)
{
bufferOut[res]= '\0';
if(p==0)
{
buffer = calloc(strlen(bufferOut)+1,sizeof(char));
}
else
{
buffer = realloc(buffer,strlen(buffer)*sizeof(char)+strlen(bufferOut)+1*sizeof(char));
}
strcat(buffer,bufferOut);
if(isEnough(bufferOut)!=0 && boolk == 0)
{
index = getIndex(buffer);
kk = getCode(buffer);
len = getLen(buffer);
boolk = 1;
}
if(strlen(bufferOut)>=len+index && boolk == 1)
{
break;
}
p++;
}
function isEnough only looks if full http header arrived getIndex get lenght of header and getLen get length of that file so i wanted to iterate until buffer is same lenght as header file which should be
You're working much too hard.
int
read_until_enough(int client_socket, char **buf_out, size_t *buflen_out)
{
char *rdbuf = xmalloc(INITIAL_BUFFER_SIZE);
size_t buflen = 0, bufalloc = INITIAL_BUFFER_SIZE;
for (;;) {
ssize_t n = recv(client_socket, rdbuf + buflen, bufalloc - buflen);
if (n <= 0) goto recv_failure;
if (is_enough(rdbuf, buflen)) break;
buflen += n;
if (buflen == bufalloc) {
bufalloc *= 2;
rdbuf = xrealloc(rdbuf, bufalloc);
}
}
*rdbuf_out = rdbuf;
*buflen_out = buflen;
return 0;
recv_failure:;
int save_errno = errno;
free(rdbuf);
errno = save_errno;
*rdbuf_out = 0;
*buflen_out = 0;
return -1;
}
Please note that is_enough now takes the size of the received data as an argument, caller gets both the received data and its size, and the buffer is NOT nul-terminated. HTTP is a binary-transparent protocol; you MUST NOTrfc2119 assume that you never receive nul bytes.
Please also note that is_enough is stateless and is expected to do all of the parsing. This is better separation of concerns. Worry about the performance cost of re-parsing the HTTP headers each time around the loop only after you have everything working.
Finally, the functions xmalloc and xrealloc are wrappers around malloc and realloc that either succeed or crash the program. I am using them here to avoid cluttering up the example with error-recovery logic for memory allocation. You will have to decide whether they are appropriate for whatever you're doing.
Related
I have a python tcp server that accepts connections and generates a random string of length between (0,1M) characters, on the other side I have a c client that needs to listen on that socket and read the string and convert it into a single char of the same length as the string returned by the server
int receiver(int soc_desc, char * buffer)
{
char *arr = (char *)malloc(sizeof(char));
unsigned int received , total_received;
while (1)
{
memset(arr, 0, MAX); // clear the buffer
if ( received = recv(soc_desc, arr , MAX, 0) < 0)
{
break;
}
else
{
total_received += received;
}
}
printf("%s\n",arr);
return received;
}
// soc_desc is the socket descriptor
// buffer is the buffer that will hold the final output
The only way that I can think of is using malloc to read chunks of the data returned from the server but I am having bad time trying to figure it out and I need to convert the array of char pointers into a single char when the client is done receiving data from the server
Reassembling network data, particularly from TCP, can get tricky. The following code is untested and surely doesn't account for all contingencies, but hopefully is down the right path of what you need to do.
ssize_t receiver(int soc_desc, char * buffer)
{
// Whats the buffer argument used for?
// what you had before only allocated space for 1 char. That's not what you want
// This allocates for MAX+1 chars (I'm doing +1 for a NUL terminator)
char *arr = malloc(MAX+1);
// if MAX is small enough, you could do
// char arr[MAX+1];
// 0 buffer. You could use calloc instead of malloc + memset
memset(arr, 0, MAX+1);
// initialize total_received to 0
ssize_t received , total_received = 0;
size_t spaceLeftInBuf = MAX;
while (1)
{
// don't memset here, you'll erase the data you received last iteration
// write data to arr+total_receieved. This way you won't overwrite what
// you received the last iteration
received = recv(soc_desc, arr+total_received, spaceLeftInBuf, 0);
if (received < 0)
{
// there was an error
perror("recv failed: ");
// do something with the data already received? Ok, break and
// print what we've got
break;
}
else if (received == 0)
{
// socket closed gracefully, suppose we can break again and print
// what we've got
break;
else
{
// update counters
total_received += received;
spaceLeftInBuf -= received;
// is our buffer full? This may not be the right check, you need to
// decide when to process the data
// total_received better not ever be > MAX...
if (total_received >= MAX)
{
// "process" the data by printing it
printf("%s\n", arr);
// reset
total_received = 0;
spaceLeftInBuf = MAX;
// not particularly necessary to reset this to all 0s, but should
// make sure printing goes smoothly if we break out of this loop
memset(arr, 0, MAX); // arr[MAX] should already be '\0' from above
}
}
}
printf("%s\n",arr);
return received;
}
See Do I cast the result of malloc?
I found a way to do it but this is not tested enough and for sure will cause memory issues
char *arr = malloc(sizeof(char));
char tmp_buff[MAX];
memset(arr,0,MAX);
while (recv(soc_desc, tmp_buff , MAX, 0) > 0 )
{
strcat(arr , tmp_buff);
printf("Size : %ld arr : %s\n",strlen(tmp_buff),tmp_buff);
}
I want to write a recvall function for socket in C. I assume, that every message in my protocol ends with \r\n. I wrote something like this below:
int recvall (int socket, char *buffer, int numBytes)
{
int bytesRcvd = 0;
numBytes = numBytes - 1;
while(bytesRcvd < numBytes)
{
int chunk = recv(socket, buffer + bytesRcvd, numBytes - bytesRcvd, 0);
if (chunk == 0 || chunk == -1)
break;
if(strstr(buffer, "\r\n"))
break;
bytesRcvd += (chunk);
}
buffer[bytesRcvd] = '\0';
return bytesRcvd;
}
But it shows me that it returns and reads 0 bytes. On the other hand, when I remove:
if(strstr(buffer, "\r\n"))
break;
it hangs. How to improve it?
One mistake here is that strstr expects a zero-terminated string. A fix:
buffer[chunk] = 0;
if(strstr(buffer, "\r\n"))
break;
However, there may be more data following "\r\n", and that data gets lost here.
A common design pattern for receiving data is:
Have a class that maintains a connection with send/receive buffers.
When it has received data into the receive buffer, it calls a message parsing callback passing buffer and buffer_size.
The message parsing callback consumes all available complete messages in the buffer and returns the number of bytes consumed (as you do).
The message parsing callback calls another callback, passing a complete message to it.
Something like this:
typedef struct
{
int socket;
// Message-parsing callback.
size_t(*on_recv_cb)(void*, void*, size_t);
void* on_recv_cb_data;
unsigned char recv_buffer[256 * 1024];
size_t recv_buffer_size; // Initialized with 0;
} Connection;
void Connection_on_socket_ready_for_read(Connection* self) {
// Assumes a non-blocking socket. Read once to avoid starvation of other sockets.
ssize_t received = read(self->socket, self->recv_buffer + self->recv_buffer_size, sizeof self->recv_buffer - self->recv_buffer_size);
if(received > 0) {
self->recv_buffer_size += received;
size_t consumed = self->on_recv_cb(self->on_recv_cb_data, self->recv_buffer, self->recv_buffer_size);
self->recv_buffer_size -= consumed;
memmove(self->on_recv_cb_data, self->recv_buffer + consumed, self->recv_buffer_size);
}
else if(received < 0) {
if(EAGAIN == errno)
return;
perror("error");
// Handle error.
}
else {
// Handle EOF.
}
}
typedef struct {
// Message callback.
void(*on_message_cb)(void*, char*, size_t);
void* on_message_cb_data;
} MessageParserCrLf;
size_t MessageParserCrLf_on_recv(void* cb_data, void* data, size_t data_size) {
MessageParserCrLf* self = cb_data;
char* message_begin = data;
char* message_end = data;
while(data_size - (message_end - (char*)data) >= 2) {
if(message_end[0] == '\r' && message_end[1] == '\n') {
message_end += 2;
self->on_message_cb(self->on_message_cb_data, message_begin, message_end - message_begin);
message_begin = message_end;
}
else {
++message_end;
}
}
return message_begin - (char*)data;
}
void on_message(void* cb_data, char* message, size_t message_size) {
(void)cb_data; // Unused here.
printf("on_message: %.*s\n", (int)message_size, message);
}
int main() {
MessageParserCrLf message_parser = {on_message, NULL};
Connection connection = {0, MessageParserCrLf_on_recv, &message_parser, {}, 0};
Connection_on_socket_ready_for_read(&connection);
}
Outputs:
[~/src/test]$ cat lines.txt
abc
def
end
[~/src/test]$ ./test < lines.txt
on_message: abc
on_message: def
on_message: end
If there is "\r\n" in the received buffer, the following line will break and end the while loop,
if(strstr(buffer, "\r\n")) break;
but bytesRcvd hasn't increased, so the function will return with bytesRcvd=0.
Try to put bytesRcvd increase before judging like this,
bytesRcvd += (chunk);
if(strstr(buffer, "\r\n"))
break;
I've been playing around libcrypto's BIO, and I can't find a way to detect errors during base64 decoding.
Even if data is complete garbage, BIO_read just returns zero, and error queue - as examined with ERR_get_error - remains empty.
The same issue happens with BIO_FLAGS_BASE64_NO_NL flag: in case of mismatch (that is, data contains newlines, but the flag is set; and vice versa), there's no indication of error, there's just no data.
So, is there a way to catch decoding errors?
static unsigned char *base64_decode(unsigned char *data, size_t len, size_t *out_len)
{
// chain should look like this
// b64 - mem
// so when we read from b64, it gets data from mem and decodes it
BIO *bio_b64;
BIO *bio_mem;
size_t res_capacity;
size_t res_size;
unsigned char *res;
size_t ret;
bio_b64 = BIO_new(BIO_f_base64());
bio_mem = BIO_new_mem_buf(data, len);
res_capacity = 1024;
res_size = 0;
res = malloc(res_capacity);
// don't care about newlines
BIO_set_flags(bio_b64, BIO_FLAGS_BASE64_NO_NL);
BIO_push(bio_b64, bio_mem);
// empty error queue, just in case
while (ERR_get_error() != 0);
while (1) {
ret = BIO_read(bio_b64, &res[res_size], res_capacity - res_size);
if (ret == (res_capacity - res_size)) {
res_size += ret;
res_capacity *= 2;
res = realloc(res, res_capacity);
} else if (ret == 0) {
break;
} else {
res_size += ret;
}
}
if (ERR_get_error() != 0) {
free(res);
return NULL;
}
BIO_free_all(bio_b64);
*out_len = res_size;
return res;
}
Unfortunately BIO_read does't emit an error if malformed data. You can either check expected size (4/3) or reimplement without openssl.
I am currently working on the below requirement.
Here is the requirement: On the server side a large file is divided into 4000-byte blocks (frames). Each block is in turn compressed (using zlib) and sent to client process. For instance, if a file is 12000 bytes in size then it is divided into 3 blocks.
Above file will have 3 blocks => Block-0, Block-1, Block-2
On receipt, client decompresses each block (or frame) and writes to buffer allocated on the heap.When all the blocks corresponding to the entire file is received by the client, then the uncompressed version of the resultant file is written to the disk.
I have written a routine inflateData that does the following based on the block # received:
When the first block is received,
- inflateInit
- inflate
- inflateReset
When the intermediate blocks are received,
- inflate
- inflateReset
When the last block is received,
- inflate
- inflateEnd
With the above routine, Decompression of blocks happens as expected. But the issue that I face is it consumes lots of memory and at some point entire system slows down. When checked with valgrind, memory leak is reported with inflateInit2_. This causes the system resources to be exhausted.
==30359== 57,312 bytes in 6 blocks are possibly lost in loss record 64 of 67
==30359== at 0x4A069EE: malloc (vg_replace_malloc.c:270)
==30359== by 0x3E57808F1E: inflateInit2_ (in /lib64/libz.so.1.2.3)
==30359== by 0x40C220: inflateData (productMaker.c:1668)
Below is the routine inflateData.
int inflateData(
char* const inBuf,
unsigned long inLen,
unsigned int isFirstBlk,
unsigned int isLastBlk,
const char* outBuf,
unsigned long* outLen)
{
int have;
int readsz;
int bsize;
static z_stream zstrm;
int zerr;
int flush;
char out[CHUNK_SZ];
char in[CHUNK_SZ];
int ret,nwrite,idx = -1;
int savedByteCntr=0;
unsigned char *dstBuf;
int firstCall = 1;
int totalBytesIn=0;
int inflatedBytes=0;
int decompByteCounter = 0;
int num=0;
ret = Z_OK;
readsz = 0;
bsize = CHUNK_SZ;
dstBuf = (unsigned char *) outBuf;
if(isFirstBlk){
memset(&zstrm, '\0', sizeof(z_stream));
zstrm.zalloc = Z_NULL;
zstrm.zfree = Z_NULL;
zstrm.opaque = Z_NULL;
if ((zerr = inflateInit(&zstrm)) != Z_OK) {
uerror("ERROR %d inflateInit (%s)",
zerr, decode_zlib_err(zerr));
return -1;
}
}
while(totalBytesIn < inLen ) {
int compChunkSize = ((inLen - totalBytesIn) > 5120) ? 5120 :
(inLen - totalBytesIn);
memcpy(in, inBuf + totalBytesIn, compChunkSize);
zstrm.avail_in = inLen - totalBytesIn;
zstrm.next_in = in ;
zstrm.avail_out = CHUNK_SZ;
zstrm.next_out = out;
inflatedBytes = 0;
while(ret != Z_STREAM_END) {
ret = inflate(&zstrm, Z_NO_FLUSH);
if(ret < 0) {
uerror(" Error %d inflate (%s)", ret, decode_zlib_err(ret));
(void)inflateEnd(&zstrm);
return ret;
}
inflatedBytes = CHUNK_SZ - zstrm.avail_out;
if(inflatedBytes == 0) {
unotice("\n Unable to decompress data - truncated");
break;
}
totalBytesIn += zstrm.total_in;
decompByteCounter += inflatedBytes;
memcpy(dstBuf + savedByteCntr, out, inflatedBytes);
savedByteCntr = decompByteCounter;
}
// Reset inflater for additional input
ret = inflateReset(&zstrm);
if(ret == Z_STREAM_ERROR){
uerror(" Error %d inflateReset (%s)", ret, decode_zlib_err(ret));
(void)inflateEnd(&zstrm);
return ret;
}
}
if(isLastBlk){
ret = inflateEnd(&zstrm);
if(ret < 0) {
uerror("Fail inflateEnd %d [%s] ", ret, decode_zlib_err(ret));
return (ret);
}
}
*outLen = decompByteCounter;
return 0;
}
Thanks in advance for the support.
Thanks,
Sathya.
You are making an error in your use of your inflateData() routine.
First off, using a static variable in this way is a horrible idea. If you call your inflateData() twice with isFirstBlk true without an intermediate call with isLastBlk true, then you will wipe out the reference to the first set of allocations, resulting in a memory leak.
To avoid this sort of error, you should keep track of whether zstrm is initialized or not, and reject any attempt to initialize an already initialized stream. Better still would be to not even have an isFirstBlk, and simply initialize zstrm on the first call and on any call that immediately follows a call with isLastBlk true.
So you are either doing the above, calling twice with isFirstBlk true, or failing to call with isLastBlk true.
I am having an issue with recv. I wrote a function that fills a structure with data, and the length (in bytes) of that data which is read from a socket.
For testing I am just printing the data to stdout byte by byte based on the total amount of bytes read by recv. For some reason the number of bytes being read seems to be correct sometimes and incorrect other times depending on what site I am querying. For example the following code works as intended on some sites:
data->data_sz = 0;
while((i = recv(sock, data->data + data->data_sz, CHUNKSIZE, 0)) > 0)
{
data->data_sz += i;
if(databff - data->data_sz < CHUNKSIZE)
{
databff *= 2;
if(!(tmp = realloc(data->data, databff)))
{
free(data->data);
(void) WSACleanup();
return 0;
}
data->data = tmp;
}
}
i = strsbstr(data->data, "\r\n\r\n") + 4; //i = the position of the first char after header info
if(i >= 0)
{
data->data_sz = data->data_sz - i; //data->data_sz = number of bytes without header info
memmove(data->data, data->data + i, data->data_sz);
if(!(tmp = realloc(data->data, data->data_sz)))
{
free(data->data);
(void)WSACleanup();
return 0;
}
data->data = tmp;
}
else
{
free(data->data);
(void) WSACleanup();
return 0;
}
return 1;
}
To print the data to stdout I just use a for loop:
//t_html->data_sz points to my data->data_sz structure
//t_html->data points to my data->data structure
for(i = 0; i <= t_html->data_sz; i++) (void)fputc((int)t_html->data[i], stdout);
The above code works for some sites but fails on others (for example when querying http://www.google.com I expect the final characters to be </html> but I get </html>l).
Basically my problem is that data->data_sz (the amount of bytes received) is not being calculated correctly, which makes it impossible to correctly use gathered data. I am really at a loss of what to do right now.
EDIT:
here is the strsbstr function which is called in the above code:
int strsbstr(const char *str, const char *sbstr)
{
char *sbstrlc;
if(!(strcmp(str, sbstr))) return 0;
if(!(sbstrlc = strstr(str, sbstr))) return -1;
return (int) (sbstrlc - str);
}
recv(sock, data->data + data->data_sz, CHUNKSIZE, 0) is potentially a problem. Why? Because you may not have CHUNKSIZE room left in your buffer. You have databff - data->data_sz left, actually (assuming data is allocated to a size of databff). It all depends on the initial values of databff and CHUNKSIZE, which I can't see and figure I'd point this out just in case.
Data is not guaranteed to be NULL-character terminated. Your printing loop says i <= t_html->data_sz; which is wrong. It should be i < t_html->data_sz;. If you use <=, you're accessing one past your buffer, which is likely why you get a weird character sometimes, and sometimes not.