How do you properly resize an array using realloc so that the newly allocated array can have the data from the previous array plus the newly received data
int receiver (int soc_desc, char * buffer)
{
char *arr;
size_t received =0 , total_received=0;
char buff[MAX+1];
memset(buff , 0 , MAX+1);
while (1)
{
received = recv(soc_desc, buff , MAX , 0 );
if (received <= 0 )
break;
else
{
total_received = total_received + strlen(buff);
buffer = realloc(buffer, total_received);
printf("Total: %ld received: %ld \n",total_received , received);
strcat(buffer, buff);
}
printf("%s\n",buff);
}
printf("Final result: %s \n", buffer);
in this function, we pass a socket descriptor and a char *buffer = malloc(MAX) we receive data and add it to the allocated buffer and then try to reallocate the buffer for the next chunk of data, is there a way to resize the original mallocd buffer so that I can fit more characters in it without creating a new pointer for realloc each time it is called
when I compile and run this code with valgrind I get
==13850== Address 0x4a5c0e3 is 0 bytes after a block of size 3 alloc'd
==13850== at 0x483DFAF: realloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==13850== by 0x109884: ??? (in /home/User/Desktop/test)
==13850== by 0x109476: ??? (in /home/User/Desktop/test)
==13850== by 0x48870B2: (below main) (libc-start.c:308)
.
.
.
.
==13850== HEAP SUMMARY:
==13850== in use at exit: 27 bytes in 1 blocks
==13850== total heap usage: 22 allocs, 22 frees, 15,807 bytes allocated
since buffer is a function parameter I used free(buffer) outside the function
Compile using -g3 -ggdb3 -Wall -Wextra flags to get line number of error in valgrind and other sort of warnings.
Also, realloc() copies previous data to new one.
Some Points:
total_received = total_received + strlen(buff); can be written as total_received += strlen(buff);
You need to take buffer as char **buffer and then de-reference it like (*buffer), so that modification of buffer can be done permanently in another function's scope
"%zu" is valid format specifier for size_t
Make sure that buffer is heap-allocated in its definition scope
memset(buff , 0 , MAX+1); can be written as char buff[MAX+1] = {};
I don't see any use of arr variable in receiver() function
NOTE: Make sure that new size for buffer is larger than the previous size
Always check whether heap allocation was successful or not, by checking the pointer against NULL, eg., if(!buffer) { /* error */ }
Pass buffer like &buffer [give address of buffer to receiver() function]
received is unsigned long int AKA size_t which means it starts from 0, hence checking for less than 0 is not required, instead check for (received == 0)
Use memcpy() to append buff to *buffer by limiting the length of buff
Append null terminating character at the very end of *buffer
Final Code:
int receiver(int soc_desc, char **buffer)
{
char *arr; // idk
size_t received = 0, total_received = 0;
char buff[MAX + 1] = {}; // every element is now 0
while (1)
{
received = recv(soc_desc, buff, MAX, 0);
if (received == 0)
break;
else
{
total_received += received;
(*buffer) = realloc(*buffer, total_received + 1);
if (*buffer == NULL) // error occurred
{
exit(EXIT_FAILURE);
}
printf("Total: %zu received: %zu\n", total_received, received);
memcpy(*buffer, buff, received);
(*buffer)[total_received + 1] = 0; // nul terminating character
}
printf("%s\n", buff);
}
printf("Final result: %s \n", *buffer);
/*your rest of the code */
According to the documentation:
Reallocates the given area of memory. It must be previously allocated by malloc(), calloc() or realloc() and not yet freed with a call to free or realloc. Otherwise, the results are undefined.
The reallocation is done by either:
a) expanding or contracting the existing area pointed to by ptr, if possible. The contents of the area remain unchanged up to the lesser of the new and old sizes. If the area is expanded, the contents of the new part of the array are undefined. (*)
b) allocating a new memory block of size new_size bytes, copying memory area with size equal the lesser of the new and the old sizes, and freeing the old block.
If there is not enough memory, the old memory block is not freed and null pointer is returned. (#)
The formatting (bold text) and (*) and (#) have been added, and were not in the quoted text.
Solving (#):
bool realloc_buffer(void **buffer, size_t new_size)
{
void *tmp = *buffer;
*buffer = realloc(*buffer, new_size);
if (!*buffer) { // Realloc failed: restore the old pointer.
*buffer = tmp;
return false;
}
return true;
}
Then in your code:
int receiver (int soc_desc, char **buffer)
{
// ...
if (!realloc_buffer(buffer, old_size + total_received + 1)) { // You should know the old size
// Handle failure
}
}
You said:
... so that the newly allocated array can have the data from the previous array plus the newly received data
According to (*), you have to manually append the new data.
int receiver (int soc_desc, char **buffer)
{
// ...
char *copy = malloc(sizeof(char*) * (old_size+1));
// Make a copy of the old buffer
memcpy(copy, buffer, old_size+1);
if (!realloc_buffer(buffer, old_size + total_received + 1)) { // You should know the old size
// Handle failure
} else {
memcpy(buffer + old_size * sizeof(char*), copy, total_received+1);
}
free(copy);
// ...
}
Related
Update: Per the feedback below which I thought I understood, I've amended the code as follows but it is still troublesome:
unsigned int count = 0;
char* filebuffer;
filebuffer = malloc(sizeof(char));
if (!filebuffer)
{
error(500);
return false;
}
while (fread(filebuffer, sizeof(char), 1, file) == 1)
{
count++;
filebuffer = realloc(filebuffer, count * sizeof(char));
printf("%lu\n", (count + 1) * sizeof(char));
}
if (feof(file))
{
*content = filebuffer;
*length = count;
}
Below is some code which is meant to go through a file which is piped through to the function by popen (it's a php file), and store it into a buffer, and then give content* the same pointer and *length the number of bytes read.
However it's not working. Valgrind says:
==7608== Conditional jump or move depends on uninitialised value(s)
==7608== at 0x4C31FCE: strstr (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==7608== by 0x4036C0: interpret (server.c:513)
==7608== by 0x401D66: main (server.c:259)
==7608== Uninitialised value was created by a heap allocation
==7608== at 0x4C2AB80: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==7608== by 0x4C2CF1F: realloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==7608== by 0x40418C: load (server.c:662)
==7608== by 0x403672: interpret (server.c:502)
==7608== by 0x401D66: main (server.c:259)
The code is:
unsigned int count = 0;
char* filebuffer;
filebuffer = malloc(sizeof(char));
if (!filebuffer)
{
printf("oh noes\n");
error(500);
return false;
}
while (fread(filebuffer, sizeof(char), 1, file) == 1)
{
count++;
filebuffer = realloc(NULL, sizeof(filebuffer) + sizeof(char));
}
if (feof(file))
{
*content = filebuffer;
*length = count;
}
Any feedback welcome and thanks in advance.
The argument to realloc is wrong.
sizeof(filebuffer) is equal to sizeof(char*). It does not evaluate to the the size of the array allocated.
You need to keep track of the size using another variable and use that variable. count seems to be that variable but it's not clear from your code what you are doing and what those variables stand for.
Also, when you use
filebuffer = realloc(NULL, some_size);
it is equivalent to
filebuffer = malloc(some_size);
which leads to a lot of leaked memory. To stop the memory leaks, you need to use
filebuffer = realloc(filebuffer, some_size);
Your realloc does not take the buffer of previously allocated one, also you need to track the the size of the buffer.
filebuffer = realloc(NULL, sizeof(filebuffer) + sizeof(char));
It should be
filebuffer = realloc(filebuffer, <the new size>);
But filebuffer = malloc(sizeof(char)); is just looks as bad as it is, you are allocating ONE byte each type. If you don't know the size of the file in advance, I suggest you allocate block by block.
#define BLOCKSIZE 1024
char* filebuffer;
size_t current;
filebuffer = malloc(BLOCKSIZE);
current = BLOCKSIZE;
// in the loop
filebuffer = realloc(filebuffer, BLOCKSIZE + current);
current = BLOCKSIZE + current;
I'm currently writing a method that reads from an allocated block of memory and prints out its contents from a certain offset and up to a specified size, both of which are passed as parameters. I'm using char pointers to accomplish this, but keep getting a malloc error around line
char *content = (char *)malloc(size+1);
Code for the method:
int file_read(char *name, int offset, int size)
{
//First find file and its inode, if existing
int nodeNum = search_cur_dir(name);
if(nodeNum < 0) {
printf("File read error: file does not exist\n");
return -1;
}
//Size check, to avoid overflows/overreads
if(offset > inode[nodeNum].size || size > inode[nodeNum].size || (offset+size) > inode[nodeNum].size) {
printf("File read error: offset and/or size is too large\n");
return -1;
}
int i, read_size, track_size = size, content_offset = 0;
int target_block = offset / BLOCK_SIZE; //Defined as constant 512
int target_index = offset % BLOCK_SIZE;
char *raw_content = (char *)malloc(inode[nodeNum].size+1);
printf("check1\n"); //Debug statment
for(i = target_block; i < (inode[nodeNum].blockCount-(size/BLOCK_SIZE)); i++) {
disk_read(inode[nodeNum].directBlock[i], raw_content+content_offset);
content_offset += BLOCK_SIZE;
}
printf("check2\n"); //Debug statment
char *content = (char *)malloc(size+1);
memcpy(content, raw_content+target_index, size);
printf("%s\n", content);
free(raw_content);
free(content);
return 0;
}
and code for disk_read:
char disk[MAX_BLOCK][BLOCK_SIZE]; //Defined as 4096 and 512, respectively
int disk_read(int block, char *buf)
{
if(block < 0 || block >= MAX_BLOCK) {
printf("disk_read error\n");
return -1;
}
memcpy(buf, disk[block], BLOCK_SIZE);
return 0;
}
structure for node
typedef struct {
TYPE type;
int owner;
int group;
struct timeval lastAccess;
struct timeval created;
int size;
int blockCount;
int directBlock[10];
int indirectBlock;
char padding[24];
} Inode; // 128 byte
The error I get when using this method is one of memory corruption
*** glibc detected *** ./fs_sim: malloc(): memory corruption (fast): 0x00000000009f1030 ***
Now the strange part is, firstly this only occurs after I have used the method a few times - for the first two or three attempts it will work and then the error occurs. For instance, here is an example test run:
% read new 0 5
z12qY
% read new 0 4
z12q
% read new 0 3
*** glibc detected *** ./fs_sim: malloc(): memory corruption (fast): 0x00000000009f1030 ***
Even stranger still, this error disappears completely when I comment out
free(raw_content);
free(content);
Even through this would tie up the memory. I've read through previous posts regarding malloc memory corruption and understand this usually results from overwriting memory bounds or under allocating space, but I can't see where I could be doing this. I've attempted other sizes for malloc as well and these produced the best results when I commented out the lines freeing both pointers. Does anyone see what I could be missing? And why does this occur so inconsistently?
Code allocates space for characters and a null character, but does not insure the array is terminated with a null character before printing as a string.
char *content = (char *)malloc(size+1);
memcpy(content, raw_content+target_index, size);
// add
content[size] = '\0';
printf("%s\n", content);
Likely other issues too.
[Edit]
OP code is prone to mis-coding and dependent on inode[] to have coherent values (.blockCount . size). Clarify and simplify by determining the loop count and allocating per that count.
int loop_count = (inode[nodeNum].blockCount-(size/BLOCK_SIZE)) - target_block;
char *raw_content = malloc(sizeof *raw_content * loop_count * BLOCK_SIZE);
assert(raw_count);
for (loop = 0; loop < loop_count; loop++) {
i = target_block + loop;
disk_read(inode[nodeNum].directBlock[i], raw_content + content_offset);
content_offset += BLOCK_SIZE;
}
Also recommend checking the success of disk_read()
gcc (GCC) 4.7.2
valgrind-3.8.1
c89
Hello,
==1160== Invalid read of size 1
==1160== at 0x8048C94: get_input_values (parse_cmd_input.c:278)
==1160== by 0x8048BA0: parse_input (parse_cmd_input.c:245)
==1160== by 0x80489A1: main (parse_cmd_input.c:50)
==1160== Address 0x40ef02c is 0 bytes after a block of size 4 alloc'd
==1160== at 0x40072C5: calloc (vg_replace_malloc.c:593)
==1160== by 0x8048B28: parse_input (parse_cmd_input.c:239)
==1160== by 0x80489A1: main (parse_cmd_input.c:50)
So its saying the address is reading a zero bytes of a allocated size of 4, and is trying to read 1 byte from it. However, I haven't over stepped the bounds of the array and I am accessing element 0.
I have checked with gdb, and element zero contains a character.
My program doesn't crash, and seems to work fine. But it might cause a problem on a production server.
I am not sure if I am correct here:
Should this be:
cpy_input = calloc(strlen(input) + 1, sizeof(char*));
or:
cpy_input = calloc(strlen(input) + 1, sizeof(char));
A char is 1 byte, and a pointer to a char is 4 bytes on my system.
The string passed in would be something like this "25 b"
int parse_input(const char *input)
{
char *cpy_input = NULL;
int has_error = -1;
if(strlen(input) == 0) {
LOG_ERR("FAILED: Empty string");
return -1;
}
cpy_input = calloc(strlen(input) + 1, sizeof(char));
apr_cpystrn(cpy_input, input, sizeof(cpy_input));
LOG_INFO("[ %s ]", cpy_input);
memset(&channel, 0, sizeof channel);
has_error = get_input_values(cpy_input, &channel);
free(cpy_input);
return has_error;
}
int get_input_values(const char *str, channel_t *channel)
{
size_t i = 0;
size_t k = 0;
int upper_flag = 0;
/* Indicates no digits or command found*/
channel->lower = -1;
channel->upper = -1;
channel->cmd = -1;
#define DIG_BUFFER_SIZE 32
char dig_buffer_lower[DIG_BUFFER_SIZE];
char dig_buffer_upper[DIG_BUFFER_SIZE];
if(strlen(str) == 0) {
LOG_ERR("FAILED: Empty string");
return -1;
}
memset(dig_buffer_lower, 0, DIG_BUFFER_SIZE);
memset(dig_buffer_upper, 0, DIG_BUFFER_SIZE);
LOG_INFO("SIZE %d %d", sizeof(char), sizeof(char*));
/* Increament and check for digits */
for(i = 0; i < DIG_BUFFER_SIZE; i++) {
switch(str[i]) {
case 32: /* ignore space */
continue;
case 45: /* '-' Start upper bounds */
LOG_DEBUG("Found a '-' check upper value");
/* Having a second '-' means the string is invalid */
if(!upper_flag) {
upper_flag = 1;
k = 0;
}
break;
} /* switch */
/* Insert on digits into the lower and upper values */
if(isdigit(str[i])) {
if(upper_flag) {
dig_buffer_upper[k++] = str[i];
LOG_DEBUG("dig_buffer_upper[%d] %s", i, dig_buffer_upper);
}
else {
/* Add to digit buffer */
dig_buffer_lower[k++] = str[i];
LOG_DEBUG("dig_buffer_lower[%d] %s", i, dig_buffer_lower);
}
}
} /* for loop */
Many thanks for any suggestions,
sizeof(cpy_input) is just sizeof(char *), and not the string length. Instead, say:
apr_cpystrn(cpy_input, input, strlen(input) + 1);
Or better, use a naked strcpy or equivalent. Also there's no need to zero out the array with calloc, since you're just about to overwrite it anyway. And since sizeof(char) is 1 by definition, you can allocate the array with:
cpy_input = malloc(strlen(input) + 1);
(Think about strings for a minute: You're already at the mercy of having a null terminator at a sensible place, or strlen will either crash or return a huge value. Once you trust the result of strlen, you are guaranteed to allocate enough memory to strcpy the string and the null terminator. Alternatively, you can use memcpy for a possibly even more efficient copy, since you know the size.)
Ok, maybe I'm missing something, but your for loop will iterate over 0 .. DIG_BUFFER_SIZE-1, reading from str[i]. I don't see what would cause that loop to break out early, especially since it seems to immediately wrap a switch, and so any break inside the switch would exit the switch, but not the for.
Your calloc(strlen(input) + 1, sizeof(char)); correctly allocates storage for the exact length of input. The code downstream in get_input_values doesn't seem to stop if the string is shorter than DIG_BUFFER_SIZE.
(I'd love to be proven wrong, but to know, we need to see more code.)
While working on a program which requires frequent memory allocation I came across behaviour I cannot explain. I've implemented a work around but I am curious to why my previous implementation didn't work. Here's the situation:
Memory reallocation of a pointer works
This may not be best practice (and if so please let me knwow) but I recall that realloc can allocate new memory if the pointer passed in is NULL. Below is an example where I read file data into a temporary buffer, then allocate appropriate size for *data and memcopy content
I have a file structure like so
typedef struct _my_file {
int size;
char *data;
}
And the mem reallocation and copy code like so:
// cycle through decompressed file until end is reached
while ((read_size = gzread(fh, buf, sizeof(buf))) != 0 && read_size != -1) {
// allocate/reallocate memory to fit newly read buffer
if ((tmp_data = realloc(file->data, sizeof(char *)*(file->size+read_size))) == (char *)NULL) {
printf("Memory reallocation error for requested size %d.\n", file->size+read_size);
// if memory was previous allocated but realloc failed this time, free memory!
if (file->size > 0)
free(file->data);
return FH_REALLOC_ERROR;
}
// update pointer to potentially new address (man realloc)
file->data = tmp_data;
// copy data from temporary buffer
memcpy(file->data + file->size, buf, read_size);
// update total read file size
file->size += read_size;
}
Memory reallocation of pointer to pointer fails
However, here is where I'm confused. Using the same thought that reallocation of a NULL pointer will allocate new memory, I parse a string of arguments and for each argument I allocate a pointer to a pointer, then allocate a pointer that is pointed by that pointer to a pointer. Maybe code is easier to explain:
This is the structure:
typedef struct _arguments {
unsigned short int options; // options bitmap
char **regexes; // array of regexes
unsigned int nregexes; // number of regexes
char *logmatch; // log file match pattern
unsigned int limit; // log match limit
char *argv0; // executable name
} arguments;
And the memory allocation code:
int i = 0;
int len;
char **tmp;
while (strcmp(argv[i+regindex], "-logs") != 0) {
len = strlen(argv[i+regindex]);
if((tmp = realloc(args->regexes, sizeof(char **)*(i+1))) == (char **)NULL) {
printf("Cannot allocate memory for regex patterns array.\n");
return -1;
}
args->regexes = tmp;
tmp = NULL;
if((args->regexes[i] = (char *)malloc(sizeof(char *)*(len+1))) == (char *)NULL) {
printf("Cannot allocate memory for regex pattern.\n");
return -1;
}
strcpy(args->regexes[i], argv[i+regindex]);
i++;
}
When I compile and run this I get a run time error "realloc: invalid pointer "
I must be missing something obvious but after not accomplishing much trying to debug and searching for solutions online for 5 hours now, I just ran two loops, one counts the numbers of arguments and mallocs enough space for it, and the second loop allocates space for the arguments and strcpys it.
Any explanation to this behaviour is much appreciated! I really am curious to know why.
First fragment:
// cycle through decompressed file until end is reached
while (1) {
char **tmp_data;
read_size = gzread(fh, buf, sizeof buf);
if (read_size <= 0) break;
// allocate/reallocate memory to fit newly read buffer
tmp_data = realloc(file->data, (file->size+read_size) * sizeof *tmp_data );
if ( !tmp_data ) {
printf("Memory reallocation error for requested size %d.\n"
, file->size+read_size);
if (file->data) {
free(file->data)
file->data = NULL;
file->size = 0;
}
return FH_REALLOC_ERROR;
}
file->data = tmp_data;
// copy data from temporary buffer
memcpy(file->data + file->size, buf, read_size);
// update total read file size
file->size += read_size;
}
Second fragment:
unsigned i; // BTW this variable is already present as args->nregexes;
for(i =0; strcmp(argv[i+regindex], "-logs"); i++) {
char **tmp;
tmp = realloc(args->regexes, (i+1) * sizeof *tmp );
if (!tmp) {
printf("Cannot allocate memory for regex patterns array.\n");
return -1;
}
args->regexes = tmp;
args->regexes[i] = strdup( argv[i+regindex] );
if ( !args->regexes[i] ) {
printf("Cannot allocate memory for regex pattern.\n");
return -1;
}
...
return 0;
}
A few notes:
the syntax ptr = malloc ( CNT * sizeof *ptr); is more robust than the sizeof(type) variant.
strdup() does exactly the same as your malloc+strcpy()
the for(;;) loop is less error prone than a while() loop with a loose i++; at the end of the loop body. (it also makes clear that the loopcondition is never checked)
to me if ( !ptr ) {} is easyer to read than if (ptr != NULL) {}
the casts are not needed and sometimes unwanted.
I get an
malloc: *** error for object 0x1001012f8: incorrect checksum for freed object
- object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
error in the following function:
char* substr(const char* source, const char* start, const char* end) {
char *path_start, *path_end, *path;
int path_len, needle_len = strlen(start);
path_start = strcasestr(source, start);
if (path_start != NULL) {
path_start += needle_len;
path_end = strcasestr(path_start, end);
path_len = path_end - path_start;
path = malloc(path_len + 1);
strncpy(path, path_start, path_len);
path[path_len] = '\0';
} else {
path = NULL;
}
return path;
}
How can I make this work? When I rewrite the function to allocate the memory using path[path_len + 1] it works just fine.
Now, the part I don't understand is, that I never even call free in any point of my application, as every allocated memory is needed for the program until it exists (which, AFAIK will invalidate every allocated memory anyway?!)
So, how can a freed object be corrupt if I never free one?
The function is called in this one:
char *read_response(int sock) {
int bytes_read;
char *buf = (char*)malloc(BUF_SIZE);
char *cur_position = buf;
while ((bytes_read = read(sock, cur_position, BUF_SIZE)) > 0) {
cur_position += bytes_read;
buf = realloc(buf, sizeof(buf) + BUF_SIZE);
}
int status = atoi(substr(buf, "HTTP/1.0 ", " "));
There is the realloc, am I using that wrong? I want to read the complete server response, so I have to reallocate after every iteration, don't I?
In read_response, you are probably overwriting the end of the buffer pointed to by buf.
The problem is that buf is a pointer, so sizeof(buf) will return the size of a pointer (probably 4 or 8 depending on your CPU). You are using sizeof as if buf were an array, which is not really the same thing as a pointer in C although they seem interchangeable in some contexts.
Instead of using sizeof, you need to be keeping track of the last size that you allocated for buf, and add BUF_SIZE to that each time you enlarge the buffer.
You should also consider that the read operation may be returning considerably fewer characters than BUF_SIZE on each call, so doing a realloc on buf in each iteration may be overkill. That probably won't cause any problems for you in terms of correctness, though; it will just use more memory than it needs to.
I would do something more like the code below.
#define MIN_BUF_SPACE_THRESHOLD (BUF_SIZE / 2)
char *read_response(int sock) {
int bytes_read;
char *buf = (char*)malloc(BUF_SIZE);
int cur_position = 0;
int space_left = BUF_SIZE;
if (buf == NULL) {
exit(1); /* or try to cope with out-of-memory situation */
}
while ((bytes_read = read(sock, buf + cur_position, space_left)) > 0) {
cur_position += bytes_read;
space_left -= bytes_read;
if (space_left < MIN_BUF_SPACE_THRESHOLD) {
buf = realloc(buf, cur_position + space_left + BUF_SIZE);
if (buf == NULL) {
exit(1); /* or try to cope with out-of-memory situation */
}
space_left += BUF_SIZE;
}
}
This version has the advantage of not trying to allocate more space if the read call comes back with only a few bytes of data.
This line
buf = realloc(buf, sizeof(buf) + BUF_SIZE);
is wrong. All reallocations are with the same size, BUF_SIZE + sizeof(char*). Then you are writing to unallocated memory when reading from the socket, overwriting memory previously freed by a realloc.
You have to keep track of the allocated size,
size_t current_buf_size = BUF_SIZE;
/* ... */
char *temp = realloc(buf, current_buf_size + BUF_SIZE);
if (temp == NULL) {
/* die or repair */
}
buf = temp;