I'm trying to write a program that uses some basic threading for an assignment. Below are the relevant snippets that I think are causing the problem.
The program runs fine with 25 or less threads but usually results in a segfault when 26 or more are used. This led me to think that my malloc statement were incorrect. Can anyone nudge me in the right direction? If you need more code to be posted, I'd be happy to provide it.
Also, these problems only come up when I run the program on my school's student machines. It appears to work fine my local machine. Might anyone know why?
Thanks for your time!
...
struct thread_args {
struct bitmap *bm;
double xmin;
double xmax;
double ymin;
double ymax;
int max;
int start;
int end;
};
...
int num_threads; //Given by user input
struct bitmap *bm = bitmap_create(500, 500); //All threads share the same bitmap
int i;
pthread_t *thread_id = malloc(num_threads * sizeof(*thread_id));
struct thread_args *args = malloc(num_threads * sizeof(*args));
for (i = 0; i < num_threads; i++) {
args[i].bm = bm;
args[i].xmin = xcenter-scale;
args[i].xmax = xcenter+scale;
args[i].ymin = ycenter-scale;
args[i].ymax = ycenter+scale;
args[i].max = max;
args[i].start = bitmap_height(bm) * i / num_threads;
args[i].end = bitmap_height(bm) * (i + 1) / num_threads;
pthread_create(&thread_id[i], NULL, compute_image, &args[i]);
}
for (i = 0; i < num_threads; i++) {
pthread_join(thread_id[i], NULL);
}
...
void* compute_image(void *arg)
{
struct thread_args* args = (struct thread_args*) arg;
int i,j;
int width = bitmap_width(args->bm);
int height = bitmap_height(args->bm);
// For every pixel in the image...
for(j=args->start;j<args->end;j++) {
for(i=0;i<width;i++) {
// Determine the point in x,y space for that pixel.
double x = args->xmin + i*(args->xmax-args->xmin)/width;
double y = args->ymin + j*(args->ymax-args->ymin)/height;
// Compute the iterations at that point.
int iters = iterations_at_point(x,y,args->max);
// Set the pixel in the bitmap.
bitmap_set(args->bm,i,j,iters);
}
}
return 0;
}
...
valgrind log
==24919==
==24919== HEAP SUMMARY:
==24919== in use at exit: 1,000,000 bytes in 1 blocks
==24919== total heap usage: 56 allocs, 55 frees, 1,018,884 bytes allocated
==24919==
==24919== Searching for pointers to 1 not-freed blocks
==24919== Checked 87,112 bytes
==24919==
==24919== 1,000,000 bytes in 1 blocks are definitely lost in loss record 1 of 1
==24919== at 0x4A069EE: malloc (vg_replace_malloc.c:270)
==24919== by 0x401256: bitmap_create (bitmap.c:21)
==24919== by 0x400CEC: main (mandel.c:103)
==24919==
==24919== LEAK SUMMARY:
==24919== definitely lost: 1,000,000 bytes in 1 blocks
==24919== indirectly lost: 0 bytes in 0 blocks
==24919== possibly lost: 0 bytes in 0 blocks
==24919== still reachable: 0 bytes in 0 blocks
==24919== suppressed: 0 bytes in 0 blocks
==24919==
==24919== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 6 from 6)
--24919--
--24919-- used_suppression: 4 U1004-ARM-_dl_relocate_object
--24919-- used_suppression: 2 glibc-2.5.x-on-SUSE-10.2-(PPC)-2a
==24919==
==24919== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 6 from 6)
Edit: Added compute_image code and valgrind log although the log is missing the error messages it was showing earlier today. The 1,000,000 bytes lost is something I know about.
Make sure that pthread_create is not returning an error. The machine has a global limit on the number of threads that can be spawned, and is probably fluctuating close to that limit. If you fail to spawn a thread, you will have a garbage pthread_t, which will likely cause pthread_join to blow up.
Related
For my C program I have tried to allocate memory for a double array(ssi) which I then access in another function (spslicer). If I allocate this memory before I call a different function (kpmdos) which should be unrelated I get the error:
*** glibc detected *** ./SpecSli.ex: double free or corruption (!prev): 0x0000000000a86e40 ***
and if I allocate it after I get the error:
malloc.c:3830: _int_malloc: Assertion `(unsigned long)(size) >= (unsigned long)(nb)' failed.
Based on this I think that I did something wrong in kpmdos when I allocated the memory but I cannot see anything wrong:
void kpmdos(csptr A, int Mdeg, double nvec, double ctr, double wid, int n, double *mu){
// initialize variables
double *w = malloc(n*sizeof(double));
//free(w);
fill_rand(n, w);
double *v = malloc(n*sizeof(double));
divide_array(w, norm(w, n), v, n);
double *vkm1 = malloc(n*sizeof(double));
fill(n, 0, vkm1);
double *vk = malloc(n*sizeof(double));
memcpy(vk,v, n*sizeof(double));
double *temp = malloc(n*sizeof(double));
double *vkp1 = malloc(n*sizeof(double));
double thet = pi/(Mdeg+1);
double a1 = 1/(Mdeg+2.0);
double a2 = sin(thet);
double jac;
double scal;
Then I do some other coding with no memory allocation.
free(w);
free(vkp1);
free(vkm1);
free(temp);
free(vk);
free(v);
}
The output using valgrind when it is allocated before the function is:
> > ==7045== HEAP SUMMARY:
> ==7045== in use at exit: 46,452 bytes in 235 blocks
> ==7045== total heap usage: 351 allocs, 116 frees, 66,700 bytes allocated
> ==7045==
> ==7045== LEAK SUMMARY:
> ==7045== definitely lost: 22,331 bytes in 146 blocks
> ==7045== indirectly lost: 0 bytes in 0 blocks
> ==7045== possibly lost: 0 bytes in 0 blocks
> ==7045== still reachable: 24,121 bytes in 89 blocks
> ==7045== suppressed: 0 bytes in 0 blocks
> ==7045== Rerun with --leak-check=full to see details of leaked memory
> ==7045==
> ==7045== For counts of detected and suppressed errors, rerun with: -v
> ==7045== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 2 from 2)
Please let me know if you know the problem or if you need more information.
Thank you all for your help. I finally figured out most of it. During my coding which I did not post I had a function which would change all the variables in an array to something different and for this function I was inputting the wrong array size.
I have made a minimal-working example of how to add elements to an array with realloc. This will be expanded in a future program that has many more elements.
#include <stdio.h>//printf
#include <stdlib.h>//malloc, realloc
int main() {
//BEGIN REALLOCATE-ABLE ARRAY
unsigned int *array, loop_variable;
const unsigned int ORIGINAL_ARRAY_SIZE = 4, REALLOC_INDICES = 99;
array = malloc(ORIGINAL_ARRAY_SIZE*sizeof(unsigned int));
for (loop_variable = 0; loop_variable < ORIGINAL_ARRAY_SIZE; loop_variable++) {
array[loop_variable] = loop_variable;
}
//BEGIN REALLOCATION
for (loop_variable = 1; loop_variable < REALLOC_INDICES; loop_variable++) {
array = realloc(array,sizeof(unsigned int)*(ORIGINAL_ARRAY_SIZE+loop_variable));
array[ORIGINAL_ARRAY_SIZE+loop_variable-1] = 2*(ORIGINAL_ARRAY_SIZE+loop_variable-1);
printf("reallocate array[%d] = %d\n",ORIGINAL_ARRAY_SIZE+loop_variable-1,array[ORIGINAL_ARRAY_SIZE+loop_variable-1]);
}
//BEGIN PRINTING ARRAY VALUES
for (loop_variable = 0; loop_variable < ORIGINAL_ARRAY_SIZE+REALLOC_INDICES-1; loop_variable++) {
printf("array[%d] = %d\n",loop_variable,array[loop_variable]);
}
//BEGIN FREE ARRAY
free(array); array = NULL;
return 0;
}
This should compile on any computer with gcc or clang. I then run this program on valgrind to ensure there are no memory leaks, and I get this:
==10791== HEAP SUMMARY:
==10791== in use at exit: 0 bytes in 0 blocks
==10791== total heap usage: 99 allocs, 99 frees, 20,988 bytes allocated
==10791==
==10791== All heap blocks were freed -- no leaks are possible
==10791==
==10791== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
==10791== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
However, what troubles me is that I'm using 20,988 bytes for a 101-element array. The memory use grows exponentially as the array gets bigger, instead of linearly by 4 bytes/element.
If I'm understanding Valgrind's output correctly, this array should have 4*101 elements = 404 bytes memory size, but appears to use about 50 times as much memory as it should. This is a trivial problem for such a small program, but more meaningful programs will run out of memory on this computer.
My question: is this array really using 20,988 bytes, or is Valgrind double-counting the memory?
Is there a more memory-efficient way to do this? I can't understand other examples of realloc, though I have tried to follow them as closely as I can and make this question relevant to as many users as possible.
==10791== in use at exit: 0 bytes in 0 blocks
==10791== total heap usage: 99 allocs, 99 frees, 20,988 bytes allocated
is this array really using 20,988 bytes
No, it's using 0 bytes.
“allocated” obviously means “ever allocated”, including bytes that have been freed, since the amount of memory “still allocated” is zero (and another line tells you that).
The answer, thanks to user Pascal Cuoq, is that Valgrind will sum the memory allocations for every allocation. This is why I was confused, and thought that realloc was using too much memory.
I thought that if you want to find the size of the array, you can comment out the line that says
free(array); array = NULL;
to intentionally introduce a memory bug, and Valgrind will output
==11699== LEAK SUMMARY:
==11699== definitely lost: 408 bytes in 1 blocks
==11699== indirectly lost: 0 bytes in 0 blocks
==11699== possibly lost: 0 bytes in 0 blocks
==11699== still reachable: 0 bytes in 0 blocks
==11699== suppressed: 0 bytes in 0 blocks
==11699==
==11699== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
==11699== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
which gives you the true size of the array at its last iteration. There is probably a better way to do this, but my point is made.
I added improvements suggested by other users here. I had a very difficult time finding good, minimal working examples of realloc on search engines, and for future use of anyone finding this on a search engine, this is very basic minimal working example of a good, but probably not best, how to use realloc in C:
#include <stdio.h>//include printf function
#include <stdlib.h>//include malloc, realloc functions
int main() {
/*BEGIN REALLOCATE-ABLE ARRAY*/
unsigned int *array, loop_variable;
const unsigned int ORIGINAL_ARRAY_SIZE = 4, REALLOC_INDICES = 99999;
array = malloc(ORIGINAL_ARRAY_SIZE*sizeof(unsigned int));
for (loop_variable = 0; loop_variable < ORIGINAL_ARRAY_SIZE; loop_variable++) {
array[loop_variable] = loop_variable;
}
/*BEGIN REALLOCATION*/
for (loop_variable = 1; loop_variable < REALLOC_INDICES; loop_variable++) {
array = realloc(array,sizeof(unsigned int)*(ORIGINAL_ARRAY_SIZE+loop_variable));
if (array == NULL) {
printf("Array variable %d failed to reallocate :,-(\n",loop_variable);
exit(EXIT_FAILURE);
}
array[ORIGINAL_ARRAY_SIZE+loop_variable-1] = 2*(ORIGINAL_ARRAY_SIZE+loop_variable-1);
printf("reallocate array[%d] = %d\n",ORIGINAL_ARRAY_SIZE+loop_variable-1,array[ORIGINAL_ARRAY_SIZE+loop_variable-1]);
}
/*BEGIN PRINTING ARRAY VALUES*/
for (loop_variable = 0; loop_variable < ORIGINAL_ARRAY_SIZE+REALLOC_INDICES-1; loop_variable++) {
printf("array[%d] = %d\n",loop_variable,array[loop_variable]);
}
/*BEGIN FREE ARRAY*/
free(array); array = NULL;
return 0;
}
I've seen another question for allocating and freeing multi-dimensional arrays, but I suspect that it does not free correctly. For testing I made this small code extracted from my main code.
I compiled it under MacOS X.10 with XCode or gcc 4.9 with same results:
Faulty code
It runs 200000 times and the memory consumption grows up to 20GB!:
#include <stdlib.h>
typedef struct{
int lonSize;
int latSize;
double **grid;
}raf09_grid_t;
static raf09_grid_t raf09_grid;
void free_raf09_grid() {
if (raf09_grid.grid != NULL) {
int i;
for (i = 0; i < raf09_grid.lonSize; ++i) {
free(raf09_grid.grid[i]);
}
free(raf09_grid.grid);
}
raf09_grid.latSize = 0;
raf09_grid.lonSize = 0;
}
void get_raf09_grid() {
int nbElLat=381;
int nbElLon=421;
raf09_grid.grid = malloc(nbElLon*sizeof(double*));
int it;
for(it=0;it<nbElLon;it++)
raf09_grid.grid[it] = malloc(nbElLat*sizeof(double));
int i,j;
for(i=0;i<420;i++) {
for(j=0;j<380;j++) {
raf09_grid.grid[i][j]=0.0;
}
}
}
int main (int argc, char *argv[]) {
int i=0;
for (i=0;i<20000;i++) {
get_raf09_grid();
free_raf09_grid();
}
return 0;
}
I'm newbie so I suspect that my freeing is not correct...
Corrected code
With your help I corrected my code, it is now corrects and takes only 10M in ram:
#include <stdlib.h>
typedef struct{
int lonSize;
int latSize;
double **grid;
}raf09_grid_t;
static raf09_grid_t raf09_grid;
void free_raf09_grid() {
if (raf09_grid.grid != NULL) {
int i;
for (i = 0; i < raf09_grid.lonSize; ++i) {
free(raf09_grid.grid[i]);
}
free(raf09_grid.grid);
}
raf09_grid.latSize = 0;
raf09_grid.lonSize = 0;
}
void get_raf09_grid() {
raf09_grid.latSize=381;
raf09_grid.lonSize=421;
raf09_grid.grid = malloc(raf09_grid.lonSize*sizeof(double*));
int it;
for(it=0;it<raf09_grid.lonSize;it++)
raf09_grid.grid[it] = malloc(raf09_grid.latSize*sizeof(double));
int i,j;
for(i=0;i<420;i++) {
for(j=0;j<380;j++) {
raf09_grid.grid[i][j]=0.0;
}
}
}
int main (int argc, char *argv[]) {
int i=0;
for (i=0;i<20000;i++) {
get_raf09_grid();
free_raf09_grid();
}
return 0;
}
Valgrind is an invaluable tool in tracing memory leaks. Compiling your source code with debug information, and running it with:
valgrind --leak-check=full -v ./a.out
will give the following summary:
==7033== HEAP SUMMARY:
==7033== in use at exit: 1,283,208,000 bytes in 421,000 blocks
==7033== total heap usage: 422,000 allocs, 1,000 frees, 1,286,576,000 bytes allocated
==7033==
==7033== Searching for pointers to 421,000 not-freed blocks
==7033== Checked 92,040 bytes
==7033==
==7033== 18,288 bytes in 6 blocks are possibly lost in loss record 1 of 2
==7033== at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==7033== by 0x400611: get_raf09_grid (grid.c:27)
==7033== by 0x4006A8: main (grid.c:39)
==7033==
==7033== 1,283,189,712 bytes in 420,994 blocks are definitely lost in loss record 2 of 2
==7033== at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==7033== by 0x400611: get_raf09_grid (grid.c:27)
==7033== by 0x4006A8: main (grid.c:39)
==7033==
==7033== LEAK SUMMARY:
==7033== definitely lost: 1,283,189,712 bytes in 420,994 blocks
==7033== indirectly lost: 0 bytes in 0 blocks
==7033== possibly lost: 18,288 bytes in 6 blocks
==7033== still reachable: 0 bytes in 0 blocks
==7033== suppressed: 0 bytes in 0 blocks
==7033==
==7033== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 2 from 2)
--7033--
--7033-- used_suppression: 2 dl-hack3-cond-1
==7033==
==7033== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 2 from 2)
The link to line 27 shows that there is a problem with this statement:
raf09_grid.grid[it] = malloc(nbElLat*sizeof(double));
You are allocating more memory here than is being freed later in the program.
Update the raf09_grid.lonSize variable in get_raf09_grid() to equal nbElLon, and it fixes the problem.
In general, every malloc should have a corresponding free. Once you know which malloc is leaking, then you can find the code that is supposed to free that variable, and debug from there.
Note: I reduced the loop from 20000 to 1000, but it will give you the same information.
After reading through yoiur question, here are a few suggestions:
In the code:
for(i=0;i<420;i++) {
for(j=0;j<380;j++) {
raf09_grid.grid[i][j]=0.0;
}
you are looping i and j for 420 and 380, while you've defined :
int nbElLat=381;
int nbElLon=421;
So, you never process the 420th iteration for the Longitudes. The same with missing out on the 380th loop for Latitude.
If that's a desirable thing, its fine, else you should fix it to something like this:
for(i=0;i<nbElLon;i++) { //better use macros or variable names than just plain magic numbers
for(j=0;j<nbElLat;j++) {
raf09_grid.grid[i][j]=0.0;
}
Second, in your free_raf09_grid() function, you use:
for (i = 0; i < raf09_grid.lonSize; ++i) {
but you haven't initialized that variable anywhere.
Perhaps in function get_raf09_grid() just before the declaration of int i,j do this:
raf09_grid.lonSize = nbElLon;
raf09_grid.latSize = nbElLat;
The third important one.
In the lines below:
raf09_grid.grid = malloc(nbElLon*sizeof(double*));
int it;
for(it=0;it<nbElLon;it++)
raf09_grid.grid[it] = malloc(nbElLat*sizeof(double));
you should check if the mallocs have returned success or not aka check for NULL.
your for loop uses lonSize as a parameter, but it doesn't get updated anywhere
I need a string array which dynamically resizes when more items are added to it. I got the basic code working, but valgrind reports memory leaks.
The implementation which should work with static and dynamically allocated strings looks like this:
typedef struct {
char **items;
int num;
} StringArray;
StringArray* string_array_init() {
StringArray *arr = malloc(sizeof(StringArray));
arr->items = NULL;
arr->num = 0;
return arr;
}
void string_array_add(StringArray *arr, char *str, int str_len) {
void *new_items = realloc(arr->items, (arr->num + 1) * sizeof(char *));
arr->items = (char**)new_items;
arr->items[arr->num] = strndup(str, str_len);
arr->num++;
}
void string_array_cleanup(StringArray *arr) {
int i;
for (i = 0; i < arr->num ; i++) {
free(arr->items[i]);
}
free(arr);
}
int main() {
StringArray *arr = string_array_init();
string_array_add(arr, "item 1", strlen("item 1"));
string_array_add(arr, "item 2", strlen("item 2"));
string_array_add(arr, "item 3", strlen("item 3"));
string_array_cleanup(arr);
return 0;
}
Valgrind reports:
==31443== HEAP SUMMARY:
==31443== in use at exit: 12 bytes in 1 blocks
==31443== total heap usage: 7 allocs, 6 frees, 53 bytes allocated
==31443==
==31443== 12 bytes in 1 blocks are definitely lost in loss record 1 of 1
==31443== at 0x4025CCD: realloc (vg_replace_malloc.c:525)
==31443== by 0x80484A6: string_array_add (in ...)
==31443== by 0x8048593: main (in ...)
==31443==
==31443== LEAK SUMMARY:
==31443== definitely lost: 12 bytes in 1 blocks
==31443== indirectly lost: 0 bytes in 0 blocks
==31443== possibly lost: 0 bytes in 0 blocks
==31443== still reachable: 0 bytes in 0 blocks
==31443== suppressed: 0 bytes in 0 blocks
==31443==
Why is realloc leaking and how can I fix it? I thoght freeing every string separately and then freeing the struct would be enough, but I´m missing something.
You free the strings contained in arr->items, but you don't free arr->items itself. (You allocated it in string_array_add).
The problem is with this line:
arr->items = (char**)new_items;
You overwrite the old pointer. You need to realloc this pointer and not new_items.
For the following sample of code there is memory leak. How can we prevent memory leak in the following case:
1 #include <stdio.h>
2 #include <stdlib.h>
3
4 typedef struct sample_help {
5 int *value;
6 void **pointers;
7 }*sample,sample_node;
8
9
10 sample main(){
11 sample ABC=NULL;
12 sample XYZ=NULL;
13 sample kanchi = NULL;
14
15 ABC = malloc(sizeof(sample_node));
16 XYZ = malloc(sizeof(sample_node));
17 ABC->pointers = malloc(5*sizeof(void *));
18 XYZ->pointers = malloc(5*sizeof(void *));
19 ABC->value = malloc(5*sizeof(int));
20 XYZ->value = malloc(5*sizeof(int));
21
22 ABC->value[0] = 10;
23 ABC->value[1] = 20;
24
25 XYZ->pointers[0] = ABC;
26 kanchi = XYZ->pointers[0];
27
28 printf("::::%d\n",XYZ->pointers[0]);
29 printf("kanchi1:::::%d\n",kanchi->value[0]);
30
31
32 return XYZ;
33 }
34
Following is the output of valgrind.
==27448==
==27448== HEAP SUMMARY:
==27448== in use at exit: 152 bytes in 6 blocks
==27448== total heap usage: 6 allocs, 0 frees, 152 bytes allocated
==27448==
==27448== 152 (16 direct, 136 indirect) bytes in 1 blocks are definitely lost in loss record 6 of 6
==27448== at 0x4C244E8: malloc (vg_replace_malloc.c:236)
==27448== by 0x40056B: main (test2.c:16)
==27448==
==27448== LEAK SUMMARY:
==27448== definitely lost: 16 bytes in 1 blocks
==27448== indirectly lost: 136 bytes in 5 blocks
==27448== possibly lost: 0 bytes in 0 blocks
==27448== still reachable: 0 bytes in 0 blocks
==27448== suppressed: 0 bytes in 0 blocks
==27448==
Your code should really be more like this:
#include <stdio.h>
#include <stdlib.h>
typedef struct sample_help {
int *value;
void **pointers;
} *sample, sample_node;
sample foo(void)
{
sample ABC=NULL;
sample XYZ=NULL;
sample kanchi = NULL;
ABC = malloc(sizeof(sample_node));
XYZ = malloc(sizeof(sample_node));
ABC->pointers = malloc(5*sizeof(void *));
XYZ->pointers = malloc(5*sizeof(void *));
ABC->value = malloc(5*sizeof(int));
XYZ->value = malloc(5*sizeof(int));
ABC->value[0] = 10;
ABC->value[1] = 20;
XYZ->pointers[0] = ABC;
kanchi = XYZ->pointers[0];
printf("::::%d\n",XYZ->pointers[0]);
printf("kanchi1:::::%d\n",kanchi->value[0]);
return XYZ;
}
int main(void)
{
// call your function
sample xyz = foo();
// ... do something with the data structure xyz ...
// free memory allocated by your function
free(xyz->pointers[0]->value); // free ABC->value
free(xyz->pointers[0]->pointers); // free ABC->pointers
free(xyz->pointers[0]); // free ABC
free(xyz->value); // free XYZ->value
free(xyz->pointers); // free XYZ->pointers
free(xyz); // free XYZ
return 0;
}
Note that we free the data structure from within main() after we're done with it.
Just free the used memory when it's no longer needed:
free(ABC->value);
free(XYZ->value);
free(ABC->pointers);
free(XYZ->pointers);
free(ABC);
free(XYZ);
Btw: When this is the whole program, it wouldn't actually matter. Since the OS reclaims all unfreed memory when a process is ended, memory that is in use until the termination of the programm is not required to be freed. However, it is good practice.
Having now read your update in the comments, your module (not called main()) that allocates and returns the memory is fine.
Whatever module uses the value returned from your module, needs to free the data after use. So if you implement your module as
sample mymodule(void)
{
sample foo = malloc(10);
/* set up contents of foo as required */
return foo;
}
Then the caller of mymodule would go like this:
int main (int argc, char *argv[])
{
sample bar = mymodule();
/* use contents of bar as required */
free(bar);
return 0;
}