I am developing a c program which connects to a mysql database reads data from a a table of 4 million of data and writes its data to another 100 tables in another database after some calculations. To make it efficient I tried to use 100 threads to write data for 100 tables and 1 thread to read data from the database and write those in to a buffer. So that the 100 threads would read from the buffers.
But the problem is when I'm making the buffers. I used malloc to make the buffers. char *** queue; is declared in the header file so that it is global
int i = 0, j = 0;
queue = (char ***) malloc(100);
int threadRet;
for (i; i < 100; i++) {
queue[i] = (char **) malloc(2000);
for (j; j < 2000; j++) {
queue[i][j] = (char *) malloc(180);
}
}
and My buffer writing thread as I mentioned before exicutes the function void * thrededChunkPicker(void * parr)
sprintf(queue[tableNo][marker[tableNo]], "INSERT INTO usage_summary%s(mobile,`0`,ddate) VALUES('%s',%d,'%s') ON DUPLICATE KEY UPDATE `0` = VALUES(`0`), ddate=VALUES(ddate)", row[0] + 7, row[0], sumOfUsage, row[2]);
(marker[tableNo])++;
This is how I write to the buffer . As I've found out the segmentation fault occurs here.
I needed 100 buffers in each of which contains 2000 string arrays of 180. this code compiled successfully.but when runs it gives a segmentation fault.
As far as I can see, all your malloc() calls are problematic.
We pass the size in bytes to malloc(). malloc() does not have any idea of the size of the object in which you're going to store the returned pointer, whatsoever. So, it cannot automatically calculate the total size for the memory required.
We need to calculate the total size required and pass that size to malloc().
First to start with, please see this discussion on why not to cast the return value of malloc() and family in C..
That said, assuming queue is defined as char *** queue;, we need to
queue = malloc(100 * sizeof*queue );
queue[i] = malloc(2000 * sizeof(*queue[i]) );
and so on, if required.
Finally, always check for the success of malloc() through a NULL check on the returned pointer before using the same.
The biggest problem is that malloc allocates a number of bytes, not elements in an array. So you allocate 100 bytes for your three-star variable, which is not enough space for 100 elements. The same with the next call to malloc, it allocates 2000 bytes not 2000 elements.
Related
I am working with sending data to and from a client/server application. I have created my own header scheme. This is the flow I am in question about:
char *data = returnmydata();
int one = 1;
int two = 2;
int three = 3;
So what happens is I have a char buffer that contains all my data. But I need to append to the front of this data my one, two, and three integers which is used to let the other side know how to handle the data.
Instead of allocating enough at the start for the 3 header items, is it possible to reallocate an addition 12 bytes and move the start of the data to data+12?
I currently just memcpy the 3 integers at the start then offset the data +12. However I think my code would be better if I could just pass a struct that contains the 3 ints and a pointer to the data. Then I can use this function to create the full data buffer with the headers in it.
Another option is to alloc another buffer memcpy the 3 integers to the front, and then memcpy the data to the new buffer. However I would like to keep memory usage low.
Use functions realloc and memmove.
If the size of the initially allocated buffer is equal to some value N then you can write
char *tmp = realloc( data, N + 12 );
if ( tmp != NULL )
{
data = tmp;
memmove( data + 12, data, N );
// then you can insert some values in the first 12 bytes of the allocated buffer
}
For a project in my Parallel Computing class, I need to implement a parallel version of the Game of Life.
I am using a function written by my textbook's author called "read_row_stripped_matrix". This function reads in input from a file that contains the number of rows in the matrix, number of columns in the matrix, and the data in the matrix.
The function sets up the two dimensional matrix by allocating a one dimensional array called "storage", which holds all of the matrix's data. Each row of the two dimensional matrix points to its first element within storage, like in this picture:
We are required to clean up the function code so that it fits with our C style guide. So I cleaned a few things up so it would be more readable.
The issue I'm running into now occurs with pointing each row in the matrix to its first element in storage. I'm getting a segmentation fault while hooking up these pointers, specifically in this section of the function:
/* Dynamically allocate matrix. Allow double subscripting
through 'a'. */
*storage = my_malloc (id, local_rows * *n * sizeof(int));
*subs = my_malloc (id, local_rows * PTR_SIZE);
for (i = 0; i < local_rows; i++) {
*subs[i]=&(*storage[i * *n]);
}
What's puzzling me is that I'm pretty sure I've allocated enough memory for the array. In the example I'm testing, *m and *n equal 5, and local_rows equals 5. So I'm allocating 25*sizeof(int) for storage which should be sufficient to hold all elements for a 5x5 matrix.
This is the my_malloc function, which mallocs for a specific processor:
/*
* Function 'my_malloc' is called when a process wants
* to allocate some space from the heap. If the memory
* allocation fails, the process prints an error message
* and then aborts execution of the program.
*/
void* my_malloc (
int id, /* IN - Process rank */
int bytes) /* IN - Bytes to allocate */
{
void *buffer;
if ((buffer = malloc ((size_t) bytes)) == NULL) {
printf ("Error: Malloc failed for process %d\n", id);
fflush (stdout);
MPI_Abort (MPI_COMM_WORLD, MALLOC_ERROR);
}
return buffer;
}
I honestly find pointers confusing, so forgive me if the issue is obvious. I have been working on this for longer than I should so my brain is probably fried.
If you need more code, don't hesitate to ask.
First, you do this:
*subs = my_malloc (id, local_rows * PTR_SIZE);
Then, you do this:
*subs[i]=&(*storage[i * *n]);
Pretty sure that's your problem, right there. Looks to me like it should really be:
(*subs)[i]=&(*storage[i * *n]);
here is a very small structure used for indexing words of a file. Its members are a string (the word), an array of integers (the lines this word is found at), and an integer representing the index of the first free cell in the lines array.
typedef struct {
wchar_t * word;
int * lines;
int nLine;
} ndex;
ndex * words;
I am trying to allocate (ndex)es nb_words = 128 at a time, and (lines) nb_lines = 8 at a time, using malloc and realloc.
First question, what is the difference between malloc(number * size) and calloc(number, size) when allocating *words and/or *lines? Which should I choose?
Second question, I gdbed this:
Program received signal SIGSEGV, Segmentation fault.
0x0000000000400cb0 in add_line (x=43, line=3) at cx17.3.c:124
124 words[x].lines[words[x].nLine] = line;
(gdb) p words[43].nLine
$30 = 0
In other words, it consistently fails at
words[43].lines[0] = 3;
Since I allocate words by 128, and lines by 8, there is no reason the indexing worked for the 42 previous words and fail here, except if my allocating was botched, is there?
Third question: here are my allocations, what is wrong with them?
words = malloc(sizeof(ndex *) * nb_words);
short i;
for (i = 0; i < nb_words; i++) {
words[i].lines = malloc(sizeof(int) * nb_lines);
words[i].nLine = 0;
}
Should I initialize lines in a for(j) loop? I don't see why leaving it uninitialized would prevent writing to it, so long as it as been allocated.
This C is a very mysterious thing to me, thanks in advance for any hints you can provide.
Best regards.
This looks suspicious:
sizeof(ndex *)
You probably don't want the size of a pointer - you want the size of a structure. So remove the star.
Here:
words = malloc(sizeof(ndex *) * nb_words);
You are allocating space for some number of pointers (i.e., 4 bytes * nb_words). What you really need is to allocate some number of ndex's:
words = malloc(sizeof(ndex) * nb_words);
Also, calloc 0 initializes the returned buffer while malloc does not. See this answer.
malloc will allocate the requested space only. calloc will allocate the space and initialize to zero.
In your example, the segmentation fault is observed here words[x].lines[words[x].nLine] = line;. There could be 2 possibilities viz., allocation is wrong which I don't feel is the case. The more probable case would be words[x].nLine didn't evaluate to 0. Please print this value and check. I suspect this is some huge number which is forcing your program to access a memory out of your allocated space.
Others have answered this part, so I will skip it.
I have to create an array of char pointers each of them of size 10000000 using the best and optimized way to do this in C.
I think this will do (haven't checked for nulls though):
int i;
int num_arrays;
char **huge_char_array;
num_arrays = 10; //number of arrays you want.
huge_char_array = (char **)malloc(sizeof(char *) * num_arrays);
for(i = 0; i < num_arrays; i++)
{
huge_char_array[i] = (char *)malloc(sizeof(char) * 10000000);
}
I believe this is the optimal way because there is only one dynamic allocation, reducing overhead from fragmentation of the heap and the time taken to allocate. You can use the STRING_INDEX utility function to access the nth string.
Also, using calloc() instead of malloc() zeroes out buffer to ensure that all strings are NUL terminated.
#define STRING_SIZE 10000000
#define NUM_STRINGS 10
#define STRING_INDEX(array, string_idx) ((array) + (string_idx) * STRING_SIZE)
int main(int argc, char **argv) {
char *array_of_strings = calloc(NUM_STRINGS, STRING_SIZE);
// Access 8th character of 7th string
char c = STRING_INDEX(array_of_strings, 7)[8];
// Use array
// Free array when done
free(array_of_strings);
return 0;
}
I have been allocating 50K with each request I process this resulted in memory fragmentation. So I switched to allocate 50k * 100 = 5MB and reuse the pool when ever I need to allocate one more. If the requests increases beyond my pool capacity I double the pool size to avoid memory fragmentations. Based on this I would recommend allocating a huge chunk of memory maybe in your case 10 * 10000000 to reuse it. I am sorry but I cannot share the code that handle the pool here. Also use malloc and free and do not use new and delete.
malloc won't through exception it will simply return null if there are no memory available.
if you insist on using malloc for each element individually you can create a heap with maximum size to avoid fragmentations and in that case you will have to define the heap size.
please referee to MSDN for more details about how to create a heap and use it or the other allocation methods available for windows users.
http://msdn.microsoft.com/en-us/library/windows/desktop/aa366533(v=vs.85).aspx
char *arr[SOME_SIZE];
for(int i = 0; i < SOME_SIZE; ++i) {
arr[i] = malloc(10000000);
if(!arr[i])
// allocation failed, do something
}
Just realize that you're allocating ~9.5MB for every element in the array (i.e., SOME_SIZE * 10000000 total bytes)
I have been debugging a piece of legacy code, running on an XScale (arm v5te) System with linux, that crashes reproducible.
I have debugged using gdb and set MALLOC_CHECK_ to 1. It's a lot of code, so just some snippets:
We have this structure:
typedef struct {
...clipped..
char **data_column_list;
/** data column count */
int data_column_cnt;
...clipped
} csv_t;
We initialize the columns in a function, putting them in a variable "columns"
/* Allocating memory for pointer to every register id */
columns = (char **) malloc(column_cnt * sizeof(char *));
column_cnt = 0;
/* loop over all sensors */
for(i=0; i<cfg.sen_cnt; i++) {
/* loop over all registers */
for(j=0; j<cfg.sen_list[i]->data_cnt; j++) {
/* Storing all the pointers to id */
columns[column_cnt++] = cfg.sen_list[i]->data_list[j]->id;
}
}
In another function, what happens is this:
/* free the previous list */
csv_free(lc_csv);
lc_csv->data_column_list = columns;
lc_csv->data_column_cnt = column_cnt;
csv_free being:
void csv_free(csv_t *csv) {
if(csv->data_column_cnt > 0)
free(csv->data_column_list);
csv->data_column_cnt = 0;
}
Now, there is another function, building the whole "cfg"/config structure, that contains these ids.
Code frome above: cfg.sen_list[i]->data_list[j]->id; where cfg is a struct, sen_list is an array of pointers to structs, data_list is an array of pointers to other structs, that contain a string "id".
Whe the program gets a signal SIGUSR1, the config is being updated. All of these data_list and sen_list structs are being freed, then new ones are generated.
Then with the first function, new collumns of ids are generated and put into the csv structure, but the old list is being freed before.
Thats where it crashes. In csv_free.
*** glibc detected *** /root/elv: free(): invalid pointer: 0x0001ae88 ***
I thought it should be like this. You have an array of pointers. When you free the pointers, you have to free the pointer, pointing to a set of pointers (the array).
Or put in code terms, the above situation should be analog to:
char **ar = malloc(n * sizeof(char *));
char *xn = malloc(10 * sizeof(char)); // Do for 0 to n strings
...
ar[n] = xn; // Do for 0 to n strings
...do stuff...
free(xn); // Do for 0 to n strings
free(ar);
When the structs, containing the id strings, are freed, I still have my pointer arrays with (invalid) pointers, not null pointers:
(gdb) p csv
$40 = {sysid = 222, ip = '\0' <repeats 49 times>,
module = "elv_v2", '\0' <repeats 14 times>, format_type = 1, msg_id = 0,
data_column_list = 0x1ae88, data_column_cnt = 10, pub_int = 30,
line_cnt = 0, pub_seq = -1, format = 0x18260}
(gdb) p csv.data_column_list[0]
$41 = 0x1b378 "0"
But I get the above error message (or SIGABRT without the MALLOC_CHECK_).
I don't understand this at all. I have to free this array of pointers, or it will become a memory leak. There is no other call of free before that, that I could find. I don't know why csv.data_column_list is considered an invalid pointer.
Valgrind is unfortunately not availiable on arm v5te :(
Have been debugging this for hours and hours and would be happy for any help.
Thank you very much,
Cheers,
Ben
UPDATE:
I'm wondering if it could be connected to some "scope" issue. There is almost identical code in another application, which works. The function which crashes, "csv_free" is used by both programs (statically linked). The only difference is, that the struct containing the pointer to be freed is declared and defined normally in the working program and declared as external and defined in another file than main.c
Calling "free" manually in main.c works, while calling "csv_free" crashes. Riddle me this...
9 out of 10 times when I run into free() errors the problem actually started in the allocation or initialization, so let's verify that a bit:
Where do you actually assign columns to csv.data_columns_list before you call csv_free? If it's uninitialized when you free(), that would explain the error.
In the second code block, if the initial column_cnt (which I guess is set
elsewhere?) is less than the column_cnt after the loop you would be writing outside the
array. One would hope MALLOC_CHECK_ would catch that, but what happens if you assert as
follows:
/* Allocating memory for pointer to every register id */
columns = (char **) malloc(column_cnt * sizeof(char *));
int old_column_cnt = column_cnt;
column_cnt = 0;
/* loop over all sensors */
for(i=0; i<cfg.sen_cnt; i++) {
/* loop over all registers */
for(j=0; j<cfg.sen_list[i]->data_cnt; j++) {
/* Storing all the pointers to id */
columns[column_cnt++] = cfg.sen_list[i]->data_list[j]->id;
}
}
assert(old_column_cnt >= column_cnt);
Looking over my old questions I saw this. I can't really verify, since I don't work at that company anymore, but thinking of other issues we had, I think wildplasser was right.
Calling any large functions from within signal handlers is a bad idea. Especially if you haven't check whether everything you do is reentrant. It was legacy code, so at least it's not completely my fault ;)
Nowadays I would set a flag in the signal handler and call the routine in my main loop, when that flag is set (or something like that).