Hey guys I'm attempting to read in workersinfo.txt and store it into a two-dimensional char array. The file is around 4,000,000 lines with around 100 characters per line. I want to store each file line on the array. Unfortunately, I get exit code 139(Not enough memory). I'm aware I have to use malloc() and free() but I've tried a couple of things and I haven't been able to make them work.Eventually I have to sort the array by ID number but I'm stuck on declaring the array.
The file looks something like this:
First Name, Last Name,Age, ID
Carlos,Lopez,,10568
Brad, Patterson,,20586
Zack, Morris,42,05689
This is my code so far:
#include <stdio.h>
#include <stdlib.h>
int main(void) {
FILE *ptr_file;
char workers[4000000][1000];
ptr_file =fopen("workersinfo.txt","r");
if (!ptr_file)
perror("Error");
int i = 0;
while (fgets(workers[i],1000, ptr_file)!=NULL){
i++;
}
int n;
for(n = 0; n < 4000000; n++)
{
printf("%s", workers[n]);
}
fclose(ptr_file);
return 0;
}
The Stack memory is limited. As you pointed out in your question, you MUST use malloc to allocate such a big (need I say HUGE) chunk of memory, as the stack cannot contain it.
you can use ulimit to review the limits of your system (usually including the stack size limit).
On my Mac, the limit is 8Mb. After running ulimit -a I get:
...
stack size (kbytes, -s) 8192
...
Or, test the limit using:
struct rlimit slim;
getrlimit(RLIMIT_STACK, &rlim);
rlim.rlim_cur // the stack limit
I truly recommend you process each database entry separately.
As mentioned in the comments, assigning the memory as static memory would, in most implementations, circumvent the stack.
Still, IMHO, allocating 400MB of memory (or 4GB, depending which part of your question I look at), is bad form unless totally required - especially for a single function.
Follow-up Q1: How to deal with each DB entry separately
I hope I'm not doing your homework or anything... but I doubt your homework would include an assignment to load 400Mb of data to the computer's memory... so... to answer the question in your comment:
The following sketch of single entry processing isn't perfect - it's limited to 1Kb of data per entry (which I thought to be more then enough for such simple data).
Also, I didn't allow for UTF-8 encoding or anything like that (I followed the assumption that English would be used).
As you can see from the code, we read each line separately and perform error checks to check that the data is valid.
To sort the file by ID, you might consider either running two lines at a time (this would be a slow sort) and sorting them, or creating a sorted node tree with the ID data and the position of the line in the file (get the position before reading the line). Once you sorted the binary tree, you can sort the data...
... The binary tree might get a bit big. did you look up sorting algorithms?
#include <stdio.h>
// assuming this is the file structure:
//
// First Name, Last Name,Age, ID
// Carlos,Lopez,,10568
// Brad, Patterson,,20586
// Zack, Morris,42,05689
//
// Then this might be your data structure per line:
struct DBEntry {
char* last_name; // a pointer to the last name
char* age; // a pointer to the name - could probably be an int
char* id; // a pointer to the ID
char first_name[1024]; // the actual buffer...
// I unified the first name and the buffer since the first name is first.
};
// each time you read only a single line, perform an error check for overflow
// and return the parsed data.
//
// return 1 on sucesss or 0 on failure.
int read_db_line(FILE* fp, struct DBEntry* line) {
if (!fgets(line->first_name, 1024, fp))
return 0;
// parse data and review for possible overflow.
// first, zero out data
int pos = 0;
line->age = NULL;
line->id = NULL;
line->last_name = NULL;
// read each byte, looking for the EOL marker and the ',' seperators
while (pos < 1024) {
if (line->first_name[pos] == ',') {
// we encountered a devider. we should handle it.
// if the ID feild's location is already known, we have an excess comma.
if (line->id) {
fprintf(stderr, "Parsing error, invalid data - too many fields.\n");
return 0;
}
// replace the comma with 0 (seperate the strings)
line->first_name[pos] = 0;
if (line->age)
line->id = line->first_name + pos + 1;
else if (line->last_name)
line->age = line->first_name + pos + 1;
else
line->last_name = line->first_name + pos + 1;
} else if (line->first_name[pos] == '\n') {
// we encountered a terminator. we should handle it.
if (line->id) {
// if we have the id string's possition (the start marker), this is a
// valid entry and we should process the data.
line->first_name[pos] = 0;
return 1;
} else {
// we reached an EOL without enough ',' seperators, this is an invalid
// line.
fprintf(stderr, "Parsing error, invalid data - not enough fields.\n");
return 0;
}
}
pos++;
}
// we ran through all the data but there was no EOL marker...
fprintf(stderr,
"Parsing error, invalid data (data overflow or data too large).\n");
return 0;
}
// the main program
int main(int argc, char const* argv[]) {
// open file
FILE* ptr_file;
ptr_file = fopen("workersinfo.txt", "r");
if (!ptr_file)
perror("File Error");
struct DBEntry line;
while (read_db_line(ptr_file, &line)) {
// do what you want with the data... print it?
printf(
"First name:\t%s\n"
"Last name:\t%s\n"
"Age:\t\t%s\n"
"ID:\t\t%s\n"
"--------\n",
line.first_name, line.last_name, line.age, line.id);
}
// close file
fclose(ptr_file);
return 0;
}
Followup Q2: Sorting array for 400MB-4GB of data
IMHO, 400MB is already touching on the issues related to big data. For example, implementing a bubble sort on your database should be agonizing as far as performance goes (unless it's a single time task, where performance might not matter).
Creating an Array of DBEntry objects will eventually get you a larger memory foot-print then the actual data..
This will not be the optimal way to sort large data.
The correct approach will depend on your sorting algorithm. Wikipedia has a decent primer on sorting algorythms.
Since we are handling a large amount of data, there are a few things to consider:
It would make sense to partition the work, so different threads/processes sort a different section of the data.
We will need to minimize IO to the hard drive (as it will slow the sorting significantly and prevent parallel processing on the same machine/disk).
One possible approach is to create a heap for a heap sort, but only storing a priority value and storing the original position in the file.
Another option would probably be to employ a divide and conquer algorithm, such as quicksort, again, only sorting a computed sort value and the entry's position in the original file.
Either way, writing a decent sorting method will be a complicated task, probably involving threading, forking, tempfiles or other techniques.
Here's a simplified demo code... it is far from optimized, but it demonstrates the idea of the binary sort-tree that holds the sorting value and the position of the data in the file.
Be aware that using this code will be both relatively slow (although not that slow) and memory intensive...
On the other hand, it will require about 24 bytes per entry. For 4 million entries, it's 96MB, somewhat better then 400Mb and definitely better then the 4GB.
#include <stdlib.h>
#include <stdio.h>
// assuming this is the file structure:
//
// First Name, Last Name,Age, ID
// Carlos,Lopez,,10568
// Brad, Patterson,,20586
// Zack, Morris,42,05689
//
// Then this might be your data structure per line:
struct DBEntry {
char* last_name; // a pointer to the last name
char* age; // a pointer to the name - could probably be an int
char* id; // a pointer to the ID
char first_name[1024]; // the actual buffer...
// I unified the first name and the buffer since the first name is first.
};
// this might be a sorting node for a sorted bin-tree:
struct SortNode {
struct SortNode* next; // a pointer to the next node
fpos_t position; // the DB entry's position in the file
long value; // The computed sorting value
}* top_sorting_node = NULL;
// this function will free all the memory used by the global Sorting tree
void clear_sort_heap(void) {
struct SortNode* node;
// as long as there is a first node...
while ((node = top_sorting_node)) {
// step forward.
top_sorting_node = top_sorting_node->next;
// free the original first node's memory
free(node);
}
}
// each time you read only a single line, perform an error check for overflow
// and return the parsed data.
//
// return 0 on sucesss or 1 on failure.
int read_db_line(FILE* fp, struct DBEntry* line) {
if (!fgets(line->first_name, 1024, fp))
return -1;
// parse data and review for possible overflow.
// first, zero out data
int pos = 0;
line->age = NULL;
line->id = NULL;
line->last_name = NULL;
// read each byte, looking for the EOL marker and the ',' seperators
while (pos < 1024) {
if (line->first_name[pos] == ',') {
// we encountered a devider. we should handle it.
// if the ID feild's location is already known, we have an excess comma.
if (line->id) {
fprintf(stderr, "Parsing error, invalid data - too many fields.\n");
clear_sort_heap();
exit(2);
}
// replace the comma with 0 (seperate the strings)
line->first_name[pos] = 0;
if (line->age)
line->id = line->first_name + pos + 1;
else if (line->last_name)
line->age = line->first_name + pos + 1;
else
line->last_name = line->first_name + pos + 1;
} else if (line->first_name[pos] == '\n') {
// we encountered a terminator. we should handle it.
if (line->id) {
// if we have the id string's possition (the start marker), this is a
// valid entry and we should process the data.
line->first_name[pos] = 0;
return 0;
} else {
// we reached an EOL without enough ',' seperators, this is an invalid
// line.
fprintf(stderr, "Parsing error, invalid data - not enough fields.\n");
clear_sort_heap();
exit(1);
}
}
pos++;
}
// we ran through all the data but there was no EOL marker...
fprintf(stderr,
"Parsing error, invalid data (data overflow or data too large).\n");
return 0;
}
// read and sort a single line from the database.
// return 0 if there was no data to sort. return 1 if data was read and sorted.
int sort_line(FILE* fp) {
// allocate the memory for the node - use calloc for zero-out data
struct SortNode* node = calloc(sizeof(*node), 1);
// store the position on file
fgetpos(fp, &node->position);
// use a stack allocated DBEntry for processing
struct DBEntry line;
// check that the read succeeded (read_db_line will return -1 on error)
if (read_db_line(fp, &line)) {
// free the node's memory
free(node);
// return no data (0)
return 0;
}
// compute sorting value - I'll assume all IDs are numbers up to long size.
sscanf(line.id, "%ld", &node->value);
// heap sort?
// This is a questionable sort algorythm... or a questionable implementation.
// Also, I'll be using pointers to pointers, so it might be a headache to read
// (it's a headache to write, too...) ;-)
struct SortNode** tmp = &top_sorting_node;
// move up the list until we encounter something we're smaller then us,
// OR untill the list is finished.
while (*tmp && (*tmp)->value <= node->value)
tmp = &((*tmp)->next);
// update the node's `next` value.
node->next = *tmp;
// inject the new node into the tree at the position we found
*tmp = node;
// return 1 (data was read and sorted)
return 1;
}
// writes the next line in the sorting
int write_line(FILE* to, FILE* from) {
struct SortNode* node = top_sorting_node;
if (!node) // are we done? top_sorting_node == NULL ?
return 0; // return 0 - no data to write
// step top_sorting_node forward
top_sorting_node = top_sorting_node->next;
// read data from one file to the other
fsetpos(from, &node->position);
char* buffer = NULL;
ssize_t length;
size_t buff_size = 0;
length = getline(&buffer, &buff_size, from);
if (length <= 0) {
perror("Line Copy Error - Couldn't read data");
return 0;
}
fwrite(buffer, 1, length, to);
free(buffer); // getline allocates memory that we're incharge of freeing.
return 1;
}
// the main program
int main(int argc, char const* argv[]) {
// open file
FILE *fp_read, *fp_write;
fp_read = fopen("workersinfo.txt", "r");
fp_write = fopen("sorted_workersinfo.txt", "w+");
if (!fp_read) {
perror("File Error");
goto cleanup;
}
if (!fp_write) {
perror("File Error");
goto cleanup;
}
printf("\nSorting");
while (sort_line(fp_read))
printf(".");
// write all sorted data to a new file
printf("\n\nWriting sorted data");
while (write_line(fp_write, fp_read))
printf(".");
// clean up - close files and make sure the sorting tree is cleared
cleanup:
printf("\n");
fclose(fp_read);
fclose(fp_write);
clear_sort_heap();
return 0;
}
A user sent me a crash dump of my program, and I cannot understand how what I'm seeing is possible. It looks like one of the registers just changed it's value, without any visible reason. I don't have any explanation except for a CPU bug, but I'm very skeptical about that. Perhaps you can spot what's going on here.
Here's the code disassembly, as seen when opening the crash report (clickable):
Here's, roughly, how the C code looks:
void **pp = *g_some_global;
if(!pp)
return NULL;
int array_count = (int)pp[0];
void **array_ptr = (void **)pp[1];
for(i = 0; i < array_count; i++)
{
LONG_PTR *contents = array_ptr[i];
if(contents[4] == compare)
{
void **pp2 = (LONG_PTR *)contents[7]; // contents is different here! pp2 is NULL
int array_count2 = (int)pp2[0]; // the CRASH!
void **array_ptr2 = (void **)pp2[1];
// ...
}
}
I am creating a program in c which is based on a linked list, where every node (of struct) holds an integer and a pointer to the next node.
I use dynamic allocation (malloc) and deallocation (free) as new nodes are added and old nodes are deleted.
when a node is deleted a function named delete is called.
I discovered that the program crashes sometimes when this delete-function is called and I KNOW that its something with the pointers in the method but I dont know WHERE in the code (row number) and WHY this happends.
I am used to high-level languages such as Java and I am used to encircle the problem by putting print-syntax at certain places in the method just to reveal WHERE it crashes.
I thought I could do the same with c and with pointer because to my knowledge I beleive the code is read from top to bottom that is 1, 2, 3, 4, and so on. (maybe interrupt handlers behave another way?)
So in this function named delete I have gone so far by putting this printf() at the very beginning of the delete-function - and all the same the program crashes.
So my Question - is it really possible that its some syntax in the delete-function (when I loop pointers for instance) that causes the crash WHEN not even the printf() is printing?
Am I wrong when I believe that the program is executed from to to bottom - that is 1, 2, 3 ....
You can se my printf-function in the very beginning of delete-function
And by the way - how could I solve this problem when I get this cryptic crash message from windows? See the bitmap!!
Greatful for answers!!!
int delete(int data) {
printf("IN THE BEGINNING OF DELETE!!!");
int result = 0;
if (queueref.last != NULL) {
node *curr_ptr;
node *prev_ptr;
node *temp_ptr;
if (queueref.first->data == data) {
temp_ptr = queueref.first;
queueref.first = queueref.first->next;
destroy_node(temp_ptr);
result = 1;
if (queueref.first == NULL) {
queueref.last = NULL;
puts("queue is now empty!!!");
}
} else {
prev_ptr = queueref.first;
curr_ptr = queueref.first->next;
printf("prev_ptr: %d\n", prev_ptr);
printf("curr_ptr: %d\n", curr_ptr);
while(curr_ptr != NULL) {
if (curr_ptr->data == data) {
result = 1;
if (curr_ptr->next != NULL) {
temp_ptr = curr_ptr;
destroy_node(temp_ptr);
prev_ptr->next = curr_ptr->next;
} else {
temp_ptr = curr_ptr;
queueref.last = prev_ptr;
prev_ptr->next = NULL;
destroy_node(temp_ptr);
}
}
curr_ptr = curr_ptr->next;
prev_ptr = prev_ptr->next;
}
}
}
return result;
}
Common mistake, here's the deal. This
printf("IN THE BEGINNING OF DELETE!!!");
needs to be
printf("IN THE BEGINNING OF DELETE!!!\n");
^^ note the newline
The reason is because stdio does not flush stdout until it sees a newline. If you add that newline, you should see the printf when the code enters the function. Without it, the program could crash, the stdout buffer would not have been flushed and would not see the printf.
Your code seems to have lots of implementation flaws. As a general advice I would recommend using some standard well-tested queue support library and static code analyzers (in this case you would even find dynamic analyzer valgrind very helpful, I guess).
For example, if implementation of destroy_node(ptr) is equivalent to free(ptr), then your code suffers from referencing destroyed data (or ,in other words, garbage) in this code snippet:
while(curr_ptr != NULL) {
if (curr_ptr->data == data) {
result = 1;
if (curr_ptr->next != NULL) {
temp_ptr = curr_ptr;
destroy_node(temp_ptr);
prev_ptr->next = curr_ptr->next; //<- curr_ptr is still in stack
//or register, but curr->next
//is garbage
// what if curr_ptr is first node? did you forget to update queueref.first?
} else {
temp_ptr = curr_ptr;
queueref.last = prev_ptr;
prev_ptr->next = NULL;
destroy_node(temp_ptr);
}
// if you you need to destroy only one node - you can leave the loop here with break;
}
curr_ptr = curr_ptr->next; /// assigning garbage again if node is found
prev_ptr = prev_ptr->next;
The reason why using destroyed data can work in * most * (if I can say that, basically this is unpredictable) cases is that the chances that this memory can be reused by other part of program for dynamically allocated data can vary on timings and code flow.
PS
Regarding cryptic messages in the Windows box - when program crashes OS basically generates crashdump and prints registers (and dumps some relevant memory parts). Registers and memory dumps can show the place of crash and immediate register/stack values but you have to now memory map and assembler output to understand it. Crashdump can be loaded to debugger (WinDbg) together with unstripped binary to check stactrace and values of local variables at the moment of crash. All these I described very very briefly, you could find tons of books / guides searching for "windows crash or crashdump analysis"
My program crashes on this function on the 7th line, when I call malloc() when I run in release mode I get the `Program.exe has stopped working message, and when I run in debugger, most of the time it succeeds but sometimes I get this message (especially on larger input):
MONOM* polynomialsProduct(MONOM* poly1, int size1, MONOM* poly2, int size2, int* productSize)
{
int i1, i2;
int phSize = 1, logSize = 0;
MONOM* product;
product = (MONOM*)malloc(phSize*sizeof(MONOM));
monomAllocationVerification(product);
for (i1 = 0; i1 < size1; i1++)
{
for (i2 = 0; i2 < size2; i2++)
{
if (logSize == phSize)
{
phSize *= 2;
product = (MONOM*)realloc(product,phSize*sizeof(MONOM));
monomAllocationVerification(product);
}
product[logSize].coefficient = poly1[i1].coefficient * poly2[i2].coefficient;
product[logSize].power = poly1[i1].power + poly2[i2].power;
logSize++;
}
}
mergeSort(product,logSize);
*productSize = sumMonomsWithSamePower(product, logSize);
return product;
}
I understand that I'm dealing with memory errors and problems, but is there any quick way to analyze my code and look for memory errors? I look at my code a dozen of times looking for this kind of errors and found nothing. (I didn't want to post the code here since its 420 lines long).
First of all, if heap corruption is detected on the first malloc, that means it happened earlier (not in this function or on previous pass). So the problem may lie outside this code.
However, the code also looks suspicious to me.
monomAllocationVerification has no size parameter, so it should work on one monom only, yet you call it only once after realloc on pointer to first element, despite having allocated space for quite a few monoms. Please clarify your decision.
It is a bit unclear why sumMonomsWithSamePower should return a size, and thus modify an array to store a value. May be a quirk, but still suspicious.
UPDATE
The problem was in other functions; a few reallocs with wrong size.
I would check the return value of malloc() and use perror() to describe what error has occured. Also here is the documentation for malloc() and perror().
if((product = (MONOM*)malloc(phSize*sizeof(MONOM))) == NULL)
{
perror("ERROR: Failed to malloc ");
return 1;
//perror() will display a system specified string to describe the error it may tell you the error
}
Also do you know the size of MONOM? If not add the following line to your code.
printf("MONOM SIZE = %i\n", sizeof(MONOM));
I'm attempting to write a solver for a particular puzzle. It tries to find a solution by trying every possible move one at a time until it finds a solution. The first version tried to solve it depth-first by continually trying moves until it failed, then backtracking, but this turned out to be too slow. I have rewritten it to be breadth-first using a queue structure, but I'm having problems with memory management.
Here are the relevant parts:
int main(int argc, char *argv[])
{
...
int solved = 0;
do {
solved = solver(queue);
} while (!solved && !pblListIsEmpty(queue));
...
}
int solver(PblList *queue) {
state_t *state = (state_t *) pblListPoll(queue);
if (is_solution(state->pucks)) {
print_solution(state);
return 1;
}
state_t *state_cp;
puck new_location;
for (int p = 0; p < puck_count; p++) {
for (dir i = NORTH; i <= WEST; i++) {
if (!rules(state->pucks, p, i)) continue;
new_location = in_dir(state->pucks, p, i);
if (new_location.x != -1) {
state_cp = (state_t *) malloc(sizeof(state_t));
state_cp->move.from = state->pucks[p];
state_cp->move.direction = i;
state_cp->prev = state;
state_cp->pucks = (puck *) malloc (puck_count * sizeof(puck));
memcpy(state_cp->pucks, state->pucks, puck_count * sizeof(puck)); /*CRASH*/
state_cp->pucks[p] = new_location;
pblListPush(queue, state_cp);
}
}
}
free(state->pucks);
return 0;
}
When I run it I get the error:
ice(90175) malloc: *** mmap(size=2097152) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
Bus error
The error happens around iteration 93,000.
From what I can tell, the error message is from malloc failing, and the bus error is from the memcpy after it.
I have a hard time believing that I'm running out of memory, since each game state is only ~400 bytes. Yet that does seem to be what's happening, seeing as the activity monitor reports that it is using 3.99GB before it crashes. I'm using http://www.mission-base.com/peter/source/ for the queue structure (it's a linked list).
Clearly I'm doing something dumb. Any suggestions?
Check the result of malloc. If it's NULL, you might want to print out the length of that queue.
Also, the code snippet you posted didn't include any frees...
You need to free() the memory you've allocated manually after you're done with it; dynamic memory doesn't just "free itself"