Hey guys I'm attempting to read in workersinfo.txt and store it into a two-dimensional char array. The file is around 4,000,000 lines with around 100 characters per line. I want to store each file line on the array. Unfortunately, I get exit code 139(Not enough memory). I'm aware I have to use malloc() and free() but I've tried a couple of things and I haven't been able to make them work.Eventually I have to sort the array by ID number but I'm stuck on declaring the array.
The file looks something like this:
First Name, Last Name,Age, ID
Carlos,Lopez,,10568
Brad, Patterson,,20586
Zack, Morris,42,05689
This is my code so far:
#include <stdio.h>
#include <stdlib.h>
int main(void) {
FILE *ptr_file;
char workers[4000000][1000];
ptr_file =fopen("workersinfo.txt","r");
if (!ptr_file)
perror("Error");
int i = 0;
while (fgets(workers[i],1000, ptr_file)!=NULL){
i++;
}
int n;
for(n = 0; n < 4000000; n++)
{
printf("%s", workers[n]);
}
fclose(ptr_file);
return 0;
}
The Stack memory is limited. As you pointed out in your question, you MUST use malloc to allocate such a big (need I say HUGE) chunk of memory, as the stack cannot contain it.
you can use ulimit to review the limits of your system (usually including the stack size limit).
On my Mac, the limit is 8Mb. After running ulimit -a I get:
...
stack size (kbytes, -s) 8192
...
Or, test the limit using:
struct rlimit slim;
getrlimit(RLIMIT_STACK, &rlim);
rlim.rlim_cur // the stack limit
I truly recommend you process each database entry separately.
As mentioned in the comments, assigning the memory as static memory would, in most implementations, circumvent the stack.
Still, IMHO, allocating 400MB of memory (or 4GB, depending which part of your question I look at), is bad form unless totally required - especially for a single function.
Follow-up Q1: How to deal with each DB entry separately
I hope I'm not doing your homework or anything... but I doubt your homework would include an assignment to load 400Mb of data to the computer's memory... so... to answer the question in your comment:
The following sketch of single entry processing isn't perfect - it's limited to 1Kb of data per entry (which I thought to be more then enough for such simple data).
Also, I didn't allow for UTF-8 encoding or anything like that (I followed the assumption that English would be used).
As you can see from the code, we read each line separately and perform error checks to check that the data is valid.
To sort the file by ID, you might consider either running two lines at a time (this would be a slow sort) and sorting them, or creating a sorted node tree with the ID data and the position of the line in the file (get the position before reading the line). Once you sorted the binary tree, you can sort the data...
... The binary tree might get a bit big. did you look up sorting algorithms?
#include <stdio.h>
// assuming this is the file structure:
//
// First Name, Last Name,Age, ID
// Carlos,Lopez,,10568
// Brad, Patterson,,20586
// Zack, Morris,42,05689
//
// Then this might be your data structure per line:
struct DBEntry {
char* last_name; // a pointer to the last name
char* age; // a pointer to the name - could probably be an int
char* id; // a pointer to the ID
char first_name[1024]; // the actual buffer...
// I unified the first name and the buffer since the first name is first.
};
// each time you read only a single line, perform an error check for overflow
// and return the parsed data.
//
// return 1 on sucesss or 0 on failure.
int read_db_line(FILE* fp, struct DBEntry* line) {
if (!fgets(line->first_name, 1024, fp))
return 0;
// parse data and review for possible overflow.
// first, zero out data
int pos = 0;
line->age = NULL;
line->id = NULL;
line->last_name = NULL;
// read each byte, looking for the EOL marker and the ',' seperators
while (pos < 1024) {
if (line->first_name[pos] == ',') {
// we encountered a devider. we should handle it.
// if the ID feild's location is already known, we have an excess comma.
if (line->id) {
fprintf(stderr, "Parsing error, invalid data - too many fields.\n");
return 0;
}
// replace the comma with 0 (seperate the strings)
line->first_name[pos] = 0;
if (line->age)
line->id = line->first_name + pos + 1;
else if (line->last_name)
line->age = line->first_name + pos + 1;
else
line->last_name = line->first_name + pos + 1;
} else if (line->first_name[pos] == '\n') {
// we encountered a terminator. we should handle it.
if (line->id) {
// if we have the id string's possition (the start marker), this is a
// valid entry and we should process the data.
line->first_name[pos] = 0;
return 1;
} else {
// we reached an EOL without enough ',' seperators, this is an invalid
// line.
fprintf(stderr, "Parsing error, invalid data - not enough fields.\n");
return 0;
}
}
pos++;
}
// we ran through all the data but there was no EOL marker...
fprintf(stderr,
"Parsing error, invalid data (data overflow or data too large).\n");
return 0;
}
// the main program
int main(int argc, char const* argv[]) {
// open file
FILE* ptr_file;
ptr_file = fopen("workersinfo.txt", "r");
if (!ptr_file)
perror("File Error");
struct DBEntry line;
while (read_db_line(ptr_file, &line)) {
// do what you want with the data... print it?
printf(
"First name:\t%s\n"
"Last name:\t%s\n"
"Age:\t\t%s\n"
"ID:\t\t%s\n"
"--------\n",
line.first_name, line.last_name, line.age, line.id);
}
// close file
fclose(ptr_file);
return 0;
}
Followup Q2: Sorting array for 400MB-4GB of data
IMHO, 400MB is already touching on the issues related to big data. For example, implementing a bubble sort on your database should be agonizing as far as performance goes (unless it's a single time task, where performance might not matter).
Creating an Array of DBEntry objects will eventually get you a larger memory foot-print then the actual data..
This will not be the optimal way to sort large data.
The correct approach will depend on your sorting algorithm. Wikipedia has a decent primer on sorting algorythms.
Since we are handling a large amount of data, there are a few things to consider:
It would make sense to partition the work, so different threads/processes sort a different section of the data.
We will need to minimize IO to the hard drive (as it will slow the sorting significantly and prevent parallel processing on the same machine/disk).
One possible approach is to create a heap for a heap sort, but only storing a priority value and storing the original position in the file.
Another option would probably be to employ a divide and conquer algorithm, such as quicksort, again, only sorting a computed sort value and the entry's position in the original file.
Either way, writing a decent sorting method will be a complicated task, probably involving threading, forking, tempfiles or other techniques.
Here's a simplified demo code... it is far from optimized, but it demonstrates the idea of the binary sort-tree that holds the sorting value and the position of the data in the file.
Be aware that using this code will be both relatively slow (although not that slow) and memory intensive...
On the other hand, it will require about 24 bytes per entry. For 4 million entries, it's 96MB, somewhat better then 400Mb and definitely better then the 4GB.
#include <stdlib.h>
#include <stdio.h>
// assuming this is the file structure:
//
// First Name, Last Name,Age, ID
// Carlos,Lopez,,10568
// Brad, Patterson,,20586
// Zack, Morris,42,05689
//
// Then this might be your data structure per line:
struct DBEntry {
char* last_name; // a pointer to the last name
char* age; // a pointer to the name - could probably be an int
char* id; // a pointer to the ID
char first_name[1024]; // the actual buffer...
// I unified the first name and the buffer since the first name is first.
};
// this might be a sorting node for a sorted bin-tree:
struct SortNode {
struct SortNode* next; // a pointer to the next node
fpos_t position; // the DB entry's position in the file
long value; // The computed sorting value
}* top_sorting_node = NULL;
// this function will free all the memory used by the global Sorting tree
void clear_sort_heap(void) {
struct SortNode* node;
// as long as there is a first node...
while ((node = top_sorting_node)) {
// step forward.
top_sorting_node = top_sorting_node->next;
// free the original first node's memory
free(node);
}
}
// each time you read only a single line, perform an error check for overflow
// and return the parsed data.
//
// return 0 on sucesss or 1 on failure.
int read_db_line(FILE* fp, struct DBEntry* line) {
if (!fgets(line->first_name, 1024, fp))
return -1;
// parse data and review for possible overflow.
// first, zero out data
int pos = 0;
line->age = NULL;
line->id = NULL;
line->last_name = NULL;
// read each byte, looking for the EOL marker and the ',' seperators
while (pos < 1024) {
if (line->first_name[pos] == ',') {
// we encountered a devider. we should handle it.
// if the ID feild's location is already known, we have an excess comma.
if (line->id) {
fprintf(stderr, "Parsing error, invalid data - too many fields.\n");
clear_sort_heap();
exit(2);
}
// replace the comma with 0 (seperate the strings)
line->first_name[pos] = 0;
if (line->age)
line->id = line->first_name + pos + 1;
else if (line->last_name)
line->age = line->first_name + pos + 1;
else
line->last_name = line->first_name + pos + 1;
} else if (line->first_name[pos] == '\n') {
// we encountered a terminator. we should handle it.
if (line->id) {
// if we have the id string's possition (the start marker), this is a
// valid entry and we should process the data.
line->first_name[pos] = 0;
return 0;
} else {
// we reached an EOL without enough ',' seperators, this is an invalid
// line.
fprintf(stderr, "Parsing error, invalid data - not enough fields.\n");
clear_sort_heap();
exit(1);
}
}
pos++;
}
// we ran through all the data but there was no EOL marker...
fprintf(stderr,
"Parsing error, invalid data (data overflow or data too large).\n");
return 0;
}
// read and sort a single line from the database.
// return 0 if there was no data to sort. return 1 if data was read and sorted.
int sort_line(FILE* fp) {
// allocate the memory for the node - use calloc for zero-out data
struct SortNode* node = calloc(sizeof(*node), 1);
// store the position on file
fgetpos(fp, &node->position);
// use a stack allocated DBEntry for processing
struct DBEntry line;
// check that the read succeeded (read_db_line will return -1 on error)
if (read_db_line(fp, &line)) {
// free the node's memory
free(node);
// return no data (0)
return 0;
}
// compute sorting value - I'll assume all IDs are numbers up to long size.
sscanf(line.id, "%ld", &node->value);
// heap sort?
// This is a questionable sort algorythm... or a questionable implementation.
// Also, I'll be using pointers to pointers, so it might be a headache to read
// (it's a headache to write, too...) ;-)
struct SortNode** tmp = &top_sorting_node;
// move up the list until we encounter something we're smaller then us,
// OR untill the list is finished.
while (*tmp && (*tmp)->value <= node->value)
tmp = &((*tmp)->next);
// update the node's `next` value.
node->next = *tmp;
// inject the new node into the tree at the position we found
*tmp = node;
// return 1 (data was read and sorted)
return 1;
}
// writes the next line in the sorting
int write_line(FILE* to, FILE* from) {
struct SortNode* node = top_sorting_node;
if (!node) // are we done? top_sorting_node == NULL ?
return 0; // return 0 - no data to write
// step top_sorting_node forward
top_sorting_node = top_sorting_node->next;
// read data from one file to the other
fsetpos(from, &node->position);
char* buffer = NULL;
ssize_t length;
size_t buff_size = 0;
length = getline(&buffer, &buff_size, from);
if (length <= 0) {
perror("Line Copy Error - Couldn't read data");
return 0;
}
fwrite(buffer, 1, length, to);
free(buffer); // getline allocates memory that we're incharge of freeing.
return 1;
}
// the main program
int main(int argc, char const* argv[]) {
// open file
FILE *fp_read, *fp_write;
fp_read = fopen("workersinfo.txt", "r");
fp_write = fopen("sorted_workersinfo.txt", "w+");
if (!fp_read) {
perror("File Error");
goto cleanup;
}
if (!fp_write) {
perror("File Error");
goto cleanup;
}
printf("\nSorting");
while (sort_line(fp_read))
printf(".");
// write all sorted data to a new file
printf("\n\nWriting sorted data");
while (write_line(fp_write, fp_read))
printf(".");
// clean up - close files and make sure the sorting tree is cleared
cleanup:
printf("\n");
fclose(fp_read);
fclose(fp_write);
clear_sort_heap();
return 0;
}
I'm working on INI-style configuration parser for some project, and I gets next trouble.
I have 3 structures:
typedef struct {
const char* name;
unsigned tract;
int channel;
const char* imitation_type;
} module_config;
typedef struct {
int channel_number;
int isWorking;
int frequency;
int moduleCount;
} channel_config;
typedef struct {
int mode;
module_config* module;
channel_config* channel;
} settings;
And I have function for handling data in my INI-file (I working under inih parser): [pasted to pastebin cause too long]. Finally, in main(), I did the next:
settings* main_settings;
main_settings = (settings*)malloc(sizeof(settings));
main_settings->module = (module_config*)malloc(sizeof(module_config));
main_settings->channel = (channel_config*)malloc(sizeof(channel_config));
if (ini_parse("test.ini", handler, &main_settings) < 0) {
printf("Can't load 'test.ini'\n");
return 1;
}
In result, binary crashes with memory fault. I think (no, I KNOW), what I'm incorrectly allocating the memory in handler(), but I does not understand, where I do it wrong. I spent all night long trying to understand memory allocating, and I'm very tired, but now me simply interestingly, what I'm doing wrong, and HOW to force this working fine.
P.S. Sorry for ugly english
The problem seems to be related to the reallocation of your structs:
pconfig = (settings *) realloc(pconfig, (module_count + channel_count) * sizeof(channel_config));
pconfig->module = (module_config *) realloc(pconfig->module, module_count * sizeof(module_config));
pconfig->channel = (channel_config *) realloc(pconfig->channel, channel_count * sizeof(channel_config));
First of all, you must not reallocate the main settings struct. Since your handler will always be called with the original pconfig value, the reallocation of the module and channel arrays has no effect, and you'll access freed memory.
Also when reallocating the module and channel arrays you should allocate count + 1 elements, since the next invocation of handler might assign to the [count] slot.
So try to replace the three lines above with:
pconfig->module = (module_config *) realloc(pconfig->module, (module_count + 1) * sizeof(module_config));
pconfig->channel = (channel_config *) realloc(pconfig->channel, (channel_count + 1) * sizeof(channel_config));
I am working on building a threads library and for some reason have run into a simple malloc problem that I can't fix right now. I'm sure it's something simple I'm just missing it.
In my main.c I have the following code:
//declare testSem
tasem_t testSem;
int main(int argc, char **argv){
ta_libinit();
//initialize testSem
ta_sem_init(&testSem, 5);
//wait test
ta_sem_wait(&testSem);
the relevant code in my thread library is as follows:
void ta_sem_init(tasem_t *sema, int value)
{
//malloc the semaphore struct
sema = malloc(sizeof(tasem_t));
//error check
if(sema == NULL)
{
printf("could not malloc semaphore");
exit(0);
}
//initialize with the given value
sema->val = value;
printf("SemaVal = %i\n", sema->val);
}
void ta_sem_wait(tasem_t *sema)
{
printf("SemaVal = %i\n", sema->val);
if(sema->val <= 0)
{
//not done yet
printf("SWAPPING\n");
}
else
{
printf("SemaVal = %i\n", sema->val);
sema->val = sema->val + 1;
}
}
Here is the struct from my header file:
//struct to store each semas info
typedef struct tasem_t_struct
{
//value
int val;
//Q* Queue
//int numThreads
}tasem_t;
The output I get from this is:
SemaVal = 5
SemaVal = 0
SWAPPING
So evidently, I'm not mallocing my struct correctly as the value inside is lost once I go out of scope. I know I must just be forgetting something simple. Any ideas?
You can't seem to decide who's responsible for allocating your tasem_t structure. You have a global variable for it and pass its address to ta_sem_init. But then you have ta_sem_init dynamically allocate a brand new tasem_t structure, saving its address to sema, a local function argument, so that address gets lost when it falls out of scope.
Pick one, either:
Make ta_sem_init initialize an existing tasem_t variable.
Make ta_sem_init allocate and initialize a new tasem_t structure and then return its address (either directly or via a tasem_t** output parameter).
For a school project I am supposed to implement a simplified version of the UNIX filesystem using only linked list structures. I am currently having a problem with my mkfs() function, which is supposed to simply initialize a filesystem.
My header file that creates the structures I am using is here:
typedef struct Lines {
char line[82];
struct Lines *next;
} Lines;
typedef struct Node {
char *name;
int id;
struct Node *parent;
struct Node *next;
union {
char line[82];
struct Node *children;
} contents;
} Node;
typedef struct Filesystem {
char *name;
struct Node *root;
struct Node *current;
} Filesystem;
Here is the method in my separate file which #includes this header file:
void mkfs(Filesystem *files) {
Node *root = NULL; /* Creates a pointer to the directory we will use as
* the root directory for this filesystem*/
files = (Filesystem *)malloc(sizeof(*files)); /* Allocates space for the the
* filesystem structure */
if(files == NULL){ /* If there is no memory available, prints error message
* and does nothing else */
printf("Memory allocation failed!\n");
} else {
root = (Node *)malloc(sizeof(*root)); /* Allocates space for the root
* directory of the filesystem. */
if(root == NULL) { /* If there is no memory available, prints error
* message and frees memory obtained thus far, but then
* does nothing else */
printf("Memory allocation failed!\n");
free(files);
} else {
/* Allocates space for the root directory's name string */
root->name= (char *)malloc(sizeof(char)*(strlen("/")+1));
if(root->name == NULL) { /* If there is no memory available, prints error
* message and frees memory obtained thus far,
* but then does nothing else */
printf("Memory allocation failed!\n");
free(files);
free(root);
} else {
root->name = "/"; /* Defines the root directory as being named by the
* forward slash */ /* DO STR CPY HERE ITS CHANGING THE ADDRESS */
root->contents.children = NULL;
root->next = NULL;
root->parent = NULL; /* UHH CHECK ON THIS NOOO CLUE IF ITS RIGHT FUUU*/
files->root = root; /* The filesystems pointer to a directory is set
* to point to the root directory we just allocated
* space for and set up */
files->current = root; /* Sets the filesystems current directory to
* point to the root directory as well, because
* it is the only directory in existence for this
* filesystem at this point. */
}
}
}
}
The problem I am having is that when I run gdb and step through each line, the last two assignment lines ARE NOT CHANGING the contents of file->root and file->current.
For example, here I print the contents of files->root, run the line files->root = root, and then print again, and you can see the address has not changed. However if I just print root, the thing I am trying to assign it to, it clearly has a different value that files->root SHOULD have been set to:
(gdb) print files->root
$12 = (struct Node *) 0x400660
(gdb) step
(gdb) print files->root
$13 = (struct Node *) 0x400660
(gdb) print root
$14 = (Node *) 0x602030
Does anyone have any idea as to why an assignment might not work in this case? This is currently ruining my whole project, so any insight would be greatly appreciated. Thank you!!!
It looks like your mkfs function is accepting a pointer to an already-existing Filesystem and then you are trying to allocate memory for a new Filesystem at a new memory location. There are two common conventions for a function like this: either it accepts no parameters and returns a pointer to a struct, or it accepts a pointer to an already-allocated struct and populates that struct. The reason it appears like the data isn't changing is that you're actually creating and populating a second struct, and leaving the caller's struct unchanged.
Here's an example of the first case, simplifying the function to just the memory allocation part:
Filesystem * mkfs() {
Filesystem *files = (Filesystem *)malloc(sizeof(Filesystem));
// (error handing omitted for brevity)
// populate the files struct as appropriate...
Node *root = (Node *)malloc(sizeof(Node));
files->root = root;
// etc, etc as you currently have
return files;
}
// In this case you should also provide a way for the caller to free a filesystem,
// which will free everything you allocated during mkfs:
void freefs(Filessystem *files) {
// first free any buffers you allocated inside the struct. For example:
free(files->root);
// then free the main filesystem struct
free(files);
}
The caller then deals with this object using these two functions. For example:
int main() {
Filesystem *files = mkfs();
// now "files" is ready to use
freefs(files); // free the objects when we're done with them.
}
Here's an example of the second case, which assumes that the caller already allocated an appropriate buffer, and it just needs to be populated:
void mkfs(Filesystem *files) {
// populate the files struct as appropriate...
Node *root = (Node *)malloc(sizeof(Node));
files->root = root;
// etc, etc as you currently have
}
void freefs(Filesystem *files) {
// still need to clean up all of the ancillary objects
free(files->root);
// etc, etc
}
In this case the calling function has some more work to do. For example:
int main() {
Filesystem *files = (Filesystem *)malloc(sizeof(Filesystem));
mkfs(files);
// now "files" is ready to use
freefs(files); // free the objects when we're done with them.
}
Both patterns are valid; the former is useful if you expect that the caller will need to be able to control how memory is allocated. For example, the caller might decide to allocate the filesystem on the stack rather than the heap:
int main() {
Filesystem files;
mkfs(&files);
// now "files" is ready to use
freefs(&files); // free the ancillary objects when we're done with them.
// "files" is still allocated here, but it's no longer valid
}
The latter takes care of the allocation on behalf of the caller. Since your function allocates further structures on the heap it's necessary to include a cleanup function in both cases.
I am implementing FIFO in C. One thread is writing in FIFO and other is reading from it.
#define BUFFER_LENGTH 1000
struct Frame
{
char data[1024];
unsigned int data_len;
struct Frame* frame;
};
struct Frame * get_from_fifo ()
{
if (!fifo_length)
{
first = last = NULL;
return NULL;
}
struct Frame* frame = first;
first = first->frame;
fifo_length--;
return frame;
}
int add_to_fifo (const char* data, unsigned int frame_size)
{
if (fifo_length >= BUFFER_LENGTH)
{
ast_log(LOG_ERROR, "Buffer full\n");
return SURESH_ERROR;
}
struct Frame* frame = malloc(sizeof (struct Frame));
frame->data_len = frame_size;
memcpy(frame->data, data, frame_size);
if (last)
{
last->frame = frame;
last = frame;
}
if (!first)
{
first = last = frame;
}
fifo_length++;
return SURESH_SUCCESS;
}
how can I prevent functions *add_to_fifo* and *get_from_fifo* to be called at the same time by different threads. i.e. *get_from_fifo* should only be called when the other thread is not executing *add_to_fifo* and vice verca.
As you are implementing FIFO stack the only really concurrent operation you have is changing the stack size (fifo_length).
You are adding entries to the tail of the stack and removing entries from the head of the stack so these two operation will never interfere with each other. So the only part you will need to worry about is changing the stack size (fifo_length), I would put it into separate function synchronised by mutex or flag (as mentioned by "Joey" above) and call it from both add_to_fifo() and get_from_fifo() functions.
You need to use a mutex (mutual exclusion) variable. The pthread library has everything you will need. Here's a good place to start looking at the available functions:
http://pubs.opengroup.org/onlinepubs/009695399/basedefs/pthread.h.html
You'll need to init a mutex variable that each thread will have access to:
http://pubs.opengroup.org/onlinepubs/009695399/functions/pthread_mutex_init.html
Then your threads will need to lock it when they need access to shared memory, and then unlock it when they are done using the shared memory:
http://pubs.opengroup.org/onlinepubs/009695399/functions/pthread_mutex_lock.html
Here's a simple example:
http://publib.boulder.ibm.com/infocenter/iseries/v5r3/index.jsp?topic=%2Frzahw%2Frzahwe18rx.htm
Good luck!