I have use the code like below
char *es_data;
fp_input = fopen(inp_path, "rb");
fseek(fp_input, 0, SEEK_END);
file_size = ftell(fp_input);
fseek(fp_input, 0, SEEK_SET);
es_data = (char*)malloc(file_size);
fread(es_data, 1, file_size, fp_input);
I have a file of 185mb, i.e., file_size = 190108334 bytes. For this file, malloc is crashing, and program is getting struck at this stage.
If i use any other file of lower size, it works fine.
What can I do ?
You should test at least that fopen succeeds:
fp_input = fopen(inp_path, "rb");
if (!fp_input) { popen(inp_path); exit(EXIT_FAILURE); };
Your malloc is probably not crashing, but failing (by returning NULL): read malloc(3) so code at least:
es_data = malloc(file_size);
if (!es_data) { perror("malloc"); exit(EXIT_FAILURE); }
BTW, you probably want to memory map a file. If you are on Linux or a Posix system, learn about mmap(2) (and use fstat(2) to query the size of an open(2)-ed file descriptor). Windows can also memory map a file with CreateFileMapping
If your malloc is indeed crashing this probably means that you have memory corruption (before that malloc call) so some internal invariant of your system malloc is violated. Use some memory debugger tool (like valgrind on Linux, or purify on Windows) to detect it.
Related
I have a doubt dealing with pointers in C.
During my program execution I must open and close a configuration file but since a different program updates its content i must do this several times while the program runs. So i made a little function which reads the file and closes it and i call it every time I need it.
int readConfigFile()
{
FILE* pfile = fopen("path to file", "r");
fseek(pFile, 0, SEEK_END);
lSize = ftell(pFile);
rewind(pFile);
text = (char*)malloc(sizeof(char) * lSize);
if (text == NULL) {
fputs("Memory error", stderr);
return (0);
}
if (fread(text, 1, lSize, pFile) <= 0) {
printf("Nothing read");
return 0;
}
text[lSize] = '\0';
//Do the actual reading
}
If i am not mistaken this functions creates a new FILE* pointer every time it runs however I was wondering if using a FILE* pointer with a larger scope and recycling it every time the function runs would be better. Meaning:
FILE* globalVaraible;
int readConfigFile(FILE* pFile)
{
*pfile = fopen("path to file", "r");
//Do some stuff here
}
I am worried that, since there is no garbage collector in C++ as far as i know, the first function will create too many FILE* pointers ,if runs for long enough, and overflow my memory. On the other hand having a global pointer, that will open and close in the context of a single function, seems rather wasteful and not very modular, at least for my point of view.
So should I recycle my pointer, or any reference for that matter, or is it okay to declare new ones as needed?
Thanks in advance.
You could use freopen().
http://en.cppreference.com/w/cpp/io/c/freopen
There is no guarantee that it will actually reuse dynamic memory allocated for the "old" file. This function may as well be a wrapper for fclose() and fopen() - it's all implementation defined.
From what I see, newlib (a popular libc for embedded targets) reuses the FILE storage, so there's a small gain here. Whether or not it makes a noticeable difference is another matter.
https://sourceware.org/git/gitweb.cgi?p=newlib-cygwin.git;a=blob;f=newlib/libc/stdio/freopen.c;h=fb1f6c4db3f294f672d7e6998e8c69215b7482c1;hb=HEAD
I have a legacy function accepting a FILE* pointer in a library. The contents I would like to parse is actually in memory, not on disk.
So I came up with the following steps to work around this issue:
the data is in memory at this point
fopen a temporary file (using tmpnam or tmpfile) on disk for writing
fclose the file
fopen the same file again for reading - guaranteed to exist
change the buffer using setvbuf(buffer, size)
do the legacy FILE* stuff
close the file
remove the temporary file
the data can be discarded
On windows, it looks like this:
int bufferSize;
char buffer[bufferSize];
// set up the buffer here
// temporary file name
char tempName [L_tmpnam_s];
tmpnam_s(tempName, L_tmpnam_s);
// open/close/reopen
fopen_s(&fp, tempName,"wb");
fclose(fp);
freopen_s(&fp, tempName,"rb", fp);
// replace the internal buffer
setvbuf(fp, buffer, _IONBF, bufferSize);
fp->_ptr = buffer;
fp->_cnt = bufferSize;
// do the FILE* reading here
// close and remove tmp file
fclose(fp);
remove(tempName);
Works, but quite cumbersome. The main problem, aside from the backwardness of this approach, are:
the temporary name needs to be determined
the temporary file is actually written to disk
the temporary file needs to be removed afterwards
I'd like to keep things portable, so using Windows memory-mapped functions or boost's facilities is not an option. The problem is mainly that, while it is possible to convert a FILE* to an std::fstream, the reverse seems to be impossible, or at least not supported on C++99.
All suggestions welcome!
Update 1
Using a pipe/fdopen/setvbuf as suggested by Speed8ump and a bit of twiddling seems to work. It does no longer create files on disk nor does it consume extra memory. One step closer, except, for some reason, setvbuf is not working as expected. Manually fixing it up is possible, but of course not portable.
// create a pipe for reading, do not allocate memory
int pipefd[2];
_pipe(pipefd, 0, _O_RDONLY | _O_BINARY);
// open the read pipe for binary reading as a file
fp = _fdopen(pipefd[0], "rb");
// try to switch the buffer ptr and size to our buffer, (no buffering)
setvbuf(fp, buffer, _IONBF, bufferSize);
// for some reason, setvbuf does not set the correct ptr/sizes
fp->_ptr = buffer;
fp->_charbuf = fp->_bufsiz = fp->_cnt = bufferSize;
Update 2
Wow. So it seems that unless I dive into the MS-specific implementation CreateNamedPipe / CreateFileMapping, POSIX portability costs us an entire memcopy (of any size!), be it to file or into a pipe. Hopefully the compiler understands that this is just a temporary and optimizes this. Hopefully.
Still, we eliminated the silly device writing intermediate. Yay!
int pipefd[2];
pipe(pipefd, bufferSize, _O_BINARY); // setting internal buffer size
FILE* in = fdopen(pipefd[0], "rb");
FILE* out = fdopen(pipefd[1], "wb");
// the actual copy
fwrite(buffer, 1, bufferSize, out);
fclose(out);
// fread(in), fseek(in), etc..
fclose(in);
You might try using a pipe and fdopen, that seems to be portable, is in-memory, and you might still be able to do the setvbuf trick you are using.
Your setvbuf hack is a nice idea, but not portable. C11 (n1570):
7.21.5.6 The setvbuf function
Synopsis
#include <stdio.h>
int setvbuf(FILE * restrict stream,
char * restrict buf,
int mode, size_t size);
Description
[...] If buf is not a null pointer, the array it points to may be used instead of a buffer allocated by the setvbuf function [...] and the argument size specifies the size of the array; otherwise, size may determine the size of a buffer allocated by the setvbuf function. The contents of the array at any time are indeterminate.
There is neither a guarantee that the provided buffer is used at all, nor about what it contains at any point after the setvbuf call until the file is closed or setvbuf is called again (POSIX doesn't give more guarantees).
The easiest portable solution, I think, is using tmpfile, fwrite the data into that file, fseek to the beginning (I'm not sure if temporary files are guaranteed to be seekable, on my Linux system, it appears they are, and I'd expect them to be elsewhere), and pass the FILE pointer to the function. This still requires copying in memory, but I guess usually no writing of the data to the disk (POSIX, unfortunately, implicitly requires a real file to exist). A file obtained by tmpfile is deleted after closing.
I'm writing a buffer into a binary file. Code is as in the following :
FILE *outwav = fopen(outwav_path, "wb");
if(!outwav)
{
fprintf(stderr, "Can't open file %s for writing.\n", outwav_path);
exit(1);
}
[...]
//Create sample buffer
short *samples = malloc((loopcount*(blockamount-looppos)+looppos) << 5);
if(!samples)
{
fprintf(stderr, "Error : Can't allocate memory.\n");
exit(1);
}
[...]
fwrite(samples, 2, 16*k, outwav); //write samplebuffer to file
fflush(outwav);
fclose(outwav);
free(samples);
The last free() call causes me random segfaults.
After several headaches I thought it was probably because the fwrite call would execute only after a delay, and then it would read freed memory. So I added the fflush call, yet, the problem STILL occurs.
The only way to get rid of it is to not free the memory and let the OS do it for me. This is supposed to be bad practice though, so I'd rather ask if there is no better solution.
Before anyone asks, yes I check that the file is opened correctly, and yes I test that the memory is allocated properly, and no, I don't touch the returned pointers in any way.
Once fwrite returns you are free to do whatever you want with the buffer. You can remove the fflush call.
It sounds like a buffer overflow error in a totally unrelated part of the program is writing over the book-keeping information that free needs to do its work. Run your program under a tool like valgrind to find out if this is the problem and to find the part of the program that has a buffer overflow.
All,
I'm using MapViewOfFile to hold part of a file in memory. There is a stream that points to this file and writes to it, and then is rewound. I use the pointer to the beginning of the mapped file, and read until I get to the null char I write as the final character.
int fd;
yyout = tmpfile();
fd = fileno(yyout);
#ifdef WIN32
HANDLE fm;
HANDLE h = (HANDLE) _get_osfhandle (fd);
fm = CreateFileMapping(
h,
NULL,
PAGE_READWRITE|SEC_RESERVE,
0,
4096,
NULL);
if (fm == NULL) {
fprintf (stderr, "%s: Couldn't access memory space! %s\n", argv[0], strerror (GetLastError()));
exit(GetLastError());
}
bp = (char*)MapViewOfFile(
fm,
FILE_MAP_ALL_ACCESS,
0,
0,
0);
if (bp == NULL) {
fprintf (stderr, "%s: Couldn't fill memory space! %s\n", argv[0], strerror (GetLastError()));
exit(GetLastError());
}
Data is sent to the yyout stream, until flushData() is called. This writes a null to the stream, flushes, and then rewinds the stream. Then I start from the beginning of the mapped memory, and read chars until I get to the null.
void flushData(void) {
/* write out data in the stream and reset */
fprintf(yyout, "%c%c%c", 13, 10, '\0');
fflush(yyout);
rewind(yyout);
if (faqLine == 1) {
faqLine = 0; /* don't print faq's to the data file */
}
else {
char * ps = bp;
while (*ps != '\0') {
fprintf(outstream, "%c%c", *ps, blank);
ps++;
}
fflush(outfile);
}
fflush(yyout);
rewind(yyout);
}
After flushing, more data is written to the stream, which should be set to the start of the memory area. As near as I can determine with gdb, the stream is not getting rewound, and eventually fills up the allocated space.
Since the stream points to the underlying file, this does not cause a problem initially. But, when I attempt to walk the memory, I never find the null. This leads to a SIGSEV. If you want more details of why I need this, see here.
Why am I not reusing the memory space as expected?
I think this line from the MSDN documentation for CreateFileMapping might be the clue.
A mapped file and a file that is accessed by using the input and output (I/O) functions (ReadFile and WriteFile) are not necessarily coherent.
You're not apparently using Read/WriteFile, but the documentation should be understood in terms of mapped views versus explicit I/O calls. In any case, the C RTL is surely implemented using the Win32 API.
In short, this approach is problematic.
I don't know why changing the view/file size helps; perhaps it just shifts the undefined behaviour in a direction that happens to be beneficial.
Well, after working on this for a while, I have a working solution. I don't know why this succeeds, so if someone comes up with something better, I'll be happy to accept their answer instead.
fm = CreateFileMapping(
h,
NULL,
PAGE_READWRITE|SEC_RESERVE,
0,
16384,
NULL);
As you can see, the only change is to the size declared from 4096 to 16384. Why this works when the total chars input at a time is no more than 1200, I don't know. If someone could provide details on this, I would appreciate it.
When you're done with the map, simply un-map it.
UnmapViewOfFile(bp);
I often use the website www.cplusplus.com as a reference when writing C code.
I was reading the example cited on the page for fread and had a question.
As an example they post:
/* fread example: read a complete file */
#include <stdio.h>
#include <stdlib.h>
int main () {
FILE * pFile;
long lSize;
char * buffer;
size_t result;
pFile = fopen ( "myfile.bin" , "rb" );
if (pFile==NULL) {fputs ("File error",stderr); exit (1);}
// obtain file size:
fseek (pFile , 0 , SEEK_END);
lSize = ftell (pFile);
rewind (pFile);
// allocate memory to contain the whole file:
buffer = (char*) malloc (sizeof(char)*lSize);
if (buffer == NULL) {fputs ("Memory error",stderr); exit (2);}
// copy the file into the buffer:
result = fread (buffer,1,lSize,pFile);
if (result != lSize) {fputs ("Reading error",stderr); exit (3);}
/* the whole file is now loaded in the memory buffer. */
// terminate
fclose (pFile);
free (buffer);
return 0;
}
It seems to me that that if result != lSize, then free(buffer) will never get called. Would this be a memory leak in this example?
I have always thought the examples on their site are of a very high quality. Perhaps I am not understanding correctly?
It wouldn't be a memory leak in this example, because terminating the program (by calling exit()) frees all memory associated with it.
However, it would be a memory leak if you used this piece of code as a subroutine and called something like return 1; in place of exit().
Technically, yes it is a memory leak. But any memory allocated by a process is automatically freed when that process terminates, so in this example the calls to free (and fclose) are not really required.
In a more complex program, this would probably be a real problem. The missing free would create a memory leak and the missing fclose would cause a resource leak.
The operating system cleans up any unfreed memory by a process when that process closes. At least, modern operating systems do.
If the program were not exiting at the point result != lSize, that is, it continued with some other path of execution, then yes - it is a guaranteed memory leak.
There are two possible paths.
(1) result != lSize - in this case, exit(0) is called. This kills the process and the operating system will clean up the memory.
(2) result == lsize - in this case, the buffer is explicitly freed, but return is called right afterwards so the free is mostly just good style because this also kills the process and the operating system will, again, clean up the memory.
So in this simple case, there is no memory leak. But it is probably a good practice to just make sure you're freeing any memory you've allocated in any application you write. Getting into this habit will prevent many headaches for you in the future.
As to possible memory leakage, other's have already answered that question. A while ago, I posted a variation of the given code which should handle all possible error conditions correctly:
fsize()
fget_contents()