I am using fopen to write to a binary file and using the cstdio (stdio.h) library due to legacy code and it must be cross-platform compatible with Windows and Linux.
For the prototype, FILE * fopen ( const char * filename, const char * mode );, I am using const char * mode = "ab", which will append to a binary file. Writing operations append data at the end of the file. The file is created if it does not exist.
I have N number of input files where I process data from and write to one output file for each type, where I have M types. I process one input file and write the data to each corresponding output file. I then will close that ith input file and open (i + 1)th, and repeat the process by appending the data from the input file to the output files.
If an output file exists at the beginning on the executable, I want it deleted. If it exists and I don't delete it, then when I use the "wb" mode, then it will just append data to the output file, which will result in duplication I don't want. I am open to a boost solution and I like to maintain standards best as possible (i.e avoid POSIX if possible)
fopen C++ reference
Here is one way
char* infile[N] = //input names
char* outfile[M] = //output names
int i, j;
for (i = 0; i < N; i++){
//process input
char* mode = "ab";
if (i == 0) mode = "wb";
for (j = 0; j < M; j++){
FILE* f = fopen(outfile[j], mode);
//write to file
fclose(f);
}
}
The "w" mode should overwrite the file. It is "a" mode that will avoid deleting a file that already exists.
EDIT: You can also remove (const char * filename) if you want to delete the files at the beginning of execution. If that's the case then you never have to use the "wb" mode.
One possibility would be to use open (_open for Windows) to create the appropriate file handle and then use fdopen (_fdopen for Windows) to create a stdio handle out of it. You will need some preprocessor magic to handle the fact that the names are not exactly the same in Linux and Windows:
// Allow us to use Posix compatible names on Windows
#ifdef WINDOWS
#define open _open
#define O_CREAT _O_CREAT
#define O_TRUNC _O_TRUNC
#define O_APEND _O_APPEND
#define S_IREAD _S_IREAD
#define S_IWRITE _S_IWRITE
#define fdopen _fdopen
#endif
int fd = open(filename, O_CREAT | O_TRUNC | O_APPEND, S_IREAD | S_IWRITE);
FILE *fp = fdopen(fd, "a");
If you want to overwrite rather than append, why not just use the mode "wb"? "w" overwrites the file when writing.
Related
I have some C code running on a linux OS into a portable device. I'm using a VRmagic system, similar to BeagleBone etc...).
In this code, I'm using the following function to write results inside a txt file.
//globale definition
FILE *logfile;
const char *logpath = "/MY_DEVICE/log.txt";
const char *main_folder_result_path = "/MEASUREMENT_RESULTS/";
const char *all_measurement_results_file_name = "all_computation_data.txt";
void save_to_log_file(const char *logpath,const char *message){
#ifdef savealllog
logfile = fopen(logpath,"a"); //logpath = "/MY_DEVICE/log.txt";
fprintf(logfile,"%s",message);
fclose(logfile);
#endif
#ifdef printalloutput
printf("%s",message);
#endif
}
void append_all_measurement_file(){
char buff[255];
char filename[255];
save_to_log_file(logpath," Appending all measurement file...");
sprintf(filename,"%s%s",main_folder_result_path,all_measurement_results_file_name);
//here after we create the header if the file does not exist already
FILE *pFile_all_measurement_results = fopen(filename, "r"); //lets try to read the file
if (pFile_all_measurement_results == NULL){ // if file does not exist
pFile_all_measurement_results = fopen(filename, "w");
fprintf(pFile_all_measurement_results,"date-time S_type Part_name batch count abs value_1 value_2 value_3 value_4\n");
fclose(pFile_all_measurement_results);
}
else{
fclose(pFile_all_measurement_results); //if file does exist then we have to close it here
}
//here after we are going to write results
pFile_all_measurement_results = fopen(filename, "a"); //lets open the file in append mode
fprintf(pFile_all_measurement_results,"%s %s %s %d %d %d ",dateandtimetps.dt,measurement_type_str,Name_Str, batch_number,count_number,absolute_measurement_number);
fprintf(pFile_all_measurement_results,"%.03f ", value_1);
fprintf(pFile_all_measurement_results,"%.03f ", value_2);
fprintf(pFile_all_measurement_results,"%.03f ", value_3);
fprintf(pFile_all_measurement_results,"%.03f\n", value_4); //(there are a bit more in reality.....)
fclose(pFile_all_measurement_results); //we can now close the file
save_to_log_file(logpath,"done.\n");
}
99.9% of the time all is OK.
But, randomly, I do have some NUL character in my file, and this happen when I turn OFF my system.
Looks like the file has not been closed properly or something like that for some reason.....
When I get my txt file, and open it with notepad++ on my computer, it does look like the following:
I can confirm that the device has been turned OFF between line 172 and line 174.
Many thanks for help
Just closing a file on Unix systems does not implicate the contents are instantly written to disk; usually buffering/caching is enabled.
To make sure the contents are written you can use sync() immediately after a write and/or close operation, which will minimize the risk of lost updates.
Additionally I recommend to use a journaling filesystem like ext4 (which may not be an option on external drives like USB pens, but is strongly recommend for all system and data partitions). This will not save you from data loss in case of a power failure or crash, but will avoid inconsistencies/partial writes like you are experiencing.
I need to concurrently read from a file in different offsets using C.
dup unforunately creates a file descriptor that shares offset and flags with the original.
Is there a function like dup that does not share the offset and flags?
EDIT I only have access to the file pointer FILE* fp; I do not have the file path
EDIT This program is compiled for windows in addition to mac and many flavors of linux
SOLUTION
We can use pread on posix systems, and I wrote a pread function for windows which solves this problem
https://github.com/Storj/libstorj/blob/master/src/utils.c#L227
On Linux, you can recover the filename from /proc/self/fd/N, where N is the integral value of the file descriptor:
sprintf( linkname, "/proc/self/fd/%d", fd );
Then use readlink() on the resulting link name.
If the file has been renamed or deleted, you may be out of luck.
But why do you need another file descriptor? You can use pread() and/or pwrite() on the original file descriptor to read/write from/to the file without affecting the current offset. (caveat: on Linux, pwrite() to a file opened in append mode is buggy - POSIX states that pwrite() to a file opened in append mode will write to the offset specified in the pwrite() call, but the Linux pwrite() implementation is broken and will ignore the offset and append the data to the end of the file - see the BUGS section of the Linux man page)
No, neither C nor POSIX (since you mention dup()) has a function for opening a new, independent file handle based on an existing file handle. As you observed, you can dup() a file descriptor, but the result refers to the same underlying open file description.
To get an independent handle, you need to open() or fopen() the same path (which is possible only if the FILE refers to an object accessible through the file system). If you don't know what path that is, or if there isn't any in the first place, then you'll need a different approach.
Some alternatives to consider:
buffer some or all of the file contents in memory, and read as needed from the buffer to serve your needs for independent file offsets;
build an internal equivalent of the tee command; this will probably require a second thread, and you'll probably not be able to read one file too far ahead of the other, or to seek in either one;
copy the file contents to a temp file with a known name, and open that as many times as you want;
if the FILE corresponds to a regular file, map it into memory and access its contents there. The POSIX function fmemopen() could be useful in this case to adapt the memory mapping to your existing stream-based usage.
On windows (assuming VisualStudio), you can get access to the OS file handle from the stdio FILE handle.
From there, reopen it and convert back to a new FILE handle.
This is windows only, but I think Andrews answer will work for Linux and probably the Mac as well - unfortunately there is no portable way to have it work on all systems.
#include <Windows.h>
#include <fcntl.h>
#include <io.h>
#include <stdio.h>
FILE *jreopen(FILE* f)
{
int n = _fileno(f);
HANDLE h = (HANDLE)_get_osfhandle(n);
HANDLE h2 = ReOpenFile(h, GENERIC_READ, FILE_SHARE_READ, 0);
int n2 = _open_osfhandle((intptr_t)h2, _O_RDONLY);
FILE* g = _fdopen(n2, "r");
return g;
}
I was able to use pread and pwrite on POSIX systems, and I wrapped ReadFile/WriteFile on Windows Systems into pread and pwrite functions
#ifdef _WIN32
ssize_t pread(int fd, void *buf, size_t count, uint64_t offset)
{
long unsigned int read_bytes = 0;
OVERLAPPED overlapped;
memset(&overlapped, 0, sizeof(OVERLAPPED));
overlapped.OffsetHigh = (uint32_t)((offset & 0xFFFFFFFF00000000LL) >> 32);
overlapped.Offset = (uint32_t)(offset & 0xFFFFFFFFLL);
HANDLE file = (HANDLE)_get_osfhandle(fd);
SetLastError(0);
bool RF = ReadFile(file, buf, count, &read_bytes, &overlapped);
// For some reason it errors when it hits end of file so we don't want to check that
if ((RF == 0) && GetLastError() != ERROR_HANDLE_EOF) {
errno = GetLastError();
// printf ("Error reading file : %d\n", GetLastError());
return -1;
}
return read_bytes;
}
ssize_t pwrite(int fd, const void *buf, size_t count, uint64_t offset)
{
long unsigned int written_bytes = 0;
OVERLAPPED overlapped;
memset(&overlapped, 0, sizeof(OVERLAPPED));
overlapped.OffsetHigh = (uint32_t)((offset & 0xFFFFFFFF00000000LL) >> 32);
overlapped.Offset = (uint32_t)(offset & 0xFFFFFFFFLL);
HANDLE file = (HANDLE)_get_osfhandle(fd);
SetLastError(0);
bool RF = WriteFile(file, buf, count, &written_bytes, &overlapped);
if ((RF == 0)) {
errno = GetLastError();
// printf ("Error reading file :%d\n", GetLastError());
return -1;
}
return written_bytes;
}
#endif
This question already has answers here:
Closed 12 years ago.
Possible Duplicate:
Getting Filename from file descriptor in C
Is there a simple and (reasonably) portable way of getting the filename from a FILE*?
I open a file using f = fopen(filename, ...) and then pass down f to various other functions, some of which may report an error. I'd like to report the filename in the error message but avoid having to pass around the extra parameter.
I could create a custom wrapper struct { FILE *f, const char *name }, but is there perhaps a simpler way? (If the FILE* wasn't opened using fopen I don't care about the result.)
On some platforms (such as Linux), you may be able to fetch it by reading the link of /proc/self/fd/<number>, as so:
#include <stdio.h>
#include <unistd.h>
#include <string.h>
int main(void)
{
char path[1024];
char result[1024];
/* Open a file, get the file descriptor. */
FILE *f = fopen("/etc/passwd", "r");
int fd = fileno(f);
/* Read out the link to our file descriptor. */
sprintf(path, "/proc/self/fd/%d", fd);
memset(result, 0, sizeof(result));
readlink(path, result, sizeof(result)-1);
/* Print the result. */
printf("%s\n", result);
}
This will, on my system, print out /etc/passwd, as desired.
It's a bit difficult, because a FILE* can read/write from a file handle which isn't associated with a named file at all (for example an unnamed pipe or a socket). You can obtain the file handle with fileno() and then there are system specific ways to learn about the file name. Here's a discussion on how to do this under Linux:
Getting Filename from file descriptor in C
and under Windows this isn't much easier either:
http://msdn.microsoft.com/en-us/library/aa366789(VS.85).aspx (as an extra step here, you use _get_osfhandle() to get the Windows file handle from the c-library file descriptor)
This question already has answers here:
Closed 12 years ago.
Possible Duplicate:
Getting Filename from file descriptor in C
Is there a simple and (reasonably) portable way of getting the filename from a FILE*?
I open a file using f = fopen(filename, ...) and then pass down f to various other functions, some of which may report an error. I'd like to report the filename in the error message but avoid having to pass around the extra parameter.
I could create a custom wrapper struct { FILE *f, const char *name }, but is there perhaps a simpler way? (If the FILE* wasn't opened using fopen I don't care about the result.)
On some platforms (such as Linux), you may be able to fetch it by reading the link of /proc/self/fd/<number>, as so:
#include <stdio.h>
#include <unistd.h>
#include <string.h>
int main(void)
{
char path[1024];
char result[1024];
/* Open a file, get the file descriptor. */
FILE *f = fopen("/etc/passwd", "r");
int fd = fileno(f);
/* Read out the link to our file descriptor. */
sprintf(path, "/proc/self/fd/%d", fd);
memset(result, 0, sizeof(result));
readlink(path, result, sizeof(result)-1);
/* Print the result. */
printf("%s\n", result);
}
This will, on my system, print out /etc/passwd, as desired.
It's a bit difficult, because a FILE* can read/write from a file handle which isn't associated with a named file at all (for example an unnamed pipe or a socket). You can obtain the file handle with fileno() and then there are system specific ways to learn about the file name. Here's a discussion on how to do this under Linux:
Getting Filename from file descriptor in C
and under Windows this isn't much easier either:
http://msdn.microsoft.com/en-us/library/aa366789(VS.85).aspx (as an extra step here, you use _get_osfhandle() to get the Windows file handle from the c-library file descriptor)
I would like to create a binary file representing an integer. I think the file should be 4 bytes. I use linux. How to do that?
Another question: How do I assign the content of that file to an integer in C?
In standard C, fopen() allows the mode "wb" to write (and "rb" to read) in binary mode, thus:
#include <stdio.h>
int main() {
/* Create the file */
int x = 1;
FILE *fh = fopen ("file.bin", "wb");
if (fh != NULL) {
fwrite (&x, sizeof (x), 1, fh);
fclose (fh);
}
/* Read the file back in */
x = 7;
fh = fopen ("file.bin", "rb");
if (fh != NULL) {
fread (&x, sizeof (x), 1, fh);
fclose (fh);
}
/* Check that it worked */
printf ("Value is: %d\n", x);
return 0;
}
This outputs:
Value is: 1
From the operating system's point of view, all files are binary files. C (and C++) provide a special "text mode" that does stuff like expanding newline characters to newline/carriage-return pairs (on Windows), but the OS doesn't know about this.
In a C program, to create a file without this special treatment, use the "b" flag of fopen():
FILE * f = fopen("somefile", "wb" );
Open the file for binary read/write. fopen takes a b switch for file access mode parameter - see here
See the fopen page in Wikipedia for the difference between text and binary files as well as a code sample for writing data to a binary file
See man for syscalls open, write and read.