I want to add a standard header to a file (or group of files) using C. These files could be quite large, so it would be a bad idea to load them into memory, or copy them into temporary files (I think).
Is there a way to simply prepend the header directly to each file?
The header itself is quite small, not more than 1 KB
You cannot insert data into a file.
However, there is no need to load the entire file in memory. Just create a new file, write the data you are inserting, then copy the contents of the original file to the new file (do it block by block instead of loading the entire file into memory).
Finally, delete the original file and rename the new file to match the original file.
This is the most efficient way to do this and it is reasonably efficient.
It should be possible without a temporary file - you can read the file from the end, block by block, writing each block back at (original_position + header_size). The first block would be written back at header_size, leaving room for the header.
However, you don't really want to do this. It would corrupt the file if aborted (think: out of disk space, other I/O error, power down, whatever).
Thus, you should actually use temporary file - write to it everything you need, then rename it to the original file's name (assuming you create temporary file on the same file system, otherwise you'd need to copy).
Edit: to clarify what I mean, simplified solution when the whole file fits in RAM:
allocate buffer same size as the file
open the file, and read it into the buffer
seek(file, header_size) and write the buffer here
seek(file, 0) write the header
If the file is to big, you can allocate smaller buffer and repeat reads/writes starting with read at file_size - buffer_size and write at file_size - buffer_size + header_size. Then repeat with next chunk read at file_size - 2 * buffer_size, write at file_size - 2 * buffer_size + header_size, and so on.
But let me repeat: you risk corrupting your file if it fails!
Related
Is it possible to write at the middle of a file for example I want to insert some string at the 5th position in the 2nd line of a file in c ?
I'm not very familiar with some of C functions that are related to handling files , if someone could help me I would appreciate it
I tried using fputs but I couldn't insert characters at the desired location
open a new output file
read the input file line by line (fgets) writing each line out to a new file as you read.
When you hit the place you want to insert write the new line(s)
The carry on copy the old lines to the new file
close input and output
rename output file to input
Continuing from my comments above. Here's what I'd do:
Create two large, static char[] buffers of the same size--each large enough to store the largest file you could possibly ever need to read in (ex: 10 MiB). Ex:
#define MAX_FILE_SIZE_10_MIB (10*1024*1024)
static char buffer_file_in[MAX_FILE_SIZE_10_MIB];
static char buffer_file_out[MAX_FILE_SIZE_10_MIB];
Use fopen(filename, "r+") to open the file as read/update. See: https://cplusplus.com/reference/cstdio/fopen/. Read the chars one-by-one using fgetc() (see my file_load() function for how to use fgetc()) into the first large char buffer you created, buffer_file_in. Continue until you've read the whole file into that buffer.
Find the location of the place you'd like to do the insertion. Note: you could do this live as you read the file into buffer_file_in the first time by counting newline chars ('\n') to see what line you are on. Copy chars from buffer_file_in to buffer_file_out up to that point. Now, write your new contents into buffer_file_out at that point. Then, finish copying the rest of buffer_file_in into buffer_file_out after your inserted chars.
Seek to the beginning of the file with fseek(file_pointer, 0, SEEK_SET);
Write the buffer_file_out buffer contents into the file with fwrite().
Close the file with fclose().
There are some optimizations you could do here, such as storing the index where you want to begin your insertion, and not copying the chars up to that point into buffer_file_in, but rather, simply copying the remaining of the file after that into buffer_file_in, and then seeking to that point later and writing only your new contents plus the rest of the file. This avoids unnecessarily rewriting the very beginning of the fie prior to the insertion point is all.
(Probably preferred) you could also just copy the file and the changes you insert straight into buffer_file_out in one shot, then write that back to the file starting at the beginning of the file. This would be very similar to #pm100's approach, except using 1 file + 1 buffer rather than 2 files.
Look for other optimizations and reductions of redundancy as applicable.
My approach above uses 1 file and 1 or 2 buffers in RAM, depending on implementation. #pm100's approach uses 2 files and 0 buffers in RAM (very similar to what my 1 file and 1 buffer approach would look like), depending on implementation. Both approaches are valid.
I've written a simple http server in C and am now trying to implement HTML files.
For this I need send a response, containing the content of the HTML file.
How do I do that best?
Do I read the file line by line, and if so how do I store them in a single string?
Thanks already!
Here is an example of reading a text file by chunks which, if the file is big, would be faster than reading the file line by line.
As #tadman said in his comment, text files aren't generally big so reading them in chunks doesn't make any real difference in speed but web servers serve other files too - like perhaps photos or movies (which are big). So if you are only going to read text files then reading line by line might be simpler (you could use fgets instead of fread) but if you are going to read other kinds of files then reading all of them in chunks means you can do it the same way for all of them.
However, as #chux said in his comment, there is another difference between reading text files and binary files. The difference is that text files are opened in text mode: fopen(filename,"r"); and binary files must be opened in binary mode: fopen(filename,"rb"); A web server could probably open all files in binary mode because web browsers ignore whitespace anyway but other kinds of programs need to know what the line endings will be so it can make a difference.
https://onlinegdb.com/HkM---r2X
#include <stdio.h>
int main()
{
// we will make the buffer 200 bytes in size
// this is big enough for the whole file
// in reality you would probably stat the file
// to find it's size and then malloc the memory
// or you could read the file twice:
// - first time counting the bytes
// - second time reading the bytes
char buffer[200]="", *current=buffer;
// we will read 20 bytes at a time to show that the loop works
// in reality you would pick something approaching the page size
// perhaps 4096? Benchmarking might help choose a good size
int bytes, chunk=20, size=sizeof(buffer)/sizeof(char);
// open the text file in text mode
// if it was a binary file you would need "rb" instead of "r"
FILE *file=fopen("test.html","r");
if(file)
{
// loop through reading the bytes
do {
bytes=fread(current,sizeof(char),chunk,file);
current+=bytes;
} while (bytes==chunk);
// close the file
fclose(file);
// terminate the buffer so that string function will work
*current='\0';
// print the buffer
printf("%s",buffer);
}
return 0;
}
I created a File of 4000 blocks with a blocksize of 4096 Bytes.
Now I want to manipulate single blocks and read them again without changeing the files' size.
Actually I want to write blocks out of another file to specific blocks in the file I created.
Therefore I am opening the Files in binarymode like this:
FILE * storeFile=fopen(targetFile, "wb"); // this one I created before
FILE * sourceFILE=fopen(sourceFile,"rb");
now I am trying to read stuff to a pointer
char * ptr=malloc(4096);
...
for(i=0; i<blocks_needed; i++)
{
fread(ptr,4096,1,sourceFile);
// now I am going to the position of the blocks I want to write to
fseek(storeFile,freeBlocks[i]*4096,SEEK_SET);
// and now I am writing it to the File I created before
fwrite(ptr,4096,1,storeFile);
...
}
For some reason the File I created before changes it's size and becomes a copy of the file I wanted to write into it.
What am I doing wrong?
Thank you in advance!
From the fopen man page:
``w'' Truncate to zero length or create text file for writing. The stream is positioned at the beginning of the file.
You're erasing the destination file every time you open it. You might be interested in a or a+:
``a'' Open for writing. The file is created if it does not exist. The stream is positioned at the end of the file. Subsequent writes to the file will always end up at the then current end of file, irrespective of any intervening fseek(3) or similar.
``a+'' Open for reading and writing. The file is created if it does not exist. The stream is positioned at the end of the file. Subsequent writes to the file will always end up at the then current end of file, irrespective of any intervening fseek(3) or similar.
The problem is that your seek needs to be to some byte offset from the start of the file.
As the blocks are 4096 in length the offset would be (long)i * 4096;
I think you are seeking to the wrong position as the freeBlocks[i] is presumably an address.
I want to read a file and change its content and write it back to the file.
I use open to read a file as follows:
bfd = open(m_file_name.c_str(), O_RDWR)
But when I write, it is kinda append it to the old one. How can I overwrite it?
You can use lseek(2)
bfd = open(m_file_name.c_str(), O_RDWR);
// read your file
lseek(bfd, 0, SEEK_SET);
// do whatever manipulation & write file
If your file is now less in size than the original, you will need to truncate the size to the new size, or you'll leave the bytes from the old end of the file at the end. If it's larger, the file should grow automatically as you write.
You have to seek with lseek if you want to write at a specific position.
You are not trying to overwrite the entire file, right? Just a small part of it? If you are trying to overwrite the entire file then this is a bad way to do it.
How can I go back to the very beginning of a csv file and add rows?
(I'm printing to a CSV file from C using fprintf(). At the end of printing thousands of rows (5 columns) of data, I would like to go back to the top of the file and insert some dynamic header data (based on how things went printing everything). )
Thank You.
Due to the way files are structured, this is more or less impossible. In order to accomplish what you want:
write csv data to file1
write header to file2
copy contents of file1 to file2
delete file1
Or you can hold the csv data in ram and write it to file after you're finished processing and know the header.
Another option is to set aside a certain number of bytes for the header, which will work much faster for large files at minimal space cost. Since the space is allocated in the file at the start of the write, there aren't any issues going back and filling it in. Reopen the file as random access ("r+"), which points to the top of the file by default, write header, and close.
The simplest way would be to simply store the entire contents of the file in memory until you are finished, write out the header, and then write out the rest of the file.
If memory is an issue and you can't safely store the entire file in memory, or just don't want to, then you could write out the bulk of the CSV data to a temporary file, then when you are finished, write the header out to the primary file, and copy the data from the temporary file to the primary file in a loop.
If you wanted to be fancy, after writing the main CSV data out to the primary file, you could loop through the file from the beginning, read into memory the data that you're about to overwrite with the header, then write the header over top of that data, and so forth, read each chunk into memory, overwrite it with the previous one until you reach the end and append the final chunk. In this way you "insert" data at the beginning, my moving the rest of the file down. I really wouldn't recommend this as it will mostly just add complexity without much benefit, unless there is a specific reason you can't do something simpler like using a temporary file.
I think that is not possible. Probably the easiest way would be to write the output to a temporary file, then create the data you need as the dynamic header, write them to the target file and append the previously created temporary file.
write enough blank spaces in the first line
write data
seek(0)
write header - last column will be padded with spaces