I'm trying to read data from a file into a buffer. The data in file is of 900K bytes. (seek to end of file and ftell()). Allocated the buffer in which the data is to be read of size 900K + 1 (to null terminate). My question is that fread() returns 900K but the I see the strlen(buffer) it shows lesser value and in the buffer at the last I can see something like ".....(truncated)". Why is this behavior? Is there a limit with fread() beyond which we cannot read into buffer and it will truncate it. Also why the return value of fread() says 900K even though actually it has read even less.?
strlen does something along these lines:
int strlen(char *str)
{
int len = 0;
while(*str++) len++;
return len;
}
If your file contains binary data (or if it's a text file with a UTF encoding and unused upper bytes) strlen is going to stop at the first 0x00 byte it encounters and return how many bytes into the file that was encountered. If you read a text file in a single-byte encoding like ANSI there won't be a null terminator and calling strlen will invoke undefined behavior.
If you want to determine how many bytes that fread successfully read out of the file, check its return value.1
If you want to determine the file size before reading a file, do this:
size_t len;
fseek(fp, 0, SEEK_END);
len = ftell(fp);
rewind(fp);
len will contain the file's size in bytes.
1: Assuming you called fread with parameter 2 set to 1 byte per element and didn't try to read more bytes than are actually in the file.
Your main question has already been answered, though it's worth notice that strlen is not designed to measure the size of an array but a NULL-terminated string. It probably prints a lower value because strlen returns the number of characters that appear before a null-char, so if you have nullchars ('\0') through your data, strlen will stop as soon as it finds one of them.
You should trust fread 's return value.
EDIT: as a note, fread MAY read less bytes than requested, and it can be caused by an error or an end of file. You can check it with ferror and feof, respectively.
Related
Consider this code to read a text based file. This sort of fread() usage was briefly touched upon in the excellent book C Programming: A Modern Approach by K.N. King.
There are other methods of reading text based files, but here I am concerned with fread() only.
#include <stdio.h>
#include <stdlib.h>
int main(void)
{
// Declare file stream pointer.
FILE *fp = fopen("Note.txt", "r");
// fopen() call successful.
if(fp != NULL)
{
// Navigate through to end of the file.
fseek(fp, 0, SEEK_END);
// Calculate the total bytes navigated.
long filesize = ftell(fp);
// Navigate to the beginning of the file so
// it can be read.
rewind(fp);
// Declare array of char with appropriate size.
char content[filesize + 1];
// Set last char of array to contain NULL char.
content[filesize] = '\0';
// Read the file content.
fread(content, filesize, 1, fp);
// Close file stream pointer.
fclose(fp);
// Print file content.
printf("%s\n", content);
}
// fopen() call unsuccessful.
else
{
printf("File could not be read.\n");
}
return 0;
}
There are some problems I have with this method. My opinion is that this is not a safe method of performing fread() since there might be an overflow if we try to read an extremely large string. Is this opinion valid?
To circumvent this issue, we may use a buffer size and keep on reading into a char array of that size. If filesize is less than buffer size, then we simply perform fread() once as described in the above code. Otherwise, We divide the total file size by the buffer size and get a result, whose int portion we will use as the total number of times to iterate a loop where we will invoke fread() each time, appending the read buffer array into a larger string. Now, for the final fread(), which we will perform after the loop, we will have to read exactly (filesize % buffersize) bytes of data into an array of that size and finally append this array into the larger string (Which we would have malloc-ed with filesize + 1 beforehand). I find that if we perform fread() for the last chunk of data using buffersize as its second parameter, then extra garbage data of size (buffersize - chunksize) will be read in and the data might become corrupted. Are my assumptions here correct? Please explain if/ how I have overlooked something.
Also, there is the issue that non-ASCII characters might not have size of 1 byte. In that case I would assume the proper amount is being read, but each byte is being read as a single char, so the text is distorted somehow? How is fread() handling reading of multi-byte chars?
this is not a safe method of performing fread() since there might be an overflow if we try to read an extremely large string. Is this opinion valid?
fread() does not care about strings (null character terminated arrays). It reads data as if it was in multiples of unsigned char*1 with no special concern to the data content if the stream opened in binary mode and perhaps some data processing (e.g. end-of-line, byte-order-mark) in text mode.
Are my assumptions here correct?
Failed assumptions:
Assuming ftell() return value equals the sum of fread() bytes.
The assumption can be false in text mode (as OP opened the file) and fseek() to the end is technical undefined behavior in binary mode.
Assuming not checking the return value of fread() is OK. Use the return value of fread() to know if an error occurred, end-of-file and how many multiples of bytes were read.
Assuming error checking is not required. , ftell(), fread(), fseek() instead of rewind() all deserve error checks. In particular, ftell() readily fails on streams that have no certain end.
Assuming no null characters are read. A text file is not certainly made into one string by reading all and appending a null character. Robust code detects and/or copes with embedded null characters.
Multi-byte: assuming input meets the encoding requirements. Example: robust code detects (and rejects) invalid UTF8 sequences - perhaps after reading the entire file.
Extreme: Assuming a file length <= LONG_MAX, the max value returned from ftell(). Files may be larger.
but each byte is being read as a single char, so the text is distorted somehow? How is fread() handling reading of multi-byte chars?
fread() does not function on multi-byte boundaries, only multiples of unsigned char. A given fread() may end with a portion of a multi-byte and the next fread() will continue from mid-multi-byte.
Instead of of 2 pass approach consider 1 single pass
// Pseudo code
total_read = 0
Allocate buffer, say 4096
forever
if buffer full
double buffer_size (`realloc()`)
u = unused portion of buffer
fread u bytes into unused portion of buffer
total_read += number_just_read
if (number_just_read < u)
quit loop
Resize buffer total_read (+ 1 if appending a '\0')
Alternatively consider the need to read the entire file in before processing the data. I do not know the higher level goal, but often processing data as it arrives makes for less resource impact and faster throughput.
Advanced
Text files may be simple ASCII only, 8-bit code page defined, one of various UTF encodings (byte-order-mark, etc. The last line may or may not end with a '\n'. Robust text processing beyond simple ASCII is non-trivial.
ASCII and UTF-8 are the most common. IMO, handle 1 or both of those and error out on anything that does not meet their requirements.
*1 fread() reads in multiple of bytes as per the 3rd argument, which is 1 in OP's case.
// v --- multiple of 1 byte
fread(content, filesize, 1, fp);
So what I'm trying to do is open a file and read it until the end in blocks that are 256 bytes long each time it is called. My dilemma is using fgets() or fread() to do it.
I was using fgets() initially, because it returns a string of the bytes that were read, which is great because I can store that data and work with it. However, in my particular file that I'm reading, the 256 bytes often happen over a more than 2 lines, which is a problem because fgets() stops reading when it hits a newline character or the end of the file.
I then thought of using fread(), but I don't know how to save the line that I'm referring to with it because fread() returns an int referring to the number of elements successfully read (according to its documentation).
I've searched and thought of solutions for a while now and can't find anything that works with my particular scenario. I would like some guidance on how to go about this issue, how would you go about this in my position?
You can use fread() to read each 256 bytes block and keep a lineCount variable to keep track of the number of new line characters you have encountered so far in the input. Since you have to process the blocks already this wouldn't mean much of an overhead in the processing.
To read a block of 256 chars, which is what I think you are doing, you just need to create a buffer of chars that can hold 256 of them, in other words a char array of size 256.
#define BLOCK_SIZE 256
char block[BLOCK_SIZE];
Then if you check the documentation for fread() it shows the following signature:
Following is the declaration for fread() function.
size_t fread(void *ptr, size_t size, size_t nmemb, FILE *stream)
Parameters
ptr -- This is the pointer to a block of memory with a minimum size of size*nmemb bytes.
size -- This is the size in bytes of each element to be read.
nmemb -- This is the number of elements, each one with a size of size bytes.
stream -- This is the pointer to a FILE object that an input stream.
So this means it takes a pointer to the buffer where it will write the read information, the size of each element it's supposed to read, the maximum amount of elements you want it to read and the file pointer. In your case it would be:
int read = fread(block, sizeof(char), BLOCK_SIZE, file);
This will copy the information from the file to the block array, which you can later process and keep track of the lines. The characters that were read by fread are in the block array, so the first char in the last read block would be block[0], the second block[1] and so on. The returned value in read indicates how many elements (in your case chars) were inserted in the array block when you call fread, this number will be equal to BLOCK_SIZE for every call, unless you reach the end of the file or there's an error.
I suggest you read some documentation for a full example, play a little with the code and do some reading on pointers in C to gain a better understanding of how everything works in general. If you still have questions after that, we can take it from there or you can create a new SO question.
isn't it possible to read bytes left in a file that is smaller than buffer size?
char * buffer = (char *)malloc(size);
FILE * fp = fopen(filename, "rb");
while(fread(buffer, size, 1, fp)){
// do something
}
Let's assume size is 4 and file size is 17 bytes. I thought fread can handle last operation as well even if bytes left in file is smaller than buffer size, but apparently it just terminates while loop without reading one last byte.
I tried to use lower system call read() but I couldn't read any byte for some reason.
What should I do if fread cannot handle last part of bytes that is smaller than buffer size?
Yep, turn your parameters around.
Instead of requesting one block of size bytes, you should request size blocks of 1 bytes. Then the function will return how many blocks (bytes) it was able to read:
int nread;
while( 0 < (nread = fread(buffer, 1, size, fp)) ) ...
try using "man fread"
it clearly mention following things which itself answers your question:
SYNOPSIS
size_t fread(void *ptr, size_t size, size_t nitems, FILE *stream);
DESCRIPTION
fread() copies, into an array pointed to by ptr, up to nitems items of
data from the named input stream, where an item of data is a sequence
of bytes (not necessarily terminated by a null byte) of length size.
fread() stops appending bytes if an end-of-file or error condition is
encountered while reading stream, or if nitems items have been read.
fread() leaves the file pointer in stream, if defined, pointing to the
byte following the last byte read if there is one.
The argument size is typically sizeof(*ptr) where the pseudo-function
sizeof specifies the length of an item pointed to by ptr.
RETURN VALUE
fread(), return the number of items read.If size or nitems is 0, no
characters are read or written and 0 is returned.
The value returned will be less than nitems only if a read error or
end-of-file is encountered. The ferror() or feof() functions must be
used to distinguish between an error condition and an end-of-file
condition.
I have a question:
I am using fread to read a file.
typedef struct {
int ID1;
int ID2;
char string[256];
} Reg;
Reg *A = (Reg*) malloc(sizeof(Reg)*size);
size = FILESIZE/sizeof(Reg);
fread (A, sizeof(Reg), size, FILEREAD);
Using a loop, consecutively call this call, to make me read my entire file.
What will happen when I get near the end of the file, and I can not read "size" * sizeof (Reg), or if you are only available to read half this amount, what will happen with my array A. It will be completed? The function will return an error?
Knowing how the file was read by the fread through?
Edi1: Exactly, if the division is not exact, when I read the last bit smaller file size that I'll read things that are not on file, I'm wondering with my vector resize to the amount of bytes that I can read, or develop a dynamic better.
fread returns the number of records it read. Anything beyond that in your buffer may be mangled, do not rely on that data.
fread returns the number of full items actually read, which may be
less than count if an error occurs or if the end of the file is
encountered before reaching count.
The function will not read past the end of the file : the most likely occurrence is that you will get a bunch of full buffers and then a (final) partial buffer read, unless the file size is an exact multiple of your buffer length.
Your logic needs to accommodate this - the file size gives you the expected total number of records so it should not be hard to ignore trailing data in the buffer (after the final fread call) that corresponds to uninitialized records. A 'records remaining to read' counter would be one approach.
fread() returns the number of elements it could read. So you have to check the return value of fread() to find out how many elements in your array are valid.
It will return a short item count or zero if either an error occurred or EOF has is reached. You will have to use feof() ond ferror() in this case to check what condition is met.
I want to read some data from the file, the data will have different sizes at different times.
If I use the below code, then:
char dataStr[256];
fread(dataStr, strlen(dataStr), 1, dFd);
fread is returning 0 for the above call and not reading any thing from the file.
But, if I give size as 1 then it successfully reads one char from the file.
What should be the value of size argument to the fread() function when we do not know how much is the size of the data in the file?
strlen counts the number of characters until it hits \0.
In this case you probably hit \0 on the very first character hence strlen returns 0 as the length and nothing is read.
You sould use sizeof instead of strlen.
You can't do that, obviously.
You can read until a known delimiter, often line feed, using fgets() to read a line. Or you can read a known-in-advance byte count, using that argument.
Of course, if there's an upper bound on the amount of data, you can read that always, and then somehow inspect the data to see what you got.
Also, in your example you're using strlen() on the argument that is going to be overwritten, that implies that it already contains a proper string of the exact same size as the data that is going to be read. This seems unlikely, you probably mean sizeof dataStr there.
You should use:
fread(dataStr, 1, sizeof dataStr, dFd);
to indicate that you want to read the number of bytes equal to the size of your array buffer.
The reason why your code doesn't work is that strlen() finds the length of a NULL-terminated string, not the size of the buffer. In your case, you run it on an uninitialized buffer and simply get lucky, your first byte in the buffer is NULL, so strlen(dataStr) returns 0, but is just as likely to crash or return some random number greater than your buffer size.
Also note that fread() returns the number of items read, not the number of characters (I swapped the second and the third arguments so that each character is equivalent to one item).
fread returns the number of successfully readed numblocks.
You can:
if( 1==fread(dataStr, 256, 1, dFd) )
puts("OK");
It reads ever the full length of your defined data; fread can't break on '\0'.