Opening big files with fopen [duplicate] - c

Question

On a 32-bit system, what does ftell return if the current position indicator of a file opened in binary mode is past the 2GB point? In the C99 standard, is this undefined behavior since ftell must return a long int (maximum value being 2**31-1)?

Answer 1

on long int
long int is supposed to be AT LEAST 32-bits, but C99 standard does NOT limit it to 32-bit.
C99 standard does provide convenience types like int16_t & int32_t etc that map to correct bit sizes for a target platform.
on ftell/fseek
ftell() and fseek() are limited to 32 bits (including sign bit) on the vast majority of 32-bit architecture systems. So when there is large file support you run into this 2GB issue.
POSIX.1-2001 and SysV functions for fseek and ftell are fseeko and ftello because they use off_t as the parameter for the offset.
you do need to define compile with -D_FILE_OFFSET_BITS=64 or define it somewhere before including stdio.h to ensure that off_t is 64-bits.
Read about this at the cert.org secure coding guide.
On confusion about ftell and size of long int
C99 says long int must be at least 32-bits it does NOT say that it cannot be bigger
try the following on x86_64 architecture:
#include <stdio.h>
int main(int argc, char *argv[]) {
FILE *fp;
fp = fopen( "test.out", "w");
if ( !fp )
return -1;
fseek(fp, (1L << 34), SEEK_SET);
fprintf(fp, "\nhello world\n");
fclose(fp);
return 0;
}
Notice that 1L is just a long, this will produce a file that's 17GB and sticks a "\nhello world\n" to the end of it. Which you can verify is there by trivially using tail -n1 test.out or explicitly using:
dd if=test.out skip=$((1 << 25))
Note that dd typically uses block size of (1 << 9) so 34 - 9 = 25 will dump out '\nhello world\n'

Answer 2

At least on a 32bit OS ftell() it will overflow or error or simply run into Undefined Behaviour.
To get around this you might like to use off_t ftello(FILE *stream); and #define _FILE_OFFSET_BITS 64.
Verbatim from man ftello:
The fseeko() and ftello() functions are identical to fseek(3) and ftell(3) (see fseek(3)), respectively, except that the offset argument of fseeko() and the return value of ftello() is of type off_t instead of long.
On many architectures both off_t and long are 32-bit types, but compilation with
#define _FILE_OFFSET_BITS 64
will turn off_t into a 64-bit type.
Update:
According to IEEE Std 1003.1, 2013 Edition ftell() shall return -1 and set errno to EOVERFLOW in such cases:
EOVERFLOW
For ftell(), the current file offset cannot be represented correctly in an object of type long.

Answer 3

There is no 64b aware method in C99 standard. What OS/environment are you using? On windows, there is _ftelli64.
On other platforms, look at http://forums.codeguru.com/showthread.php?277234-Cannot-use-fopen()-open-file-larger-than-4-GB

Answer 4

This worked for me on Windows32/MinGW to play with a 6GB file
#define _FILE_OFFSET_BITS 64
#include<stdio.h>
int main() {
FILE *f = fopen("largefile.zip","rb");
fseeko64(f, 0, SEEK_END);
off64_t size = ftello64(f);
printf("%llu\n", size);
}
gcc readlargefile.c -c -std=C99 -o readlargefile.exe
Every single detail, the macro, the compiler option, matters.