Opening and Reading from a "large" gzip compressed file in C - c

Question

I have been trying to open and read a gzip compressed file using gzip-based file IO functions in C. The compressed file that I have with me is quite large of size 12 GB. The uncompressed file was ~260 GB and hence I am not prepared to uncompress the file using gunzip and go ahead from there.
I am specifically using the below code to read and write into the buffers available to us-
#define windowBits 15
#define ENABLE_ZLIB_GZIP 32
#define CHUNK 0x4000
#define CALL_ZLIB(x) { \
int status; \
status = x; \
if (status < 0) \
{ \
fprintf(stderr, "%s:%d: %s returned a bad status of %d.\n", __FILE__, __LINE__, #x, status); \
exit(EXIT_FAILURE);\
} \
} \
int main ()
{
const char * file_name = "test.gz";
FILE * file;
z_stream strm = {0};
unsigned char in[CHUNK];
unsigned char out[CHUNK];
strm.zalloc = Z_NULL;
strm.zfree = Z_NULL;
strm.opaque = Z_NULL;
strm.next_in = in;
strm.avail_in = 0;
CALL_ZLIB (inflateInit2 (& strm, windowBits | ENABLE_ZLIB_GZIP));
/* Open the file. */
file = fopen (file_name, "rb");
while (1) {
int bytes_read;
bytes_read = fread (in, sizeof (char), sizeof (in), file);
strm.avail_in = bytes_read;
do {
unsigned have;
strm.avail_out = CHUNK;
strm.next_out = out;
CALL_ZLIB (inflate (& strm, Z_NO_FLUSH));
have = CHUNK - strm.avail_out;
fwrite (out, sizeof (unsigned char), have, stdout);
}
while (strm.avail_out == 0);
if (feof (file)) {
inflateEnd (& strm);
break;
}
}
return 0;
}
The code accurately reads and writes from the zlib file based on a buffer that you specify initially. The buffer size is being fixed to a certain value (in the above case to 0x4000).
The problem now is that I cannot increase the size of this buffer beyond a certain value ( I can use 3276008 as buffer size, but not 32760008 ). To read a 12 GB compressed value, would need me to use a very big buffer. As specified in my edits, this looks like some kind of a DATA_ERROR not a BUFFERerror... so it is not a buffer error after all!
Is there any way how I can be able to record the whole 12 GB compressed file using the zlib functions above ?
EDIT #1
The error code returned by the function inflate is encapsulated by the CALL_ZLIB function which I am sorry to have not included. So I get the below error code when I run with the buffer size of 0x4000. I have added the CALL_ZLIB function to the code for your reference also.
Error msg :
parser.c:96: inflate(&strm, Z_NO_FLUSH) returned a bad status of -3. This obviously looks like a **DATA_ERROR.
EDIT #2
I have tried adding a negative value of windowBits to InflateInit2() but that did not solve any of my problems. The inflate() function initially reads my file correctly -- displaying all of my data the way I want it to..
0x55b0 [0x40]: event: 3
.
. ... raw event: size 64 bytes
. 0000: 03 00 00 00 00 00 40 00 18 03 00 00 18 03 00 00 ......#.........
. 0010: 4d 6f 64 65 6d 4d 61 6e 61 67 65 72 00 00 00 00 ModemManager....
. 0020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
. 0030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0 0 0x55b0 [0x40]: PERF_RECORD_COMM: ModemManager:792/792
0x55f0 [0x40]: event: 7
.
. ... raw event: size 64 bytes
. 0000: 07 00 00 00 00 00 40 00 19 03 00 00 01 00 00 00 ......#.........
. 0010: 19 03 00 00 01 00 00 00 00 00 00 00 00 00 00 00 ................
. 0020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
. 0030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0 0 0x55f0 [0x40]: PERF_RECORD_FORK(793:793):(1:1)
0x5630 [0x40]: event: 3
.
But after some time, the displayed output becomes garbled and I cannot read from it anymore..
0x4d68 [0x38]: ........... 001 0..
0 0 00 00 00 0 00 000 00 ze 64s
. 0000: 07 00 00 00 00 00 40 00 19 03 00 00 01 00 00 00 .. 00 0 event: size 64 bytes
. 0000: 03 00 00 00 si sisizsiz4s
. 0000: 07 00 00 00 00 00 40 00 19 0....
. 0030: 00 00 00 00 00 00 00 00 00 00 00 00 ..#.#. 0010: 19 03 00 00 [0x38]: ........... 001 0..
0 0 00 00 00 0 00 000 00 ze 64s
. 0000: 07 00 00 00 00 00 40 00 100 00 00 00 00 ..............0 0 0x4d28 [0x40]: PERF_RECORD_FORK(135:135):(2:62)
0x4d68 [0x38]: ........... 001 0..
0 0 00 00 00 0 00 000 00 00 00 00: PERORD_FORK(135:135):(2:2)
This finally terminates with the error message I described in the Edit #1

Answer 1

I have resolved the problem.
The basic problem was that I was not initializing the strm.next_in member of z_stream in my code inside the loop. Hence after doing 1 iteration, the buffer got corrupted and I was getting the above errors.
I modified my code to --
strm.next_in = in;
strm.avail_in = 0;
CALL_ZLIB(inflateInit2 (&strm, windowBits | ENABLE_ZLIB_GZIP));
file = fopen(filename, "rb");
while(1)
{
int bytes_read;
strm.next_in = in; // added this line
bytes_read = fread(in, sizeof(char), sizeof(in), file);
strm.avail_in = bytes_read;
do
{
unsigned have;
strm.avail_out = CHUNK;
strm.next_out = out;