How to read a very large dataset from an HDF5 file? - c

Question

I've started learning HDF5 API very recently. Suppose that I'm going to read a very large vector (i.e. one-dimentional array) which is stored as a dataset in an HDF5 file. Its size N_SIZE is so large that malloc(N_SIZE) fails. So it seems to me that I have to read it chunk by chunk. Should I use H5Dread_chunk() here?

Answer 1

Yes, you will have to read the dataset chunk by chunk if it does not fit in main memory. Also, please note that your dataset must be created with a chunked storage layout. Then you can read one chunk at the time using hyperslabs (i.e. slices).
All these can be greatly simplified with HDFql. HDFql is a high-level language that alleviates you from the low-level details of handling HDF5 files.
As an example, you could do the following in C using HDFql:
// declare variables
char script[100];
int data[1024][1024];
int i;
// create a HDF5 file named 'my_file.h5' and use (i.e. open) it
hdfql_execute("CREATE AND USE FILE my_file.h5");
// create a three dimensional chunked dataset named 'my_dataset' (each chunk is 1 MB)
hdfql_execute("CREATE CHUNKED(1, 1024, 1024) DATASET my_dataset AS INT(100, 1024, 1024)");
// register variable 'data' for subsequent usage
hdfql_variable_register(data);
// loop 100 times (i.e. number of chunks that exists in dataset 'my_dataset')
for(i = 0; i < 100; i++)
{
// prepare script to read one chunk at the time using an hyperslab
sprintf(script, "SELECT FROM my_dataset(%d:::1) INTO MEMORY 0", i);
// execute script
hdfql_execute(script);
// call hypothetical function 'process' passing variable 'data' that contains the chunked data
process(data);
}
Additional examples on how to use HDFql in C can be found here.