How to unpack a msgpack file? - c

Question

I am writing msgpack-encoded data to a file. On writing, I am just using the fbuffer of the C API. As in (I striped all error handling for the example):
FILE *fp = fopen(filename, "ab");
msgpack_packer pk;
msgpack_packer_init(pk, fp, msgpack_fbuffer_write);
msgpack_pack_int(pk, 42);
// more data ...
How do I read this file back in? All the example I found assume that the data is in memory, however, my files are up to 5GB, it is not exactly a good idea to hold this in memory completely. Also I do not want to read in chunks myself. After all, I do not know how long the msgpack objects are, so chances are I end up with half an integer in my buffer.
Can msgpack's unpack somehow read from disk directly? Or is there some standard pattern to do this?

Answer 1

You might consider using "msgpack_unpacker" for that instead, which seems to be the official way that MessagePack implements a 'streaming' deserializer. Have a look at msgpack-c/example/c/lib_buffer_unpack.c
Regards, NiteHawk

Answer 2

Okay, I managed to do it.
Here is how to write:
#include <stdlib.h>
#include <msgpack.h>
#include <msgpack/fbuffer.h>
int main(int argc, char **argv) {
if(2 != argc) {
fprintf(stderr, "Call all writeFile <file>");
return;
}
FILE *fp = fopen(argv[1], "ab");
msgpack_packer pk;
msgpack_packer_init(&pk, fp, msgpack_fbuffer_write);
for(int i=0;i<2048;i++) {
msgpack_pack_int(&pk, i);
}
fclose(fp);
}
And this is what the read looks like:
#include <stdlib.h>
#include <msgpack.h>
static const int BUFFERSIZE = 2048;
int main(int argc, char **argv) {
if(2 != argc) {
fprintf(stderr, "Call with readFile <file>");
return 1;
}
char *inbuffer = (char *) malloc(BUFFERSIZE);
if(NULL == inbuffer) {
fprintf(stderr, "Out of memory!");
return 1;
}
FILE *fp = fopen(argv[1], "rb");
size_t off = 0;
size_t read = 0;
msgpack_unpacked unpacked;
msgpack_unpacked_init(&unpacked);
do {
read = fread(inbuffer, sizeof(char), BUFFERSIZE - off, fp);
off = 0;
while(msgpack_unpack_next(&unpacked, inbuffer, read, &off)) {
msgpack_object_print(stdout, unpacked.data);
puts("");
}
memcpy(inbuffer, &(inbuffer[off]), read-off);
off = read - off;
} while(read != 0);
free(inbuffer);
fclose(fp);
msgpack_unpacked_destroy(&unpacked);
return 0;
}
I did not try, but I think it will work with larger objects (arrays, maps etc.) as well.