knowing end of message in protocol with msgpack - c

Question

How do you manage end of a message in a protocol ? I use msgpack-c and the only solution I found is to send the header before the payload (separately).
Send the header to client :
// header
{
"message_type": "hello",
"payload_size": 10
}
The client received the header, unpack it, and allocate a buffer of "payload_size", receive data from stream, and if the buffer is complete the message is finish.
I want to send header and body succinctly
{
"header": { "message_type":"hello", "payload_size": 10},
"payload": {...} // can come in multiple frame
}
My problem is that I don't know if it's possible to partially unpack the header for knowing the size before receiving the full message (splitted if > 4096kb due to libevent restriction)
How would you do that ? I am open to all solutions.

Answer 1

C++
Using unpack() function
You can use offset parameter of unpack() function.
See https://github.com/msgpack/msgpack-c/wiki/v2_0_cpp_unpacker#client-controls-a-buffer
Here is a code example:
#include <iostream>
#incluee <msgpack.hpp>
int main() {
msgpack::sbuffer buf;
msgpack::pack(buf, std::make_tuple("first message", 123, 56.78));
msgpack::pack(buf, std::make_tuple("second message", 42));
std::size_t off = 0; // cursor of buf
{
// unpack() function starts parse from off (0)
auto oh = msgpack::unpack(buf.data(), buf.size(), off);
// off is updated to 25. 25 is MessagePack formatted byte size
// of ["first message",123,56.78]
// (I use JSON notation but actual format is MessagePack)
std::cout << "off:" << off << std::endl;
std::cout << *oh << std::endl;
}
{
// unpack() function starts parse from off (25)
auto oh = msgpack::unpack(buf.data(), buf.size(), off);
// off is updated to 42.
// 42 - 25 = 17. 17 is MessagePack formatted byte size
// of ["second message",42]
// (I use JSON notation but actual format is MessagePack)
std::cout << "off:" << off << std::endl;
std::cout << *oh << std::endl;
}
}
Output is
off:25
["first message",123,56.78]
off:42
["second message",42]
msgpack-c unpack() manage the position of buffer internally.
You don't need to pass payload_size.
In addition you can mix non-msgpack format data in the buffer.
+--------------------+-----------------------------+--------------------+
| MessagePackBytes1 | Any format user knows size | MessagePackBytes2 |
+--------------------+-----------------------------+--------------------+
Let's say user knows the data structure that contains MessgePackBytes1(unknown size), any format data (known size), and MessgePackBytes1(unknown size).
#include <iostream>
#incluee <msgpack.hpp>
int main() {
msgpack::sbuffer buf;
msgpack::pack(buf, std::make_tuple("first message", 123, 56.78));
std::string non_mp = "non mp format data";
buf.write(non_mp.data(), non_mp.size());
msgpack::pack(buf, std::make_tuple("second message", 42));
std::size_t off = 0; // cursor of buf
{
auto oh = msgpack::unpack(buf.data(), buf.size(), off);
std::cout << "off:" << off << std::endl;
std::cout << *oh << std::endl;
}
{
std::string extracted{buf.data() + off, non_mp.size()};
std::cout << extracted << std::endl;
off += non_mp.size();
}
{
auto oh = msgpack::unpack(buf.data(), buf.size(), off);
std::cout << "off:" << off << std::endl;
std::cout << *oh << std::endl;
}
}
Output is
off:25
["first message",123,56.78]
non mp format data
off:60
["second message",42]
Using unpacker
It is a little advanced but it might fit streaming usecases.
https://github.com/msgpack/msgpack-c/wiki/v2_0_cpp_unpacker#msgpack-controls-a-buffer
Here is an example that unpacking MessagePack from continuous and scattered receiving message.
https://github.com/msgpack/msgpack-c/blob/700167995927f0348fb08ae2579440c1bc135480/example/boost/asio_send_recv.cpp#L41-L64
C
C version is basically similar to C++.
Using unpack() function
C version has the similar unpack function.
Here is the prototype:
msgpack_unpack_return
msgpack_unpack_next(msgpack_unpacked* result,
const char* data, size_t len, size_t* off);
You can pass off as offset similar to C++ version. C doesn't have reference so you need to pass the address of off using &off.
See https://github.com/msgpack/msgpack-c/wiki/v2_0_c_overview#using-unpack-function
If you want to know individual variable length field size such as stirng, you can access size member variable of unpacked object.
For example:
typedef struct {
uint32_t size;
struct msgpack_object* ptr;
} msgpack_object_array;
typedef struct {
uint32_t size;
const char* ptr;
} msgpack_object_str;
typedef struct {
uint32_t size;
const char* ptr;
} msgpack_object_bin;
typedef struct {
int8_t type;
uint32_t size;
const char* ptr;
} msgpack_object_ext;
See https://github.com/msgpack/msgpack-c/wiki/v2_0_c_overview#object
Using unpacker
See https://github.com/msgpack/msgpack-c/wiki/v2_0_c_overview#using-unpacker