How to access content of repeated message data - arrays

Question

I want to use protocol buffer to send and receive the following type using gRPC
std:array<std::complex<int>, 2> bar_array;
Sources used to get idea: 1, 2
What I have done so far
My approach (Intentionally I am omitting here the unnecessary code)
proto file
syntax = "proto3";
package expcmake;
message child_data {
repeated int32 child_data_list = 1;
}
message Address {
child_data child_data_var = 8;
repeated child_data parent_data_list = 9;
}
server.cpp
Here, at first I have made a dummy std:array<std::complex<int>, 2> data. Then, I have filled the child_data_list with the std::complex<int> data. After each filling of real and imaginary part I have pushed them in the parent_data_list. Also, at this moment I have cleared the child_data_list.
Client message name is NameQuery, while Server message name is Address
In Server side both message are passed as pointer
class AddressBookService final : public expcmake::AddressBook::Service {
public:
virtual ::grpc::Status GetAddress(::grpc::ServerContext* context, const ::expcmake::NameQuery* request, ::expcmake::Address* response)
{
// omitting unnecessary lines
// populate bar_array with std::complex<int> data
std::complex<int> z4 = 1. + 2i, z5 = 1. - 2i; // conjugates
bar_array = {z4, z5};
std::cout << "bar array printing whose size: " << bar_array.size() << std::endl;
for(int i = 0; i < bar_array.size(); i++) {
std::cout << bar_array[i] << " ";
}
std::cout << std::endl;
// Use parent_data_list protocol buffer message type to fill with the content of bar_array
for (int i = 0; i < bar_array.size(); i++){
// Use child_data protocol buffer message type to fill with the content of complex int data
response->mutable_child_data_var()->add_child_data_list(real(bar_array[i]));
response->mutable_child_data_var()->add_child_data_list(imag(bar_array[i]));
// Use parent_data_list protocol buffer message type to fill with the content of child_data -> child_data_list data
response->add_parent_data_list() -> child_data_list();
// clear the child_data message. Reason to keep child_data_list new in every iteration otherwise add_child_data_list will append new data (eg: 1,2,1,-2) which is wrong. Expected is (1,2) then (1,-2)
response->mutable_child_data_var()->clear_child_data_list();
}
// This is zero which I have got. Without clearing it is 4 which is also correct I believe as per the concept of protocol buffer message type
std::cout << "response->mutable_child_data_var()->child_data_list_size(): " << response->mutable_child_data_var()->child_data_list_size() << std::endl;
// This is 2 which meets my requirement
std::cout << "response->parent_data_list_size(): " << response->parent_data_list_size() << std::endl;
// omitting unnecessary lines
}
};
client.cpp
int main(int argc, char* argv[])
{
// Setup request
expcmake::NameQuery query;
expcmake::Address result;
// printing the content of child_data -> child_data_list data array/container. There I have seen 1,2,1,-2 if I don't do the clear operation on child_data_list in server side. So, I guess it is correctly got the data
for(int i = 0; i < result.mutable_child_data_var()->child_data_list_size(); i++)
std::cout << "Child Data at index [" << i << "]: " << result.mutable_child_data_var()->child_data_list(i) << std::endl;
// This one making problem
// printing the content of parent_data_list type/container
// for(int i = 0; i < result.parent_data_list_size(); i++){
// std::cout << "Parent Data at index [" << i << "]: " << result.mutable_parent_data_list(i) << std::endl; // This give me the memory address
// Tried others to fetch the data but failed. Eg: result.parent_data_list(i) // failed
//
// }
}
Snippet from the generated pb file
// repeated int32 child_data_list = 1;
int child_data_list_size() const;
private:
int _internal_child_data_list_size() const;
public:
void clear_child_data_list();
private:
::PROTOBUF_NAMESPACE_ID::int32 _internal_child_data_list(int index) const;
const ::PROTOBUF_NAMESPACE_ID::RepeatedField< ::PROTOBUF_NAMESPACE_ID::int32 >&
_internal_child_data_list() const;
void _internal_add_child_data_list(::PROTOBUF_NAMESPACE_ID::int32 value);
::PROTOBUF_NAMESPACE_ID::RepeatedField< ::PROTOBUF_NAMESPACE_ID::int32 >*
_internal_mutable_child_data_list();
public:
::PROTOBUF_NAMESPACE_ID::int32 child_data_list(int index) const;
void set_child_data_list(int index, ::PROTOBUF_NAMESPACE_ID::int32 value);
void add_child_data_list(::PROTOBUF_NAMESPACE_ID::int32 value);
const ::PROTOBUF_NAMESPACE_ID::RepeatedField< ::PROTOBUF_NAMESPACE_ID::int32 >&
child_data_list() const;
::PROTOBUF_NAMESPACE_ID::RepeatedField< ::PROTOBUF_NAMESPACE_ID::int32 >*
mutable_child_data_list();
// .expcmake.child_data child_data_var = 8;
bool has_child_data_var() const;
private:
bool _internal_has_child_data_var() const;
public:
void clear_child_data_var();
const ::expcmake::child_data& child_data_var() const;
PROTOBUF_MUST_USE_RESULT ::expcmake::child_data* release_child_data_var();
::expcmake::child_data* mutable_child_data_var();
void set_allocated_child_data_var(::expcmake::child_data* child_data_var);
private:
const ::expcmake::child_data& _internal_child_data_var() const;
::expcmake::child_data* _internal_mutable_child_data_var();
public:
void unsafe_arena_set_allocated_child_data_var(
::expcmake::child_data* child_data_var);
::expcmake::child_data* unsafe_arena_release_child_data_var();
// repeated .expcmake.child_data parent_data_list = 9;
int parent_data_list_size() const;
private:
int _internal_parent_data_list_size() const;
public:
void clear_parent_data_list();
::expcmake::child_data* mutable_parent_data_list(int index);
::PROTOBUF_NAMESPACE_ID::RepeatedPtrField< ::expcmake::child_data >*
mutable_parent_data_list();
private:
const ::expcmake::child_data& _internal_parent_data_list(int index) const;
::expcmake::child_data* _internal_add_parent_data_list();
public:
const ::expcmake::child_data& parent_data_list(int index) const;
::expcmake::child_data* add_parent_data_list();
const ::PROTOBUF_NAMESPACE_ID::RepeatedPtrField< ::expcmake::child_data >&
parent_data_list() const;
I guess
Is the filling of the message field is incorrectly done !! Though the size is not saying that
I am not catching the protobuf syntax(which is generated in the pb file) in right way to fetch the data
Need suggestions(helpful if can provide the syntax too).

Answer 1

The mutable_* prefixed APIs are for mutating (modifying/adding) elements and they return a pointer. The mutable_* and set_* APIs should only be used while filling the data. Once data is filled, you should not be using mutable_* APIs to check sizes. After receiving the response from the server, the client will most probably be consuming it, not mutating. You need to update the client-side accordingly.
Also, you can use a range-based for loop to iterate over child_data_list of child_data_var like this:
const auto child_data_list = result.child_data_var().child_data_list();
for (const auto element : child_data_list)
{
// process element
}
Apart from that, you're using std::array (a fixed-size array) i.e. std:array<std::complex<int>, 2>, alternatively here's another way to model this without repeated which does not represent a fixed-size array.
complex.proto
syntax = "proto3";
package pb;
message Complex {
int32 real = 1;
int32 imag = 2;
}
message Compound {
Complex c1 = 1;
Complex c2 = 2;
}
The message Complex emulates an std::complex type and Compound an std::array with only 2 elements c1 and c2.
With this, you can easily convert the protobuf message to std::complex and vice versa. Once converted to std::complex, you can perform its supported operations as needed.
Compile complex.proto:
protoc --cpp_out=. ./complex.proto
For the C++ API, look for the accessors in the generated complex.pb.h file under public. The official doc Protocol Buffer Basics: C++ also lists the C++ API examples under The Protocol Buffer API section.
Here's a complete C++ example with the above complex.proto:
main.cpp
#include <iostream>
#include <complex>
#include <array>
#include <cstdlib>
#include "complex.pb.h"
namespace my {
using complex = std::complex<int>;
using compound = std::array<std::complex<int>, 2>;
}
int main()
{
const auto my_c1 = my::complex{ 1, 2 };
const auto my_c2 = my::complex{ 2, 4 };
const auto my_compound_1 = my::compound{ my_c1, my_c2 };
std::cout << "my_compound_1 [size: " << my_compound_1.size() << "]\n";
std::cout << "my_c1: " << my_c1 << '\n';
std::cout << "my_c2: " << my_c2 << '\n';
pb::Compound pb_compound;
pb_compound.mutable_c1()->set_real(my_compound_1[0].real());
pb_compound.mutable_c1()->set_imag(my_compound_1[0].imag());
pb_compound.mutable_c2()->set_real(my_compound_1[1].real());
pb_compound.mutable_c2()->set_imag(my_compound_1[1].imag());
std::cout << "\npb_compound:\n";
pb_compound.PrintDebugString();
const auto serialized_compound = pb_compound.SerializeAsString();
// send
// receive
pb::Compound deserialized_compound;
if (!deserialized_compound.ParseFromString(serialized_compound))
{
std::cerr << "[ERROR] Parsing failed!\n";
return EXIT_FAILURE;
}
std::cout << "\n\npb_compound (deserialized):\n";
deserialized_compound.PrintDebugString();
const auto pb_c1 = deserialized_compound.c1();
const auto pb_c2 = deserialized_compound.c2();
const auto my_c3 = my::complex{ pb_c1.real(), pb_c1.imag() };
const auto my_c4 = my::complex{ pb_c2.real(), pb_c2.imag() };
const auto my_compound_2 = my::compound{ my_c3, my_c4 };
std::cout << "my_compound_2 [size: " << my_compound_2.size() << "]\n";
std::cout << "my_c3: " << my_c3 << '\n';
std::cout << "my_c4: " << my_c4 << '\n';
const auto sum = my_c3 + my_c4;
std::cout << "sum: " << sum << '\n';
return EXIT_SUCCESS;
}
Compile with g++:
g++ main.cpp complex.pb.cc -o pb_complex `pkg-config --cflags --libs protobuf`
Run:
$ ./pb_complex
my_compound_1 [size: 2]
my_c1: (1,2)
my_c2: (2,4)
pb_compound:
c1 {
real: 1
imag: 2
}
c2 {
real: 2
imag: 4
}
pb_compound (deserialized):
c1 {
real: 1
imag: 2
}
c2 {
real: 2
imag: 4
}
my_compound_2 [size: 2]
my_c3: (1,2)
my_c4: (2,4)
sum: (3,6)

Answer 2

The way I was accessing the filed parent_data_list was wrong. I am posting the approach which has solved the issue.
A little change in proto file
syntax = "proto3";
package expcmake;
message child_data {
repeated int32 child_data_list = 1 [packed=true];
}
message Address {
repeated child_data parent_data_list = 1;
}
Reason to use packed see this
server.cpp
class AddressBookService final : public expcmake::AddressBook::Service {
public:
virtual ::grpc::Status GetAddress(::grpc::ServerContext* context, const ::expcmake::NameQuery* request, ::expcmake::Address* response){
// omitting unnecessary lines
// populate bar_array with std::complex<int> data
std::complex<int> z4 = 1. + 2i, z5 = 1. - 2i; // conjugates
bar_array = {z4, z5};
// Now goal is to pass this bar_array as a protobuf message
// Use parent_data_list protocol buffer message type to fill with the content of bar_array
for (int i = 0; i < bar_array.size(); i++){
// make a 2D array. Eg: std::array<std::complex<int>>. In each iteration, new sub_content will be added (here sub_content means child_data_list)
response->add_parent_data_list() -> child_data_list();
// Followings are filling the child_data_list with the required data(Real and Imag part of the complex data which is already generated)
response->mutable_parent_data_list(i)->add_child_data_list(real(bar_array[i]));
response->mutable_parent_data_list(i)->add_child_data_list(imag(bar_array[i]));
}
}
return grpc::Status::OK;
};
After the completion of the iteration, A 2D type of message will be generated which is parent_data_list = [[child_data_list], [child_data_list]]. Don't consider it as a real scenario. Just for presentation I am using it.
client.cpp
std::cout <<"Using Mutable" << std::endl;
for(int i = 0; i < result.parent_data_list_size(); i++){
for(int j = 0 ; j<result.mutable_parent_data_list(i)->child_data_list_size(); j++){
std::cout << "Value [" << i << "][" << j << "]: " << result.mutable_parent_data_list(i)->child_data_list(j) << std::endl;
}
}
// Following is same as before. Added as I was in turmoil to understand the properties generated by protobuf. Hope could be helpful for beginners like me
std::cout <<"Without Mutable property" << std::endl;
for(int i = 0; i < result.parent_data_list_size(); i++){
for(int j = 0 ; j < result.parent_data_list(i).child_data_list_size(); j++){
std::cout << "Value [" << i << "][" << j << "]: " << result.parent_data_list(i).child_data_list(j) << std::endl;
}
}
In client side I am accessing the child_data_list with index which is not properly mimic the C++ style. Would be great if I can in one line print the value of parent_data_list[n] data. As far I have studied the generated pb.h file my understanding is that it is not possible as no return function I have not found without index for the repeated field. Or may be I am missing something important again.