I'm simply trying to add one message to another message (up to 60 times times)
My .proto file looks as follows;
syntax = "proto3";
message FeatureFile {
string fileName= 2;
string Id= 3;
repeated Feature features = 1;
}
message Feature {
int32 version = 1;
int32 epochTime = 2;
int32 noOfObs= 3;
int32 frequency = 4;
}
I have tried to make a callback function to add repeated data, but cannot make it work.
bool encode_string(pb_ostream_t* stream, const pb_field_t* field, void* const* arg)
{
const char* str = (const char*)(*arg);
if (!pb_encode_tag_for_field(stream, field))
return false;
return pb_encode_string(stream, (uint8_t*)str, strlen(str));
}
bool encode_repeatedMsg(pb_ostream_t* stream, const pb_field_t* field, void* const* arg)
{
const char* obj = (const char*)(*arg);
int i;
for (i = 0; i < 60; i++)
{
if (!pb_encode_tag_for_field(stream, field))
return false;
if (!pb_encode_submessage(stream, Feature_fields, *arg))
return false;
}
return true;
}
int main()
{
FeatureFile featurefile = FeatureFile_init_zero;
Feature feature = Feature_init_zero;
featurefile.fileName.arg = "092536.csv";
featurefile.fileName.funcs.encode = &encode_string;
featurefile.Id.arg = "";
featurefile.Id.funcs.encode = &encode_string;
feature.version = 1;
feature.epochTime = 12566232;
feature.noOfObs = 260;
feature.frequency = 200;
featurefile.features.funcs.encode = &encode_repeatedMsg;
I thought I could call the repeated encoding like the last line of code shows, but I doesn't allow me.
The callback itself is supposed to add 60 of the same messages (feature) to the the featurefile.
Can anyone help me here?
I myself have never used the callbacks in nanopb. I do have been using the .options file to statically allocate the desired array size. Your case this might be a bit much as your require 60 messages but this is how you do it:
You create a file with the same name as your .proto file but give it the extension .options. You place it in the same folder as your .proto file. In that file you mention there repeated variable name and assign it a size:
# XXXX.options
FeatureFile.features max_count:16
More information on the nanopb options can be found here.
Related
I'm new to HTTP and MFC. I suppose to upload a binary file to a given URL, by sending PUT request. I'm sure the requesting URL and the header are well formed and the file works well, because I tried them by Postman then get a SUCCESS returned.
But when I try to do this by the following MFC code, it always returns a 403 error. Please help me.
bool UploadFile(LPCTSTR strUrl, LPCTSTR filenameFullpath, CString header)
{
BOOL bResult = FALSE;
DWORD dwType = 0;
CString strServer;
CString strObject;
INTERNET_PORT wPort = 0;
DWORD dwFileLength = 0;
BYTE * pFileBuff = NULL;
CHttpConnection * pHC = NULL;
CHttpFile * pHF = NULL;
CInternetSession cis;
bResult = AfxParseURL(strUrl, dwType, strServer, strObject, wPort);
if (!bResult)
return FALSE;
CFile file;
try
{
// Read the file
if (!file.Open(filenameFullpath, CFile::shareDenyNone | CFile::modeRead))
return FALSE;
dwFileLength = file.GetLength();
if (dwFileLength <= 0)
return FALSE;
pFileBuff = new BYTE[dwFileLength];
memset(pFileBuff, 0, sizeof(BYTE) * dwFileLength);
file.Read(pFileBuff, dwFileLength);
// Set up internet connection
const int nTimeOut = 5000;
cis.SetOption(INTERNET_OPTION_CONNECT_TIMEOUT, nTimeOut);
cis.SetOption(INTERNET_OPTION_CONNECT_RETRIES, 2); // Retry once if failed
pHC = cis.GetHttpConnection(strServer, dwType == AFX_INET_SERVICE_HTTP ? NORMAL_CONNECT : SECURE_CONNECT, wPort); // Get a HTTP connection
pHF = pHC->OpenRequest(CHttpConnection::HTTP_VERB_PUT, strObject);//strObject
pHF->AddRequestHeaders(header , HTTP_ADDREQ_FLAG_ADD_IF_NEW);
// I set the header separately instead of passing it as the first parameter of the next call.
// If I don't do so, another error occurs
if (!pHF->SendRequest(NULL, 0, pFileBuff, dwFileLength))
{
delete[]pFileBuff;
pFileBuff = NULL;
pHF->Close();
pHC->Close();
cis.Close();
return FALSE;
}
DWORD dwStateCode = 0;
pHF->QueryInfoStatusCode(dwStateCode);
if (dwStateCode == HTTP_STATUS_OK)
bResult = TRUE;
}
catch (CInternetException * pEx)
{
char sz[256] = "";
pEx->GetErrorMessage(sz, 25);
CString str;
str.Format("InternetException occur!\r\n%s", sz);
AfxMessageBox(str);
}
delete[]pFileBuff;
pFileBuff = NULL;
file.Close();
pHF->Close();
pHC->Close();
cis.Close();
return bResult;
}
And the calling of this function is just something like this:
CHttpClient hc;
hc.UploadFile(csUrl, "E:\\blah blah blah\\blah.tma", csUploadHeader);
while csUrl and csUploadHeader are well formed CString;
I got it! I should give pHF the header items (the key-value pairs) one by one, by calling the AddRequestHeaders() method several times, instead of wrapping and passing them together to pHF.
Then the following codes works very well:
CString sHeader1, sHeader2, sHeader3;
sHeader1.Format(_T("%s : %s"), sKey1, sValue1); // sKey_, sValue_ are strings
sHeader2.Format(_T("%s : %s"), sKey2, sValue2);
sHeader3.Format(_T("%s : %s"), sKey3, sValue3);
pHF->AddRequestHeaders(sHeader1, HTTP_ADDREQ_FLAG_ADD_IF_NEW); // Keep the second paeramter as these
pHF->AddRequestHeaders(sHeader2, HTTP_ADDREQ_FLAG_COALESCE);
pHF->AddRequestHeaders(sHeader3, HTTP_ADDREQ_FLAG_COALESCE);
And a very important notification: the string we pass to AddRequestHeaders() should not have the quotation("") around either the key or the value. In other words, string like "MyKey":"MyValue" have to be modified as MyKey:MyValue before giving to AddRequestHeaders();
*Edit: updated *
My message is defined as:
message Repeat {
int32 inum = 1;
float fnum = 2;
}
message NotSimpleMessage {
repeated Repeat repeat = 1;
}
I'm trying to write a decoder and encoder using the callback option. I think my encoding works fine, but my decoder fails.
My code is:
definitions:
typedef struct{
Repeat rep[MAX_NUMBERS];
int32_t numbers_count;
}Messer;
typedef struct{
Mess mess[MAX_NUMBERS];
int32_t numbers_count;
}MessList;
void mess_add_number(MessList * list, int32_t inum, float fnum)
{
if (list->numbers_count < MAX_NUMBERS)
{
(list->mess[list->numbers_count]).inumber = inum;
(list->mess[list->numbers_count]).fnumber = fnum;
list->numbers_count++;
}
}
void messer_add_number(Messer * list, int32_t inum, float fnum)
{
if (list->numbers_count < MAX_NUMBERS)
{
(list->rep[list->numbers_count]).inum = inum;
(list->rep[list->numbers_count]).fnum = fnum;
(list->rep[list->numbers_count]).has_inum = true;
(list->rep[list->numbers_count]).has_fnum = true;
list->numbers_count++;
}
}
encoder/decoder functions:
bool NestedMessage_encode_numbers(pb_ostream_t *ostream, const pb_field_t *field, void * const *arg)
{
Messer * source = (Messer*)(*arg);
int i;
// encode all numbers
for ( i = 0; i < source->numbers_count; i++)
{
if (!pb_encode_tag_for_field(ostream, field))
{
const char * error = PB_GET_ERROR(ostream);
printf("SimpleMessage_encode_numbers error: %s\n", error);
return false;
}
if (!pb_encode_submessage(ostream, Repeat_fields, &(source->rep[i])))
{
const char * error = PB_GET_ERROR(ostream);
printf("SimpleMessage_encode_numbers error: %s\n", error);
return false;
}
}
return true;
c}
bool NestedMessage_decode_numbers(pb_istream_t *istream, const pb_field_t *field, void **arg)
{
MessList * dest = (MessList*)(*arg);
Repeat rep;
// decode single number
Mess decmess;
printf("decoding...\n");
if (!pb_decode(istream, Repeat_fields ,&rep))
{
const char * error = PB_GET_ERROR(istream);
printf("decode error: %s\n", error);
return false;
}
// add to destination list
mess_add_number(dest, rep.inum, rep.fnum);
return true;
}
and the main is:
int main(void) {
uint8_t buffer[128];
size_t total_bytes_encoded = 0;
// encoding
// prepare the actual "variable" array
Messer actualData = { 0 };
messer_add_number(&actualData, 123, 1.2);
messer_add_number(&actualData, 456, 2.3);
messer_add_number(&actualData, 789, 3.4);
printf("Size: %d\n",actualData.numbers_count);
printf("data to be encoded: %d - %f, %d-%f, %d-%f\n",actualData.rep[0].inum,actualData.rep[0].fnum,
actualData.rep[1].inum, actualData.rep[1].fnum,
actualData.rep[2].inum,actualData.rep[2].fnum);
// prepare the nanopb ENCODING callback
NotSimpleMessage msg = NotSimpleMessage_init_zero;
msg.repeat.arg = &actualData;
msg.repeat.funcs.encode = NestedMessage_encode_numbers;
// call nanopb
pb_ostream_t ostream = pb_ostream_from_buffer(buffer, sizeof(buffer));
if (!pb_encode(&ostream, NotSimpleMessage_fields, &msg))
{
const char * error = PB_GET_ERROR(&ostream);
printf("pb_encode error: %s\n", error);
return EXIT_FAILURE;
}
total_bytes_encoded = ostream.bytes_written;
printf("Encoded size: %d\n", total_bytes_encoded);
// decoding
// empty array for decoding
Messer decodedData = { 0 };
// prepare the nanopb DECODING callback
NotSimpleMessage msgdec = NotSimpleMessage_init_zero;
msgdec.repeat.arg = &decodedData;
msgdec.repeat.funcs.decode = NestedMessage_decode_numbers;
// call nanopb
pb_istream_t istream = pb_istream_from_buffer(buffer, total_bytes_encoded);
if (!pb_decode(&istream, NotSimpleMessage_fields, &msgdec))
{
const char * error = PB_GET_ERROR(&istream);
printf("pb_decode error: %s", error);
return EXIT_FAILURE;
}
printf("Bytes decoded: %d\n", total_bytes_encoded - istream.bytes_left);
printf("decoded data: %d - %f, %d-%f, %d-%f\n",decodedData.rep[0].inum,decodedData.rep[0].fnum,
decodedData.rep[1].inum, decodedData.rep[1].fnum,
decodedData.rep[2].inum,decodedData.rep[2].fnum);
}
the output I get is:
Size: 3 data to be encoded: 123 - 1.200000, 456-2.300000, 789-3.400000
Encoded size: 29 Bytes decoded: 1 decoded data: 0 - 0.000000,
0-0.000000, 0-0.000000
print of the encoded buffer:
0a07087b15ffffff9affffff99ffffff993f0a0808ffffffc80315333313400a0808ffffff950615ffffff9affffff995940
I've tried some different structs inside the decoder but it just doesn't work.
pretty sure it some dumb small thing I'm missing, but I'm clueless about it.
Ah, there is a small gotcha in encoding/decoding submessages in callbacks.
When decoding, pb_decode() works fine because the submessage tag and length has already been parsed by nanopb. However, when encoding, the length of the message needs to be calculated and encoded separately. So instead of pb_encode(), you need to use pb_encode_submessage() here:
if (!pb_encode_submessage(ostream, Repeat_fields, &(source->rep[i])))
{
const char * error = PB_GET_ERROR(ostream);
printf("SimpleMessage_encode_numbers error: %s\n", error);
return false;
}
(For reference, here is a relevant part of an example.)
--
Regarding your update, this hex text:
0a07087b15ffffff9affffff99ffffff993f0a0808ffffffc80315333313400a0808ffffff950615ffffff9affffff995940
is somewhat corrupted, because your printing function seems to print "ffffff9a" instead of just "9a". Probably a signed to unsigned cast behaving unexpectedly. But that can be fixed with a simple search & replace, which gives:
0a07087b159a99993f0a0808c80315333313400a08089506159a995940
Decoding this with protoc:
echo 0a07087b159a99993f0a0808c80315333313400a08089506159a995940 | xxd -r -p | protoc --decode=NotSimpleMessage test.proto
Gives:
repeat {
inum: 123
fnum: 1.2
}
repeat {
inum: 456
fnum: 2.3
}
repeat {
inum: 789
fnum: 3.4
}
So seems your encoding is now working correctly.
Not sure what is causing the decode end so early (only 1 byte read) without error message. Maybe try stepping through it with a debugger and see what is going on. One reason might be if the data in the buffer would somehow get corrupted before the decode call, but can't see why that would happen.
In these definitions what structure is for Repeat and Mess? It´s important to try to compile.
I have a proto message defined as:
message SimpleMessage {
repeated int32 number = 1;}
now, after compiling, the field is of pb_callback_t and I suppose to write that function. (without .options file)
now, where and what should the function contain? where does the data itself being stored and how can I access it/ assign new data to it?
* EDIT *
according to #Groo 's answer, this is the code I tried:
typedef struct {
int numbers_decoded;
} DecodingState;
bool read_single_number(pb_istream_t *istream, const pb_field_t *field, void **arg)
{
// get the pointer to the custom state
DecodingState *state = (DecodingState*)(*arg);
int32_t value;
if (!pb_decode_varint32(istream, &value))
{
const char * error = PB_GET_ERROR(istream);
printf("Protobuf error: %s", error);
return false;
}
printf("Decoded successfully: %d", value);
state->numbers_decoded++;
return true;
}
int main(void) {
int32_t arr[3] = {10, 22, 342};
uint8_t buffer[128];
size_t message_length;
bool status;
SimpleMessage simple = SimpleMessage_init_zero;
printf("\nbefore : arr[0] = %d\n",arr[0]);
// set the argument and the callback fn
simple.number.arg = &arr;
simple.number.funcs.decode = read_single_number;
pb_ostream_t ostream = pb_ostream_from_buffer(buffer, sizeof(buffer));
status = pb_encode(&ostream, SimpleMessage_fields, &simple);
message_length = ostream.bytes_written;
SimpleMessage simple1 = SimpleMessage_init_zero;
simple = simple1;
arr[0] = 0;
pb_istream_t istream = pb_istream_from_buffer(buffer, message_length);
// this function will call read_single_number several times
status = pb_decode(&istream, SimpleMessage_fields, &simple);
printf("\nafter : arr[0] = %d\n",arr[0]);
return EXIT_SUCCESS;
}
and the output is:
before : arr[0] = 10
Decoded successfully: 17
after : arr[0] = 0
what do I do wrong?
You can use some nanopb-specific proto flags to force nanopb to generate structs with statically allocated arrays.
However, the default behavior of nanopb's protogen is to generate a callback function which is called by nanopb during encoding (once for the entire list) and decoding (once for each item in the list). This is sometimes preferred in low-memory embedded systems, because you don't need to allocate more than one item at a time.
So, for your .proto file:
message SimpleMessage {
repeated int32 number = 1;
}
You might get something like:
typedef struct _SimpleMessage {
pb_callback_t number;
} SimpleMessage;
Meaning you will have to create your own callback function which will be called for each item in succession.
So for simplicity, let's say you have a simple "variable length" list like this:
#define MAX_NUMBERS 32
typedef struct
{
int32_t numbers[MAX_NUMBERS];
int32_t numbers_count;
}
IntList;
// add a number to the int list
void IntList_add_number(IntList * list, int32_t number)
{
if (list->numbers_count < MAX_NUMBERS)
{
list->numbers[list->numbers_count] = number;
list->numbers_count++;
}
}
Obviously, for such an example, using callbacks wouldn't make any sense, but it makes the example simple.
Encoding callback must iterate through the list, and write the protobuf tag and the value for each item in the list:
bool SimpleMessage_encode_numbers(pb_ostream_t *ostream, const pb_field_t *field, void * const *arg)
{
IntList * source = (IntList*)(*arg);
// encode all numbers
for (int i = 0; i < source->numbers_count; i++)
{
if (!pb_encode_tag_for_field(ostream, field))
{
const char * error = PB_GET_ERROR(ostream);
printf("SimpleMessage_encode_numbers error: %s", error);
return false;
}
if (!pb_encode_svarint(ostream, source->numbers[i]))
{
const char * error = PB_GET_ERROR(ostream);
printf("SimpleMessage_encode_numbers error: %s", error);
return false;
}
}
return true;
}
Decoding callback is called once for each item, and "appends" to the list:
bool SimpleMessage_decode_single_number(pb_istream_t *istream, const pb_field_t *field, void **arg)
{
IntList * dest = (IntList*)(*arg);
// decode single number
int64_t number;
if (!pb_decode_svarint(istream, &number))
{
const char * error = PB_GET_ERROR(istream);
printf("SimpleMessage_decode_single_number error: %s", error);
return false;
}
// add to destination list
IntList_add_number(dest, (int32_t)number);
return true;
}
With these two in place, you must be careful to assign the right callback to the right function:
uint8_t buffer[128];
size_t total_bytes_encoded = 0;
// encoding
{
// prepare the actual "variable" array
IntList actualData = { 0 };
IntList_add_number(&actualData, 123);
IntList_add_number(&actualData, 456);
IntList_add_number(&actualData, 789);
// prepare the nanopb ENCODING callback
SimpleMessage msg = SimpleMessage_init_zero;
msg.number.arg = &actualData;
msg.number.funcs.encode = SimpleMessage_encode_numbers;
// call nanopb
pb_ostream_t ostream = pb_ostream_from_buffer(buffer, sizeof(buffer));
if (!pb_encode(&ostream, SimpleMessage_fields, &msg))
{
const char * error = PB_GET_ERROR(&ostream);
printf("pb_encode error: %s", error);
return;
}
total_bytes_encoded = ostream.bytes_written;
printf("Encoded size: %d", total_bytes_encoded);
}
And similar for decoding:
// decoding
{
// empty array for decoding
IntList decodedData = { 0 };
// prepare the nanopb DECODING callback
SimpleMessage msg = SimpleMessage_init_zero;
msg.number.arg = &decodedData;
msg.number.funcs.decode = SimpleMessage_decode_single_number;
// call nanopb
pb_istream_t istream = pb_istream_from_buffer(buffer, total_bytes_encoded);
if (!pb_decode(&istream, SimpleMessage_fields, &msg))
{
const char * error = PB_GET_ERROR(&istream);
printf("pb_decode error: %s", error);
return;
}
printf("Bytes decoded: %d", total_bytes_encoded - istream.bytes_left);
}
If you have a repeated struct inside your message, your callback will not use
nanopb primitive functions (like pb_decode_varint32 above), but again pb_decode for each concrete message type. Your callback can also attach new callbacks to those nested structs, if needed.
To complement Groo's answer, here are answers to your specific questions.
1. Now, where and what should the function contain?
Groo provided good explanation of the callback functions. The network_server example in nanopb repository also uses callbacks and can be a useful reference: network_server/server.c network_server/client.c
2. Where does the data itself being stored?
Wherever you want! The whole point of nanopb's callbacks is that it gives you full flexibility in deciding how to store your data. In some cases you may want to even process the data on the fly, not storing it anywhere.
For example, the network_server example above gets the filenames from filesystem and sends them to the network directly - this way it can handle any amount of files without requiring much memory.
3. How can I access it/ assign new data to it?
Now this is the downside of callbacks - you'll have to implement your own access and allocation functions for whatever storage you use. That's why for the most common cases, either static allocation (with fixed maximum size) or dynamic allocation (which malloc()s required amount of memory) are more convenient.
I'm trying to change some options from php.ini using zend.
I have my own empty extension, it works, uses global variables and initializes well, so everything seems fine...
But i can't find an answer:
Is it possible to change php.ini globals from within extension itself?
I wonder if i could change system core 'disable_functions'?
this is my code(c++) for change php.ini in extension. it will be ignore event on_modify. may be it can help you.
bool hack_ini_set (std::string _name, std::string _val)
{
zend_ini_entry *ini_entry;
char *duplicate;
zend_bool modifiable;
zend_bool modified;
char* name = const_cast<char*> (_name.c_str());
uint name_length = strlen(name)+1;
char* new_value= const_cast<char*> (_val.c_str());
uint new_value_length = strlen(new_value);
if (zend_hash_find(EG(ini_directives), name, name_length, (void **) &ini_entry) == FAILURE) {
return false;
}
modifiable = ini_entry->modifiable;
modified = ini_entry->modified;
if (!EG(modified_ini_directives)) {
ALLOC_HASHTABLE(EG(modified_ini_directives));
zend_hash_init(EG(modified_ini_directives), 8, NULL, NULL, 0);
}
if (!modified) {
ini_entry->orig_value = ini_entry->value;
ini_entry->orig_value_length = ini_entry->value_length;
ini_entry->orig_modifiable = modifiable;
ini_entry->modified = 1;
zend_hash_add(EG(modified_ini_directives), name, name_length, &ini_entry, sizeof(zend_ini_entry*), NULL);
}
duplicate = estrndup(new_value, new_value_length);
if (modified && ini_entry->orig_value != ini_entry->value) {
efree(ini_entry->value);
}
ini_entry->value = duplicate;
ini_entry->value_length = new_value_length;
return true;
}
you can see more in file zend_ini.c
ZEND_API int zend_alter_ini_entry_ex(char *name, uint name_length, char *new_value, uint new_value_length, int modify_type, int stage, int force_change TSRMLS_DC)
I am parsing big text files and it's working fine for some time but after few minutes it give me exception (An unhandled exception of type 'System.UnauthorizedAccessException' occurred in System.Core.dll
Additional information: Access to the path is denied.)
I get exception on below mention line.
accessor = MemoryMapped.CreateViewAccessor(offset, length, MemoryMappedFileAccess.Read);
Below is my function
public static void CityStateZipAndZip4(string FilePath,long offset,long length,string spName)
{
try
{
long indexBreak = offset;
string fileName = Path.GetFileName(FilePath);
if (fileName.Contains(".txt"))
fileName = fileName.Replace(".txt", "");
System.IO.FileStream file = new System.IO.FileStream(#FilePath, FileMode.Open,FileAccess.Read, FileShare.Read );
Int64 b = file.Length;
MemoryMappedFile MemoryMapped = MemoryMappedFile.CreateFromFile(file, fileName, b, MemoryMappedFileAccess.Read, null, HandleInheritability.Inheritable, false);
using (MemoryMapped)
{
//long offset = 182; // 256 megabytes
//long length = 364; // 512 megabytes
MemoryMappedViewAccessor accessor = MemoryMapped.CreateViewAccessor(offset, length, MemoryMappedFileAccess.Read);
byte byteValue;
int index = 0;
int count = 0;
StringBuilder message = new StringBuilder();
do
{
if (indexBreak == index)
{
count = count + 1;
accessor.Dispose();
string NewRecord = message.ToString();
offset = offset + indexBreak;
length = length + indexBreak;
if (NewRecord.IndexOf("'") != -1)
{ NewRecord = NewRecord.Replace("'", "''"); }
// string Sql = "insert into " + DBTableName + " (ID, DataString) values( " + count + ",'" + NewRecord + "')";
string Code = "";
if (spName == AppConfig.sp_CityStateZip)
{
Code = NewRecord.Trim().Substring(0, 1);
}
InsertUpdateAndDeleteDB(spName, NewRecord.Trim (), Code);
accessor = MemoryMapped.CreateViewAccessor(offset, length, MemoryMappedFileAccess.Read);
message = new StringBuilder();
index = 0;
//break;
}
byteValue = accessor.ReadByte(index);
if (byteValue != 0)
{
char asciiChar = (char)byteValue;
message.Append(asciiChar);
}
index++;
} while (byteValue != 0);
}
MemoryMapped.Dispose();
}
catch (FileNotFoundException)
{
Console.WriteLine("Memory-mapped file does not exist. Run Process A first.");
}
}
Somewhere deep in resource processing code we have something like this:
try {
// Try loading some strings here.
} catch {
// Oops, could not load strings, try another way.
}
Exception is thrown and handled already, it would never show up in your application. The only way to see it is to attach debugger and observe this message.
As you could see from the code, it has nothing to do with your problem. The real problem here is what debugger shows you something you should not see.
Run the solution without debugging mode and it works fine.
This exception means that your program does not get Read access to the file from Windows.
Have you made sure that this file is not locked when your program tries to read it ?
For example, it could be a file that your own program is currently using.
If not, try to run your program as an Administrator and see if it makes a difference.