Combining reading and writing of XML configuration functions. - c

I’m using libxml2 to create and read XML files in c that contain configuration information for the program I’m writing. The program makes its own configuration files (or another program sends it a configuration file and asks the program to run based off the config file), so the XML config files don’t need to be really easy for a human to read.
These configuration files contain lots of values and are really long. So right now I have a function that makes the XML files and another that reads the XML files. However any-time I change the write XML function than I need to also change the read xml function. So there isn’t actual code duplication, but something really close (ie. BAD) and because the configuration files are so long it is rather tedious to try to make sure everything is reading and writing the same thing.
This is the current set up.
struct config_data
{
// category one
int X
int Y
// category two
int Z
int A
}
int makeXMLsheet(char* fileout)
{
xmlDocPtr doc = NULL; /* document pointer */
xmlNodePtr root_node = NULL; /* node pointers */
LIBXML_TEST_VERSION;
doc = xmlNewDoc((xmlChar*) "1.0");
root_node = xmlNewNode(NULL, BAD_CAST "configuration_file");
xmlDocSetRootElement(doc, root_node);
// catogory one
xmlNodePtr category_one = xmlNewChild(root_node, NULL, BAD_CAST "category_one", NULL);
xmlNewChild(category_one, NULL, BAD_CAST "x", BAD_CAST "12345");
xmlNewChild(category_one, NULL, BAD_CAST "y", BAD_CAST "1");
// catogory two
xmlNodePtr category_two = xmlNewChild(root_node, NULL, BAD_CAST "category_two", NULL);
xmlNewChild(category_two, NULL, BAD_CAST "Z", BAD_CAST "12345");
xmlNewChild(category_two, NULL, BAD_CAST "A", BAD_CAST "1");
xmlSaveFormatFileEnc(fileout, doc, "UTF-8", 1);
xmlFreeDoc(doc);
xmlCleanupParser();
return 0;
}
int readXMLsheet(char* filename,struct *config_data)
{
xmlDocPtr doc = getdoc(filename);
config_data->X = getIntegerFromXML(0,doc,(xmlChar*)"//configuration_file/category_one/X");
config_data->Y = getIntegerFromXML(0,doc,(xmlChar*)"//configuration_file/category_one/Y");
config_data->Z = getIntegerFromXML(0,doc,(xmlChar*)"//configuration_file/category_two/Z");
config_data->A = getIntegerFromXML(0,doc,(xmlChar*)"//configuration_file/category_two/a");
xmlFreeDoc(doc);
return 0;
}
Where
int getIntegerFromXML(int defaultValue, xmlDocPtr doc, xmlChar *xpath)
Does as its name says and gets a integer from the opened XML document that has the xpath location, and if it fails then it fills it with the default value so that the program doesn't crash and burn.
So I want to try to some how combine the read and write functions into one. My sample struct config-data is tiny compared to the number of values I actually have in my configuration struct, so combining them would make keeping track of everything much easier.
So I was thinking something like this.
int openXMLvalue(X, Y, Z, readOrWrite, defaultValue, value);
where X, Y, Z are the parent nodes, but there might be more or less than 3.
Any ideas on how to do this? Maybe make some type of array?

I would make generic read and write functions that populate (or serialize) a generic configuration structure.
A simplified case would be to create an key/value structure in memory with get/set methods. The generic writeToXml function would simply create elements with key names containing the values.
If warranted, a hierarchical tree structure could be used instead, and perhaps add a few validation rules when reading a configuration file (a simple one would be to use an XML Schema for validation) to verify that required configuration values exist and are valid.
To add, change or remove configuration values would then only require the following steps (note that neither read or write functions require update):
Decide the new format of the configuration file
Update existing configuration files
Update any places in the application using the configuration values
Optionally update validation rules

Because of the large configuration file size, we switched to using sqlite. Then we made a function that would read a database and make an xml sheet, we made a function that would read an xml sheet and populate the database, and working on functions to print the database to stout and fill the C struct. Think this is going to make life much easier.

Related

How do I read OpenVINO IR models from memory with the OpenVINO C API

I am having trouble reading OpenVINO IR networks (XML and bin) from memory using ie_core_read_network_from_memory() in the OpenVINO 2021.4 C API ie_c_api.h.
I suspect that I am creating the network weight blob wrong, but I cannot find any information on how to create weight blobs correctly for networks.
I have read the OpenVINO C API docs but cannot deduce from docs what I am doing wrong. The OpenVINO code repo contains some C code samples, but none of the samples seem to use ie_core_read_network_from_memory().
Below is a cut out of the code I am having trouble with.
// void* dmem->data - network memory buffer (float32)
// size_t dmem->size - size of network memory buffer (bytes)
ie_core_t* ov_core = NULL;
IEStatusCode status = ie_core_create("", &ov_core);
if (status != OK)
{
// error handling
}
const dimensions_t weights_tensor_dims =
{ 4, { 1, 1, 1, dmem->size/sizeof(float) } };
tensor_desc_t weights_tensor_desc = { OIHW, weights_tensor_dims, FP32 };
ie_blob_t* ov_model_weight_blob = NULL;
status = ie_blob_make_memory_from_preallocated(
&weights_tensor_desc, dmem->data, dmem->size, &ov_model_weight_blob);
if (status != OK)
{
// error handling
}
// char* model_xml_desc - the model's XML string
uint8_t* ov_model_xml_content = (uint8_t*)model_xml_desc;
ie_network_t* ov_network = NULL;
size_t xml_sz = strlen(ov_model_xml_content);
status = ie_core_read_network_from_memory(
ov_core, ov_model_xml_content, xml_sz, ov_model_weight_blob, &ov_network);
if (status != OK)
{
// Always get "GENERAL_ERROR (-1)"
}
The code works fine down to the ie_core_read_network_from_memory() call which results in "GENERAL_ERROR".
I have tried two models that were converted from Tensorflow. One is a simple [X] -> [Y] regression model (single input value, single output value). The other is also a regression model [X_1, X_2, ..., X_9] -> [Y] (nine input values, single output value). They work fine when reading them from file with ie_core_read_network(), but for my use case I must provide the network as a binary memory buffer and XML string.
I would appreciate any help, either by pointing out what I am getting wrong or directing me to some code samples that use ie_core_read_network_from_memory().
System information:
Windows 10
OpenVINO v2021.4.689
Microsoft Visual Studio 2019
UPDATE: An Intel employee reached out to me in another forum and pointed out that there is a unit test for ie_core_read_network_from_memory(). The unit test successfully reads a network from memory and made clear that I was in fact using a faulty tensor description to produce the weight blob, just as I suspected. Apparently the weight blob descriptor should be one dimensional, have memory layout ANY and datatype U8 even though the model weights are fp32.
From the unit test:
std::string bin_std = TestDataHelpers::generate_model_path("test_model", "test_model_fp32.bin");
const char* bin = bin_std.c_str();
//...
std::vector<uint8_t> weights_content(content_from_file(bin, true));
tensor_desc_t weights_desc { ANY, { 1, { weights_content.size() } }, U8 };
However, simply changing the tensor descriptor was not enough to get my code to work so it remains for me to properly translate the C++ code from the unit test to my C environment before the issue to can be considered solved.
Thanks
Refer to tensor_desc struct and standard layout format.
Apart from that, it is recommended to use the Benchmark_app tool to test the inference performance.

Create shared parameter file for C and Python

I need to create a parameter file that can be managed across a Python 3.7 and a C code base. This file needs to be modifiable either by the C or the Python program with the changes being taking effect on the other software (an update function will handle reading the updated file). It's best if the file is not human readable, as it contains information that is better left obfuscated.
**Is there a recommended method to do so? **
I could create separate python and C files, but the set of parameters will change over time (for code maintenance), and the values would be changed by these programs. The list would also be very long. It would be a hassle to maintain two different files and update them over time. Also, the file may need to be exchanged between users, such that a version modified by the software ran by user1 needs to be readable by the software run by user2. The idea is that other parts of both codes could access parts of the parameter list without knowing the full contents of the list.
To clarify the example, I could have a parameter.h file containing:
struct {
double par1 =1.1;
int par 2 =2;
} par_list
And I could have a parameter.py with:
class par_list:
def(__self__):
self.par1 = double(1.1)
self.par2 = int(2)
Then, by doing a import in Python or a include in C, I could initialize the parameter list. But in this case the parameters are being read on different files.
I'm considering using some kind of binary file to keep the values, and create a script that writes both the Python and C code that reads and updates the values. I'm concerned because the binary file would need to be interchangeable between ARM architecture running Linux, and x86 architecture running Windows.
Here is an example working with numpy:
C code:
#include <stdio.h>
#include <stdint.h>
struct Struct_format{
uint8_t the_unsigned_int8;
int32_t the_signed_int32[2];
double the_double;
};
typedef struct Struct_format upperStruct;
//Use separate file to define default value:
void printStruct(upperStruct test_struct){
printf("test_struct.the_unsigned_int8 = %d\n", test_struct.the_unsigned_int8);
printf("test_struct.the_signed_int32[0] = %d\n", test_struct.the_signed_int32[0]);
printf("test_struct.the_signed_int32[1] = %d\n", test_struct.the_signed_int32[1]);
printf("test_struct.the_double = %f\n", test_struct.the_double);
}
void main(){
//Define a "default" value:
upperStruct fromC2Python = {4U,{-3,-1},2.1};
printf("Printing fromC2Python\n");
printStruct(fromC2Python);
//Save this default in a file:
FILE * fid = fopen("fromC2Python.bin","w");
fwrite((void *)&fromC2Python, sizeof(fromC2Python) ,1, fid);
fclose(fid);
//Now load the file created by Python:
upperStruct fromPython2C;
FILE * fid_py = fopen("fromPython2C.bin","r");
fread(&fromPython2C, sizeof(fromPython2C) ,1, fid_py);
fclose(fid_py);
printf("Printing fromPython2C\n");
printStruct(fromPython2C);
}
Python code:
import numpy
datatype = numpy.dtype([('potato',
[('time', numpy.uint8),
('sec', numpy.int32, 2)]),
('temp', numpy.float64)],
align=True)
fromPython2C = numpy.array([((5, (-6, -7)), 61.55)], dtype=datatype)
print(fromPython2C)
fromPython2C.tofile("fromPython2C.bin", sep="")
fromC2Python = numpy.fromfile("fromC2Python.bin", dtype=datatype, count=-1, sep="")
print(fromC2Python)
print(fromC2Python['potato'])
print(fromC2Python['potato']['time'])
print(fromC2Python['temp'])
The ideia is that numpy allows reading and writing to structured binary files. Hence, it suffices to create the dtype specification with a text parser.

How to create a soap_dom_element to assign it to __any with gSOAP in C?

I am trying to port these C++ line to C with gSOAP library:
trt__Capabilities *capabilities = ctx->getMediaServiceCapabilities(this->soap);
tds__GetServicesResponse.Service.back()->Capabilities->__any = soap_dom_element(this->soap, NULL, "trt:Capabilities", capabilities, capabilities->soap_type());
I thought this would do it but soap_dom_element cannot be used like this in the C gSOAP API
tds__GetServicesResponse->Service[1].Capabilities->__any = soap_dom_element(soap, NULL, "trt:Capabilities", capabilities, SOAP_TYPE__tds__Service_Capabilities);
Another thing I tried that compiles but crash at runtime
struct trt__Capabilities *capabilities = fillServiceCapabilities(ctx, soap);
char * tag = "trt:Capabilities";
char * type = "";
int id = -1;
soap_element_begin_out(soap, tag, soap_embedded_id(soap, id, capabilities,
SOAP_TYPE_trt__Capabilities), type);
void * elt = soap_element_end_out(soap, tag);
soap_add_elt(&tds__GetServicesResponse->Service[1].Capabilities->__any, elt);
Please someone help me find the correct way to create a new dom element and assign it to a "struct soap_dom_element __any" object.
Thanks!
All you need to do is simply set the __any member to a dynamically allocated DOM node with soap_elt_new:
tds__GetServicesResponse->Service[1].Capabilities->__any = soap_elt_new(soap, NULL, "trt:Capabilities”);
and then set the serializable "node" and "type" values of this DOM node to the serializable C capabilities data of type tds__Service_Capabilities:
soap_elt_node(tds__GetServicesResponse->Service[1].Capabilities->__any, capabilities, SOAP_TYPE__tds__Service_Capabilities);
This serializes capabilities as element <trt:Capabilities>.
See the gsoap C DOM API documentation.
EDIT: I meant soap_elt_node rather than soap_dom_node as corrected above.

Unique String generator

I want to make a program (network server-client).
One of the specification for this program is next:
The server will receive the sent packages and save it into a file, with a unique name (generated by the server at the moment the transfer starts.
Ex __tf_"unique_random_string".txt
I made a function that returns a pointer to a "unique" string created.
The problem is: If i stop the server and then start it again it will generate the same names.
Ex:this file names were generated and then i stopped the server.
__ft_apqfwk.txt
__ft_arzowk.txt
__ft_cdyggx.txt
I start it again and i try to generate 3 file names. Them will be the same.
Sorry for my english. I'm still learning it.
My function to generate this "unique string" is:
char *create_random_name(void)
{
const char charset[] = "abcdefghijklmnopqrstuvwxyz";
char *file_name;
int i=0;
int key;
if((file_name = malloc(16 * sizeof ( char )) ) == NULL)
{
printf("Failed to alloc memory space\n");
return NULL;
}
strcpy(file_name,"__ft_");
for(i=5 ; i<11 ; i++)
{
key = rand() % (int)(sizeof(charset)-1);
file_name[i]=charset[key];
}
strcat(file_name,".txt");
file_name[15] = '\0';
return file_name;
}
One option is saving to a file the names that have been used, and using them as a checklist. You also want to seed rand with something like srand(time(NULL)).
another is ignoring the randomisation, and just going in order, e.g. aaa, aab aac...aba ,abb etc. Again, save where your cycle is up to on a file.
Your question seems a little bit unclear but if you want to generate a unique string there are a couple of things you can consider:
Get System timestamp ( yyyy-MM-dd-HH-mm-ss-fff-tt)
Use Random function to generate a random number
Combine this with your function and I am sure you will get a unique string.
Hope it helps !
If it's available, you could avoid manually generating random names that might collide and let the system do it for you (and handle collision resolution by creating a new name) by using mkstemps. This is also safer because it opens the file for you, removing the risk of a random name being generated, verified to be unique, then trying to open it and discovering another thread/process raced in and created it.
char name[] = "/path/to/put/files/in/__ft_XXXXXX.txt";
int fd = mkstemps(name, strlen(".txt"));
if (fd == -1) { ... handle error ... }
After mkstemps succeeds, name will hold the path to the file (it's mutated in place, replacing the XXXXXX string), and fd will be an open file descriptor to that newly created file; if you need a FILE*, use fdopen to convert to a stdio type.
Before calling rand(),--- once and only once---, call srand(time()) to initialize the random number generator.
Before settling on any specific file name, call stat() to assure that file name does not already exist.

git diff of just one file

The only way I found to get a diff from a single file using libgit2 is through git_diff_foreach and checking the filename at the diff_file_cb callback.
It's not the way I wanted to do this, I was looking for something easier.
Is there another way to do this?
Just to clarify, git_diff_index_to_workdir (or git_diff_tree_to_tree or another such function) finds the list of changed files and then git_diff_foreach walks through the found files and the text of the diffs. You can pass a "pathspec" in the options structure for git_diff_index_to_workdir that will limit the files being checked. You would do that as mentioned in the prior answer.
As a slightly broader example, if you wanted to diff a more complex set of files, you could write something like:
git_diff *diff;
git_diff_options opts = GIT_DIFF_OPTIONS_INIT;
char *files[3];
files[0] = "myfile.txt";
files[1] = "yourfile.txt";
files[2] = "some/directory/*.h"
opts.pathspec.count = 3;
opts.pathspec.strings = files;
if (git_diff_index_to_workdir(&diff, repo, NULL, &opts) < 0) {
printf("Failed to diff\n");
exit(1);
}
git_diff_foreach(diff, file_cb, NULL, NULL, NULL);
git_diff_free(diff);
You can pass as many file names or file patterns as you like. If you want to disable the pattern matching behavior (i.e. expanding * and such), you can write opts.flags |= GIT_DIFF_DISABLE_PATHSPEC_MATCH and only exact file name matches will be used.
Much easier using the pathspec option and git_diff_index_to_workdit instead of git_diff_foreach
char *pathspec = "foo.bar";
git_diff_options opts = GIT_DIFF_OPTIONS_INIT;
opts.pathspec.strings = &pathspec;
opts.pathspec.count = 1;
git_diff_index_to_workdir(&diff, repo, NULL, &opts);

Resources