I need to create a parameter file that can be managed across a Python 3.7 and a C code base. This file needs to be modifiable either by the C or the Python program with the changes being taking effect on the other software (an update function will handle reading the updated file). It's best if the file is not human readable, as it contains information that is better left obfuscated.
**Is there a recommended method to do so? **
I could create separate python and C files, but the set of parameters will change over time (for code maintenance), and the values would be changed by these programs. The list would also be very long. It would be a hassle to maintain two different files and update them over time. Also, the file may need to be exchanged between users, such that a version modified by the software ran by user1 needs to be readable by the software run by user2. The idea is that other parts of both codes could access parts of the parameter list without knowing the full contents of the list.
To clarify the example, I could have a parameter.h file containing:
struct {
double par1 =1.1;
int par 2 =2;
} par_list
And I could have a parameter.py with:
class par_list:
def(__self__):
self.par1 = double(1.1)
self.par2 = int(2)
Then, by doing a import in Python or a include in C, I could initialize the parameter list. But in this case the parameters are being read on different files.
I'm considering using some kind of binary file to keep the values, and create a script that writes both the Python and C code that reads and updates the values. I'm concerned because the binary file would need to be interchangeable between ARM architecture running Linux, and x86 architecture running Windows.
Here is an example working with numpy:
C code:
#include <stdio.h>
#include <stdint.h>
struct Struct_format{
uint8_t the_unsigned_int8;
int32_t the_signed_int32[2];
double the_double;
};
typedef struct Struct_format upperStruct;
//Use separate file to define default value:
void printStruct(upperStruct test_struct){
printf("test_struct.the_unsigned_int8 = %d\n", test_struct.the_unsigned_int8);
printf("test_struct.the_signed_int32[0] = %d\n", test_struct.the_signed_int32[0]);
printf("test_struct.the_signed_int32[1] = %d\n", test_struct.the_signed_int32[1]);
printf("test_struct.the_double = %f\n", test_struct.the_double);
}
void main(){
//Define a "default" value:
upperStruct fromC2Python = {4U,{-3,-1},2.1};
printf("Printing fromC2Python\n");
printStruct(fromC2Python);
//Save this default in a file:
FILE * fid = fopen("fromC2Python.bin","w");
fwrite((void *)&fromC2Python, sizeof(fromC2Python) ,1, fid);
fclose(fid);
//Now load the file created by Python:
upperStruct fromPython2C;
FILE * fid_py = fopen("fromPython2C.bin","r");
fread(&fromPython2C, sizeof(fromPython2C) ,1, fid_py);
fclose(fid_py);
printf("Printing fromPython2C\n");
printStruct(fromPython2C);
}
Python code:
import numpy
datatype = numpy.dtype([('potato',
[('time', numpy.uint8),
('sec', numpy.int32, 2)]),
('temp', numpy.float64)],
align=True)
fromPython2C = numpy.array([((5, (-6, -7)), 61.55)], dtype=datatype)
print(fromPython2C)
fromPython2C.tofile("fromPython2C.bin", sep="")
fromC2Python = numpy.fromfile("fromC2Python.bin", dtype=datatype, count=-1, sep="")
print(fromC2Python)
print(fromC2Python['potato'])
print(fromC2Python['potato']['time'])
print(fromC2Python['temp'])
The ideia is that numpy allows reading and writing to structured binary files. Hence, it suffices to create the dtype specification with a text parser.
Related
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 1 year ago.
Improve this question
As they say, your learn coding techniques from others' code. I've been trying to understand couple of free stacks and they all have one thing in common: Structure of function pointers. I've following of questions related to this architecture.
Is there any specific reason behind such an architecture?
Does function call via function pointer help in any optimization?
Example:
void do_Command1(void)
{
// Do something
}
void do_Command2(void)
{
// Do something
}
Option 1: Direct execution of above functions
void do_Func(void)
{
do_Command1();
do_Command2();
}
Option 2: Indirect execution of above functions via function pointers
// Create structure for function pointers
typedef struct
{
void (*pDo_Command1)(void);
void (*pDo_Command2)(void);
}EXECUTE_FUNC_STRUCT;
// Update structure instance with functions address
EXECUTE_FUNC_STRUCT ExecFunc = {
do_Command1,
do_Command2,
};
void do_Func(void)
{
EXECUTE_FUNC_STRUCT *pExecFunc; // Create structure pointer
pExecFun = &ExecFunc; // Assign structure instance address to the structure pointer
pExecFun->pDo_Command1(); // Execute command 1 function via structure pointer
pExecFun->pDo_Command2(); // Execute command 2 function via structure pointer
}
While Option 1 is easy to understand and implement, why do we need to use Option 2?
While Option 1 is easy to understand and implement, why do we need to use Option 2?
Option 1 doesn't allow you to change the behavior without changing the code - it will always execute the same functions in the same order every time the program is executed. Which, sometimes, is the right answer.
Option 2 gives you the flexibility to execute different functions, or to execute do_Command2 before do_Command1, based decisions at runtime (say after reading a configuration file, or based on the result of another operation, etc.).
Real-world example from personal experience - I was working on an application that would read data files generated from Labview-driven instruments and load them into a database. There were four different instruments, and for each instrument there were two types of files, one for calibration and the other containing actual data. The file naming convention was such that I could select the parsing routine based on the file name. Now, I could have written my code such that:
void parse ( const char *fileName )
{
if ( fileTypeIs( fileName, "GRA" ) && fileExtIs( fileName, "DAT" ) )
parseGraDat( fileName );
else if ( fileTypeIs( fileName, "GRA" ) && fileExtIs ( fileName, "CAL" ) )
parseGraCal( fileName );
else if ( fileTypeIs( fileName, "SON" ) && fileExtIs ( fileName, "DAT" ) )
parseSonDat( fileName );
// etc.
}
and that would have worked just fine. However, at the time, there was a possibility that new instruments would be added later and that there may be additional file types for the instruments. So, I decided that instead of a long if-else chain, I would use a lookup table. That way, if I did have to add new parsing routines, all I had to do was write the new routine and add an entry for it to the lookup table - I didn't have to modify any of the main program logic. The table looked something like this:
struct lut {
const char *type;
const char *ext;
void (*parseFunc)( const char * );
} LUT[] = { {"GRA", "DAT", parseGraDat },
{"GRA", "CAL", parseGraCal },
{"SON", "DAT", parseSonDat },
{"SON", "CAL", parseSonCal },
// etc.
};
Then I had a function that would take the file name, search the lookup table, and return the appropriate parsing function (or NULL if the filename wasn't recognized):
void (*parse)(const char *) = findParseFunc( LUT, fileName );
if ( parse )
parse( fileName );
else
log( ERROR, "No parsing function for %s", fileName );
Again, there's no reason I couldn't have used the if-else chain, and in retrospect it's probably what I should have done for that particular app1. But it's a really powerful technique for writing code that needs to be flexible and responsive.
I suffer from a tendency towards premature generalization - I'm writing code to solve what I think will be issues five years from now instead of the issue today, and I wind up with code that tends to be more complex than necessary.
Best explained via Example.
Example 1:
Lets say you want to implement a Shape class with a draw() method, then you would need a function pointer in order to do that.
struct Shape {
void (*draw)(struct Shape*);
};
void draw(struct Shape* s) {
s->draw(s);
}
void draw_rect(struct Shape *s) {}
void draw_ellipse(struct Shape *s) {}
int main()
{
struct Shape rect = { .draw = draw_rect };
struct Shape ellipse = { .draw = draw_ellipse };
struct Shape *shapes[] = { &rect, &ellipse };
for (int i=0; i < 2; ++i)
draw(shapes[i]);
}
Example 2:
FILE *file = fopen(...);
FILE *mem = fmemopen(...); /* POSIX */
Without function pointers, there would be no way to implement a common interface for file and memory streams.
Addendum
Well, there is another way. Based on the Shape example:
enum ShapeId {
SHAPE_RECT,
SHAPE_ELLIPSE
};
struct Shape {
enum ShapeId id;
};
void draw(struct Shape *s)
{
switch (s->id) {
case SHAPE_RECT: draw_rect(s); break;
case SHAPE_ELLIPSE: draw_ellipse(s); break;
}
}
The advantage of the second example could be, that the compiler could inline the functions, then you would have omitted the overhead of a function call.
"Everything in computer science can be solved with one more level of indirection."
The struct-of-function-pointers "pattern", let's call it, permits runtime choices. SQLite uses it all over the place, for example, for portability. If you provide a "file system" meeting its required semantics, then you can run SQLite on it, with Posix nowhere in sight.
GnuCOBOL uses the same idea for indexed files. Cobol defines ISAM semantics, whereby a program can read a record from a file by specifying a key. The underlying name-value store can be provided by several (configurable) libraries, which all provide the same functionality, but use different names for their "read a record" function. By wrapping these up as function pointers, the Cobol runtime support library can use any of those key-value systems, or even more than one at the same time (for different files, of course).
I am trying to convert some of the ruby interpreter code called in C to mruby format. I am stuck and would appreciate help here.
My testruby.rb file content:
#require 'MyMod'
def helloworld(var1)
puts "You said #{var1}"
return MyMod.Issue1(var1).to_s
end
Below is the snippet of my C++ file:
Issue 1:
static mrb_value Issue1(mrb_state *mrb, mrb_value mrb_self)
{
mrb_??? val1; // What should be the type for string and where to find all the types?
mrb_get_args(mrb, "s", ?);
// How to manipulate val1? Say I want to concatenate few more data.
return mrb_????(val1); // How do I return this value?
}
The above method, I am sending as a module to the mruby interpreter so that .rb file can call this.
Please let me know if below format is the correct one:
struct RClass *mod = mrb_define_module(mrb, "MyMod");
mrb_define_module_function(mrb, mod, "SumI", Issue1, MRB_ARGS_REQ(1));
Issue2:
How do I convert the below ruby interpreter code to mruby?
rb_require("./testruby"); // where testruby is my testruby.rb file
Now I want to call the helloworld method from testruby.rb file. How do I call the equivalent method for mruby (for rb_funcall)?
How do I read the return value from the helloworld method in my c++ code?
Regards,
Re val1: mrb_value is the type that can hold any mruby object
Manipulating val1 could be done using mrb_funcall. That function returns a mrb_value:
mrb_value my_str = mrb_funcall(mrb_context, your_object, "your_method", 0);
printf("my_str = %s\n", RSTRING_PTR(my_str));
Re issue 2: There's no require in mruby: mrbgems are compiled and linked statically with the target binary (they are listed in the top-level build_config.rb file).
(A gem called mruby-require exists to mimic CRuby's require, but I've never used it)
I'm looking for a way to create a plugin interface for a C program. In the interest of a short and concise example, let's say I have the following program:
#include <stdio.h>
int complexFunction(int a, int b) { return a+b; }
int main() {
int a = 3;
int b = 6;
printf("The result of C(a, b) is %d.\n", complexFunction(a, b));
return 0;
}
Instead of using the defined complexFunction however, I would like to allow the user to code their own and then plug-in their implementation.
I have never worked with plugin implementations myself, but so far I have been able to come up with the following possibilities for a plugin interface:
Create a 'mini scripting language' which my program can parse and create a procedure pipeline for, then pass the variables into that pipeline. This would involve a lot of 'generic' functions for each possible action in the scripting language.
Use a pre-existing scripting language for the same purpose (however this may not be possible as I would need some very specialised scriptable functions).
Allow the user to write and compile a C file containing the definition of complexFunction(int, int), and dynamically load that at run-time. This is the least secure option but for development purposes it would be ideal as (only) I would author my own plugins and I could dynamically add functionality to my program.
My question is twofold:
What are the common ways of handling plug-in capabilities in Applications?
How can I accomplish (3) / is (3) even possible in C? After fopen-ing a file, can I somehow use it as code in the current program?
A scripting language (usually a pre-existing one like lua or something) is the most common way. 3 is possible- you could look in a predefined folder for dynamic libraries (.so or .dll files) and look for functions with specific names. Its doable, but there's huge security issues with it (you have no security really).
3)
If the user can compile your function into a .dll, you can use the following steps:
a) use LoadLibrary to load the dll
b) use GetProcAddress to get the address of the function
c) execute the function
Example:
typedef int (__cdecl *PCF)(int, int);
HMODULE hDll = LoadLibrary("x:\\Dllpath\\MyPlugIn.dll");
if (hDll != NULL)
{ PCF pfnComplexFunction = GetProcAddress(hDll, "complexFunction");
if (pfnComplexFunction != NULL)
{ int a = 3;
int b = 6;
printf("The result of C(a, b) is %d.\n", pfnComplexFunction(a, b));
}
FreeLibrary(hDll);
}
The function in the dll would need to be declared like
__declspec(dllexport) int __cdecl complexFunction(int a, int b)
{ return a + b;
}
Basically, i have a program that is given a 4 meg compressed file, it has to decode this file into uncompressed ~ 100 meg, then compress it back into ~4 meg file. I need to store this intermediate 100 meg file somewhere on the drive (dont want to keep it in memory).
Program is written in C and will be executed on MS Windows 7. At the moment of uncompressing, no guaranteed folder (with write access) is given to the program (folder with source file might be read only and folder with target file might be not yet specified).
This has proven to be not an easy task:
1) I have read about a C function that creates a temp file that will disappear when closed or program is terminated. However, from what i understand it tries to make the file on disk C, in root directory, so this will obviously fail if user has no rights for that (which normal user doesnt)
2) I had an idea to use environmental/system variable TEMP and create a file there, BUT looking on a random Win7 PC which wasnt tweaked, i see that this variable points to c:/windows/temp, and that folder has specific rights for "users" - that is, they have rights to read, execute, create and write files, but not to delete them, check their attributes, etc. This means, i assume, that if program is ran with user privilleges, it will be able to make a file but not able to delete it, so the only way to "delete" it would be to open the file for writing and then close it, making it a 0 length file. This is also not desired, and i dont know how to query for system variables from C
3) So, basically, only idea i have right now is to make a function to open file that:
tries to create a temp file in the output dir, if possible
if failed, tries to create a temp file in input dir
if failed, tries to create a temp file in TEMP dir from system variable
if failed, tries to create a temp file in TMP dir from system variable
and a delete function that:
tries to remove() the file (by its name that is stored somewhere)
if failed, it tries to open the file for write, and close it, so it becomes a 0 byte file
Are there better ideas?
Any help is appreciated, thanks!
PS: Program must not use any external libraries like MFC or something, only built-in standart C functions
GetTempPath
Retrieves the path of the directory designated for temporary files.
GetTempFileName
Creates a name for a temporary file. If a unique file name is
generated, an empty file is created and the handle to it is released;
otherwise, only a file name is generated.
These two provide you easy way to obtain a location and name for a temporary file.
UPD: Code sample on MSDN: Creating and Using a Temporary File.
#include <windows.h>
#include <iostream>
#include <chrono>
#include <string>
#include <cstdio>
#include <chrono>
using namespace std;
int FileExists(string& filepath)
{
DWORD dwAttrib = GetFileAttributes(filepath.c_str());
return (dwAttrib != INVALID_FILE_ATTRIBUTES &&
!(dwAttrib & FILE_ATTRIBUTE_DIRECTORY));
}
int GetTemporaryFilePath(
string filePrefix,
string fileExt,
string& TmpFilePath /*return*/)
{
if (fileExt[0] == '.')
fileExt.erase(0,1);
char TempPath[MAX_PATH] = { 0 };
if (!GetTempPath(MAX_PATH, TempPath))
return -1;
uint16_t tickint = 0;
while(1) {
const int nowlen = 17; char nowstr[nowlen];
const int ticklen = 5; char tickstr[ticklen];
// Milliseconds since 1970
auto ms = chrono::duration_cast<chrono::milliseconds>(
chrono::system_clock::now().time_since_epoch()
);
__int64 nowint = ms.count();
snprintf(nowstr, nowlen, "%016" "I64" "x", nowint);
snprintf(tickstr, ticklen, "%04x", tickint);
TmpFilePath = string(TempPath)
+ filePrefix
+ "." + string(nowstr)
+ "." + string(tickstr)
+ "." + fileExt;
if (!FileExists(TmpFilePath)) {
//Touch File
FILE* w = fopen(TmpFilePath.c_str(), "w");
fclose(w);
break;
}
tickint++;
}
return 0;
}
int main()
{
string TmpFilePath;
GetTemporaryFilePath("MyFile", ".txt", TmpFilePath);
cout << "TmpFilePath: " << TmpFilePath << endl;
return 0;
}
Some time ago we took over the responsibility of a legacy code base.
One of the quirks of this very badly structured/written code was that
it contained a number of really huge structs, each containing
hundreds of members. One of the many steps that we did was to clean
out as much of the code as possible that wasn't used, hence the need
to find unused structs/struct members.
Regarding the structs, I conjured up a combination of python, GNU
Global and ctags to list the struct members that are unused.
Basically, what I'm doing is to use ctags to generate a tags file,
the python-script below parses that file to locate all struct
members and then using GNU Global to do a lookup in the previously
generated global-database to see if that member is used in the code.
This approach have a number of quite serious flaws, but it sort of
solved the issue we faced and gave us a good start for further
cleanup.
There must be a better way to do this!
The question is: How to find unused structures and structure members
in a code base?
#!/usr/bin/env python
import os
import string
import sys
import operator
def printheader(word):
"""generate a nice header string"""
print "\n%s\n%s" % (word, "-" * len(word))
class StructFreqAnalysis:
""" add description"""
def __init__(self):
self.path2hfile=''
self.name=''
self.id=''
self.members=[]
def show(self):
print 'path2hfile:',self.path2hfile
print 'name:',self.name
print 'members:',self.members
print
def sort(self):
return sorted(self.members, key=operator.itemgetter(1))
def prettyprint(self):
'''display a sorted list'''
print 'struct:',self.name
print 'path:',self.path2hfile
for i in self.sort():
print ' ',i[0],':',i[1]
print
f=open('tags','r')
x={} # struct_name -> class
y={} # internal tags id -> class
for i in f:
i=i.strip()
if 'typeref:struct:' in i:
line=i.split()
x[line[0]]=StructFreqAnalysis()
x[line[0]].name=line[0]
x[line[0]].path2hfile=line[1]
for j in line:
if 'typeref' in j:
s=j.split(':')
x[line[0]].id=s[-1]
y[s[-1]]=x[line[0]]
f.seek(0)
for i in f:
i=i.strip()
if 'struct:' in i:
items=i.split()
name=items[0]
id=items[-1].split(':')[-1]
if id:
if id in y:
key=y[id]
key.members.append([name,0])
f.close()
# do frequency count
for k,v in x.iteritems():
for i in v.members:
cmd='global -a -s %s'%i[0] # -a absolute path. use global to give src-file for member
g=os.popen(cmd)
for gout in g:
if '.c' in gout:
gout=gout.strip()
f=open(gout,'r')
for line in f:
if '->'+i[0] in line or '.'+i[0] in line:
i[1]=i[1]+1
f.close()
printheader('All structures')
for k,v in x.iteritems():
v.prettyprint()
#show which structs that can be removed
printheader('These structs could perhaps be removed')
for k,v in x.iteritems():
if len(v.members)==0:
v.show()
printheader('Total number of probably unused members')
cnt=0
for k,v in x.iteritems():
for i in v.members:
if i[1]==0:
cnt=cnt+1
print cnt
Edit
As proposed by #Jens-Gustedt using the compiler is a good way to do it. I'm after a approach that can do a sort of "High Level" filtering before using the compiler-approach.
If these are only a few struct and if the code does no bad hacks of accessing a struct through another type... then you could just comment out all the fields of your first struct and let the compiler tell you.
Uncomment one used field after the other until the compiler is satisfied. Then once that compiles, to a good testing to ensure the precondition that there were no hacks.
Iterate over all struct.
Definitively not pretty, but at the end you'd have at least one person who knows the code a bit.
Use coverity. This is a wonderful tool to detect code flaws, but is a bit costly.
Although it is a very old post. But recently I did the same using python and gdb. I compiled following snippet of code with structure at the top of hierarchy and then using gdb did print type on the structure and re-cursed into its members.
#include <usedheader.h>
UsedStructureInTop *to_print = 0;
int main(){return 0;}
(gdb) p to_print
(gdb) $1 = (UsedStructureInTop *) 0x0
(gdb) pt UsedStructureInTop
type = struct StructureTag {
members displayed here line by line
}
(gdb)
Although my purpose is little different. It is to generate a header that contains only the structure UsedStructureInTop and its dependency types. There are compiler options to do this. But they do not remove unused/unlinked structures found in the included header files.
Under C rules, it's possible to access struct members via another structure which has a similar layout. That means that you can access struct Foo {int a; float b; char c; }; via struct Bar { int x; float y; }; (except of course for Foo::c).
Hence, your algorithm is potentially flawed. It's bloody hard to find what you want, which BTW is why C is hard to optimize.