I'm new to Rust. For practice I'm trying to create a dynamic library compatible with C ABI in Rust.
My use case: I have an existing program written in C and it supports runtime plugins. Basically when the program initializes, it looks at the environment and loads the plugin, which is a dynamic library. The program assumes the metadata of the plugin (a struct of pointers to functions and a const char* identifier) is a global variable in the dynamic lib, if the lib is written in C. When the program needs certain functionality, it just calls the plugin by: plugin->func(params). The plugin is what I'm trying to create here.
To clarify, the plugin would look something like this in C:
// plugin.h
struct plugin_t {
char *identifier;
int (*foo)(int bar);
};
// plugin.c
int my_foo(int bar) {
/* do something */
}
struct plugin_t plugin_impl = {
.identifier = "net-plugin",
.foo = my_foo,
};
I used bindgen to convert the plugin header file into a bindings.rs specification. bindings.rs is quite messy; complying with this specification would require a large amount of type conversion, both from and back to C. Besides, interfacing with C involves writing a lot of unsafe code, which contradicts with what makes Rust a great language to start with (memory safety).
Based on my experience and the great reputation Rust has in the community, I'm guessing I did it quite wrong. My question is, what is the optimal way to create a dynamic library compatible with C ABI in Rust? By optimal I mean to to decouple Rust code from C code as much as possible.
Related
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 1 year ago.
Improve this question
First and foremost, I'm no expert in software, and I realize this question might be as vague as it gets (it's just a curiosity), and get ready to read some barbaric guesses from my part!
The question came up from reading an article on how Linux's developers were implementing Rust into their OS (https://www.zdnet.com/article/linus-torvalds-on-where-rust-will-fit-into-linux/).
What does it even mean to implement Rust in their OS? Do they have some compiled code written in C that calls compiled code that resulted from writing in Rust? I don't see how this can efficiently be done, as you'd have different compilers probably being unable to optimize code, since, along the way, it's calling "foreign" code. I imagine the situation gets worse if you throw a language like Python or Java into the mix, which aren't precompiled. Now you would have JVM or PVM running together with compiled code, which, I imagine, would be highly impractical. One approach I can conceive, is if you see all these things as separate processes, and you'd simply have code from one language starting a process that corresponds to code from another language, but again, I imagine that wouldn't be very efficient...
Again, I realize I could've been more direct, but I'm not looking for a specific answer which addresses a problem, but rather general insight on how different languages can get used together. Thanks for understanding!
Typically when compiling a source file we have a set of options for outputs generated by the compiler, like creating a binary application from the main function, a statically linkable binary or a dynamically linkable library, or in other cases, source to source transformations.
The kernel is written in C, and to be able to compile large codebases like the kernel, we often decide to compile each, or a set of source files (the proper term would be translation unit https://en.wikipedia.org/wiki/Translation_unit_(programming)) into a statically linked library or object files. Once we've gathered all of the object files and statically (or shared) linked libraries we can link these together and produce a final binary/library.
When we're talking about integrating Rust code into the kernel, we're talking about using statically linked libraries from C in Rust and vice versa. The process of calling code produced from other compilers or languages is named Foreign Function Interface, or FFI.
There are many details and challenges with FFI including ABIs or name mangling to name a few. ABI, or application binary interface is one of the issues mentioned in your article. Unlike C, Rust does not have a stable ABI yet, meaning there's no guarantee the symbols in the static library compiled from Rust won't have different names or data layouts in the future. This means that code compiled using Rust's compiler may not be compatible with previous Rust compiler outputs which would require the C code to be updated every time there's an ABI change.
Here's how a C program is traditionally compiled:
Each source file is compiled - separately! - into an object file, which contains machine code, and placeholders for connecting all the object files together.
All the object files are joined together and the placeholders are filled in.
If you can generate some of those object files using Rust instead of C, the C compiler can't tell the difference. It's no different from a different C file!
You need to make sure the functions that cross the language barrier need to be valid functions for both languages. For example, you might not be able to pass structures as arguments, or return structures, if that's not valid in Rust (I don't actually know) - you might only be able to use primitive values, like ints and floats and pointers. If the Rust machine code expects float variables to be in certain registers, but the C machine code puts them in different registers, you might not be able to pass floats. Or you might have to put a special hint to one or the other compiler, to say "the argument goes in this register, dummy!" If the Rust compiler people didn't cooperate with the C compiler people, you are likely to run into a few of these kinds of problems, but with a bit of luck, they can all be worked around.
Since the function needs to be valid in both languages, you also need to write a C header file. You can't #include a Rust file - well you can, but it won't work - so you need to write the function declarations again in C syntax so the C compiler can understand what they are.
For languages which require VMs, it's complicated in a different way.
Usually these languages are so different from C that you can't just link together the machine code. Instead you have to use the VM's API to initialize the VM, load the code, and call the code. Something like this:
// not real code, just an illustration of how it could work
// How C starts a JVM and calls a Java method (a Java function)
void run_jvm() {
jvm_t *jvm = create_jvm();
jvm_class_t *main_class = jvm_load_class(jvm, "Main.class");
jvm_method_t *main_method = jvm_get_method(jvm, main_class, "main", NULL);
jvm_call(jvm, main_method, NULL, NULL);
delete_jvm(jvm);
}
// how Java calls a C function
void Java_HelloPrinter_printHello(jvm_t *jvm, jvm_object_t *this, jvm_args_t *args) {
printf("Hello world! My argument is %d\n", jvm_args_get_int(args, 1));
}
// how the method is declared in Java
public class HelloPrinter {
public static native void printHello(int i);
// ^^^^^^
// this means it's a C function
}
In Python or Lua, being more dynamic languages, the interpreter doesn't look for things for you. Instead, you put your functions into variables before you run any of your Python code.
// not real code, just an illustration of how it could work
// How Python calls a C function
void printHelloFunction(pvm_t *pvm, pvm_args_t *args) {
printf("Hello world! My argument is %d\n", pvm_args_get_int(args, 1));
}
// How C starts a PVM and calls a Python method
void run_python() {
pvm_t *pvm = create_pvm();
pvm_variable_t *var = pvm_create_global_variable(pvm, "printHello");
pvm_set_variable_as_c_function(pvm, var, &printHelloFunction);
pvm_load_module_from_file(pvm, "main.py");
delete_pvm(pvm);
}
These languages are not on equal footing with C, but Rust is, because Rust compiles to machine code. If you could run Java or Python code outside of a VM, they could be on equal footing. There used to be a compiler called gcj which would compile Java into machine code, but it's no longer maintained. Someone could write one, although it wouldn't run most Java programs, because a lot of those programs need to do stuff with the VM (like reflection).
I want to do basic statistics with Vlang.
Can I use C libraries? For example, Apophenia: http://apophenia.info/
Or IMSL C stat library: https://docs.roguewave.com/en/imsl/c/8.6/pdf/C_Stat_library.pdf
Thanks for your help.
Yes, you can call C libraries from V.
You need to make the C library's structs, typedefs and functions known to V by defining them in V first - before calling/using them.
For structs you conveniently only need to define the fields you need to use.
Here's some examples:
via 2D game framework wrapping several C libs
sokol in vlib
v-miniaudio wrapper (disclaimer: my own module)
Generally you can find a lot of C wrapper code in vlib itself. (We're working on replacing C with pure V)
Not sure if this is even possible.
I need to pass a function from C to C++.
It cannot be a function pointer.
C++ Function that I need to call:
template<class Lam>
void parfor(int N, Lam lam) {
lam(i);
}
C Function that I want to give to parfor (ptr can be a global pointer here):
void calc(int num) {
ptr[0] = num;
}
C Main to look like this:
include <ParFor.hpp>
parfor(calc, 1);
I could put my function definitions inside a header. On the C++ side I have a function (from an outside library) that takes a C++ lambda or a C++ functor. It's templated to a lambda
My current thinking is put my C functions inside a file, compile LLVM IR for them and somehow force inline the IR generated by clang into my C++ function.
C calls mycppfunc(mycfunc). mycppfunc has the LLVM IR for mycfunc and is able to generate proper code.
I tried this but but compiler crashes at link stage due to what seems to be incompatible IRs.
From the code snippets and the comments I understand that you attempt to launch a C kernel using a SYCL parallel_for.
From an official support perspective, since the SYCL device compiler is by definition a C++ compiler (as SYCL is a programming model based on C++), the kernel code must be parsed as C++ code. I don't think there's any official way to achieve any more interoperability with C code than to just try to compile it as C++ by the device compiler. However, I have the impression that you are more interested in experimenting with interoperability beyond this as a research project.
For this, injecting IR might be a path worthy of investigation but I don't think that it will be straight-forward. Depending on which SYCL features you use, the device compiler might need to perform additional IR transformations to achieve correct semantics. So, you might have to replicate these transformations in order to inject your IR. You might end up reinventing a SYCL compiler for C...
Since any more in-depth discussion about this requires extensive knowledge of the internal implementation details of your SYCL implementation, I would suggest that you contact the developers of your implementation directly for clarification.
So, I have a set of interdependent .c and .h files, and I'm trying to figure out exactly how C/C++-to-Pascal works (my delphi/pascal work otherwise was pretty basic). The c files are steps 1-N (in order) of a mathematical algorithm (each one is a separate function), one header specifies the variables used in each function (I'm using record to recreate the .h files in delphi), and the other header details a custom variable type. The goal is to access the functions without having to re-write the C code into Pascal/Delphi syntax, linking from delphi code to some existing c. There are too many lengthy files (12 .c and 4 .h) for me to snip, and my C is horribly rusty, but I'll try.
For example, I have a file funcs.h:
#ifndef _FUNCS_H
#ifndef _FUNCS_H
#ifdef __cplusplus
extern "C" {
#endif
#include "New_Type.h" // custom variable used in funcs.h
#include "Optional.h" // optional set of more functions
typedef struct {
int a1[4];
double b1[10];
// Computed if optional functions are requested
double c1[10];
int d1[4];
} func1;
typedef struct {
new_type c2, d2;
} func2;
}
#endif
#endif
and files func1.c and func2.c do some mathematical manipulation (mostly linear algebra) of the above variables. My immediate questions are:
Do I need multiple delphi files, much like there are multiple C headers?
What do the delphi files' "skeletons" look like (premable, function calls)?
What is {$LINKLIB C}, cdecl, and CALLBACK? Their common in other question/answers I've seen
Maybe I need to change my search/question, but I'm honestly not sure how to properly ask this. I don't need a specific answer, honestly I'd prefer if someone could point me to some links (heck a book: I'll be using this language in the future), I'd really appreciate it.
Thanks in advance for all the help.
Starting with #2: In Delphi, both the header and the implementation go in the same file. So here's a basic skeleton of a class showing where your header stuff (.h) and implementation stuff (.c) will fit into the structure of a unit/file in Delphi.
unit Unit1;
interface
uses // List of header dependencies goes here...
// Stuff from the .h file will go here
implementation
uses // List of implementation dependencies goes here...
// Stuff from the .c file will go here
end.
Back to #1: It's not strictly necessary to have a 1:1 mapping of your C header files to Delphi files, but if that was a good level of granularity per file in C, it's probably going to work out similarly in Delphi. Unless you have a good reason to change it, I'd leave them mapped at one C header file to one Delphi file.
About.com has a pretty detailed section on Delphi development, the page, for example, on working with units could be a good starting point on learning about the structure of a Delphi file. I also find myself using DelphiBasics as a reference site a lot (though it's more aimed at methods than structures). And definitely do not discount the official documentation on Embarcadero's website. Their Language Guide section is practically a textbook on learning Delphi, and is sufficiently detailed to get the fundamentals of the language.
And then as far as #3: This could easily be several separate full questions on stack overflow if you want more depth. But in general, it sounds like you're looking at code for calling c code directly from Delphi. CDecl is one of the calling conventions (the one that happens to match c...) which specifies which order parameters are passed to methods. Read more about it in Embarcadero's documentation. $LINKLIB is used to link to a library (in this case the C library, which you would do if you're calling c code), and callback is roughly equivalent to a function pointer in c.
From the overall sound of your third sub-question, I'm guessing you've been reading up on how to use C code in a Delphi project, without having to re-write the C code into Pascal/Delphi syntax, linking from delphi code to some existing c library you don't want to re-write or convert. If that's your ultimate goal, to just link to your C library without rewriting it, this article on using C in FreePascal may be of good help. For the purposes of linking to C code, the differences between FreePascal and Delphi, both variants of Pascal, are going to be relatively negligible). Basically, you have to re-write or convert the c header into Pascal syntax so that Delphi knows what functions are in your C library, so you can call them. Dr. Bob has a good detailed page about using C dlls in Delphi if you're looking for more info about converting header files. Amongst other useful information about how type names match, there is an example on that page of automated tools that exist for converting c header files to delphi headers. It may be easier to use an automated tool to convert your header to delphi syntax, if your goal is just to call or utilize your existing c code from a Delphi application without fully porting the entire code-library to Delphi.
And for the sake of example, your header from your question, converted through Dr. Bob's HeadConverter tool, gives output like this: (the actual output is longer and includes code for loading your c code as a dll and some comments as well)
unit FILE1;
interface
uses
Windows;
{$INCLUDE "New_Type.h"}
{$INCLUDE "Optional.h"}
type
func1 = record
a1: Array[0..4-1] of Integer;
b1: Array[0..10-1] of Double;
c1: Array[0..10-1] of Double;
d1: Array[0..4-1] of Integer;
end {func1};
type
func2 = record
c2, d2: NEW_TYPE;
end {func2};
{...}
implementation
{...}
end.
I have a few ideas:
char* test = "testInteger(5).workOnReturn("doIt")[10]"
int ret = execute(test);
What if I use the 'extern' keyword?
Suppose I have a whole bunch of C++ implementations and classes.
Couldn't I just define the same things in C with 'extern' and provide a dummy implementation and on runtime, it would access the C++ library with the actual implementation?
Perhaps you could customize your GCC compiler (e.g. with a MELT extension) to produce somehow a C interface to your library (perhaps by annotating relevant C++ functions with your own #pragma-s ....)
But your question is too vague to get a more precise answer.
What is your C++ library, how should it be used in C++?
Look at how other C++ libraries interface to C; e.g. look inside the source code of PPL
If you want to use C++ from C, it can be done only in an extremely limited way, and as far as the C++ code is explicitly built for it (that means basically throwing away most for what makes C++ C++). Won't happen in your case.
Your ways out are to just use C++ or get a library for C. Anything else will hurt too much to make any sense.