Add python modules to cython for use in c - c

Question

I am currently trying to write a plugin backend in c by using .so files. Doing this in c works as I expect it to. However I thought about writing python plugins for my backend. Here is when i stumbled upon cython which seems to be very promising.
My backend is calling a function within the .so files and expects a value in return.
This function currently looks like this:
cdef public size_t transform_data(char *plugin_arguments, char **buffer):
printf("Entered function\n")
print("test\n")
printf("Test passed\n")
return 5
The interesting part is, that the printf works just fine. However the print doesn't. I suspect this is because there is some sort of linking error to a python module that I am missing? Also later on I would like to be able to add any python module to that file, for example the influxdb module. A call to influxdb.InfluxDBClient doesn't work either right now, I guess for the same reason that the print is not working.
I am compiling the file using
cythonize -3b some_plugin.pyx
and I have also tried to compile using a setup file that looks like this:
from distutils.core import setup
from Cython.Build import cythonize
setup(ext_modules=cythonize("some_plugin.pyx"))
both resulting to a segfault as soon as I hit the print call.
Here is the code that I am using to call the .so file:
#include "execute_plugin.h"
#include <Python.h>
#include <dlfcn.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
size_t execute_plugin(char file_name[FILE_NAME_SIZE], char *plugin_arguments,
char **output_buffer) {
if (!Py_IsInitialized()) {
Py_SetPythonHome(L"/home/flo/.local/lib/python3.8");
Py_SetPath(L"/usr/lib/python3.8");
Py_Initialize();
}
if (!Py_IsInitialized())
return 0;
void *plugin;
size_t (*func_transform_data)(char *plugin_arguments, char **output_buffer);
char path[PATH_SIZE];
if (!get_path_to_file(path, PATH_SIZE)) {
printf("Could not receive the correct path to the plugin %s\n", file_name);
return 0;
}
plugin = dlopen(path, RTLD_LAZY | RTLD_GLOBAL);
if (!plugin) {
fprintf(stderr, "Error: %s\n", dlerror());
fprintf(stderr, "Cannot load %s\n", file_name);
return 0;
}
func_transform_data =
(size_t(*)(char *plugin_arguments, char **output_buffer))dlsym(
plugin, "transform_data");
if (!func_transform_data) {
fprintf(stderr, "Error: %s\n", dlerror());
dlclose(plugin);
return 0;
}
size_t length = func_transform_data(plugin_arguments, output_buffer);
printf("Size of answer is %ld\n", length);
dlclose(plugin);
Py_Finalize();
return length;
}
I have tried using the documentation and just copied the example: https://cython.readthedocs.io/en/latest/src/tutorial/embedding.html
In this example I didn't use an .so file but the .c and .h file that is also getting generated by the cythonize command. Interestingly enough the print function is working but as soon as I try to add another module like the influxdb module and try to call a function from it I also get errors.
Since I have not found a lot about using cython code in c I am wondering if what I am trying to do is even possible or if there is a better approach.

Answer 1

It's difficult to be certain what your issue is because you don't actually show how you call your Cython function. So what follows is a guess.
Cython does not produce standalone C functions. It produces functions that are part of a Python module and require that module to be initialized. What is probably happening is that Cython caches the lookup to the global print function as part of the module initialization. Since you skip the initialization the cached print function is not set up (hence the crash).
When you read the documentation, you'd have seen that all the examples ("Using Cython declarations from C" and "Embedding Cython modules...") import the module before using it. You must do this. The preferred way to do this is with PyImport_AppendInittab().
A second possibility is that you haven't initialized the Python interpreter. Again, this is not optional.