Pass data from a C function via double pointer in Cython - c

Question

I want to use a small C routine with Cython. The C function itself is
#include <stdio.h>
#include "examples.h"
void add_array(int **io_array, int n) {
int i;
int *array;
array = (int *) malloc(n * sizeof(int));
for(i = 0; i < n; i++) {
array[i] = i;
}
*io_array = array;
}
And the function prototype:
#ifndef EXAMPLES_H
#define EXAMPLES_H
void add_array(int **io_array, int n);
#endif
Now I want to use Cython to interface the C function with:
cdef extern from "examples.h":
void add_array(int **io_array, int n)
import numpy as np
def add(arr):
if not arr.flags['C_CONTIGUOUS']:
arr = np.ascontiguousarray(arr, dtype=np.int32)
cdef int[::1] arr_memview = arr
add_array(&arr_memview[0], arr_memview.shape[0])
return arr
When compiling it gives the error:
pyexamples.pyx:13:14: Cannot assign type 'int *' to 'int **'
What is the right way to interface this function?

Answer 1

It will not work out-of-the-box with numpy-arrays. You will have to make the memory management yourself, for example:
%%cython
from libc.stdlib cimport free
def doit():
cdef int *ptr;
add_array(&ptr, 5)
print(ptr[4])
free(ptr) #memory management
The difference to your attempt: &arr_memview[0] is pointer to an integer array, but what you need for your function is a pointer to a pointer to an integer array - that is what &ptr is.
The problem with your function is, that it has too many responsibilities:
it allocates the memory
it initializes the memory
It would be easier, if add_array would only be doing the second part, i.e.
void add_array(int *io_array, int n) {
int i;
for(i = 0; i < n; i++) {
io_array[i] = i;
}
}
And thus any memory could be initialized (also memory which was not allocated with malloc).
However, it is possible to create a numpy-array using the returned pointer ptr, it is just less straight forward:
cimport numpy as np
import numpy as np
np.import_array() # needed to initialize numpy-data structures
cdef extern from "numpy/arrayobject.h":
void PyArray_ENABLEFLAGS(np.ndarray arr, int flags) #not include in the Cython default include
def doit():
cdef int *ptr;
add_array(&ptr, 5)
# create numpy-array from data:
cdef np.npy_intp dim = 5
cdef np.ndarray[np.int32_t, ndim=1] arr = np.PyArray_SimpleNewFromData(1, &dim, np.NPY_INT32, ptr)
# transfer ownership of the data to the numpy array:
PyArray_ENABLEFLAGS(arr, np.NPY_OWNDATA)
return arr
The following is worth mentioning:
np.import_array() is needed to be able to use all of the numpy's functionality. Here is an example of what can happen, if np.import_array() isn't called.
After PyArray_SimpleNewFromData, the data itself isn't owned by the resulting numpy array, thus we need to enable the OWNDATA-flag, otherwise there will be a memory leak.
It is not obvious, that the resulting numpy-array can be responsible for freeing the data. For example instead of using malloc/free it could be using Python's memory allocator.
I would like to elaborate about point 3. above. Numpy uses a special function to allocate/deallocate memory for data - it is PyDataMem_FREE and uses system's free for it. So in your case (using system's malloc/free in add_array) everything is Ok. (PyDataMem_FREE should not be confused with PyArray_free, as I did in an earlier version of the answer. PyArray_free is responsible for freeing other elements (array itself, and dimension/strides data, not data-memory) of the numpy-array, see here and is different depending on Python version).
A more flexible/safe approach is to use PyArray_SetBaseObject as shown in this SO-post.