I am trying to figure out th right way to call this function:
size_t
fz_buffer_storage(fz_context *ctx, fz_buffer *buf, unsigned char **datap)
{
if (datap)
*datap = (buf ? buf->data : NULL);
return (buf ? buf->len : 0);
}
using CGo to get the underlying string and its length as a byte array in Go.
Is this the right way to do it?
var bufferContents *C.uchar
length := C.fz_buffer_storage(ctx, buf, &bufferContents)
bytes := C.GoBytes(unsafe.Pointer(bufferContents), C.int(length))
Since the C code overwrites *datap, I am not sure if the garbage collector will still do the right thing.
I saw an answer here suggesting something along the lines of
var tempUcharPtr *C.uchar
bufferContents := C.malloc(C.size_t(unsafe.Sizeof(tempUcharPtr)))
defer C.free(bufferContents)
length := C.fz_buffer_storage(ctx, buf, (**C.uchar)(bufferContents))
bytes := C.GoBytes(unsafe.Pointer(*(**C.uchar)(bufferContents)), C.int(length))
which also seems to work, but it's much more convoluted and I'm wondering if it's better / safer than the previous version.
Apparently, the first version is fine. Quoting the docs:
Go code may pass a Go pointer to C provided the Go memory to which it points does not contain any Go pointers.
From what I understand, since var bufferContents *C.uchar will be initialised to nil, it does not count as a "Go pointer" for the above rule. The following simplified code examples confirm this:
package main
// void F(char **p) {}
import "C"
func main() {
var p *C.char = new(C.char)
C.F(&p)
}
will trigger "panic: runtime error: cgo argument has Go pointer to Go pointer"
package main
// void F(char **p) {}
import "C"
func main() {
var p *C.char
C.F(&p)
}
works just fine, even when setting GODEBUG=cgocheck=2.
Thanks to the folks on the #cgo channel on the Gophers Slack community for helping me understand this!
Related
I'm writing an app for the windows platform using FFmpeg and it's golang wrapper goav, but I'm having trouble understanding how to use the C pointers to gain access to the data array they point to.
I'm trying to get the data stored in the AVFrame class and use Go to write it to a file, and eventually a texture in OpenGl to make a video player with cool transitions.
I think understanding how to cast and access the C data will make coding this a lot easier.
I've stripped out all the relevant parts of the C code, the wrapper and my code, shown below:
C code - libavutil/frame.h
#include <stdint.h>
typedef struct AVFrame {
#define AV_NUM_DATA_POINTERS 8
uint8_t *data[AV_NUM_DATA_POINTERS];
}
Golang goav wrapper - I don't really know whats going on here with the unsafe.Pointers and casting but it gives me access to the underlying C code
package avutil
/*
#cgo pkg-config: libavutil
#include <libavutil/frame.h>
#include <stdlib.h>
*/
import "C"
import (
"unsafe"
)
type Frame C.struct_AVFrame
func AvFrameAlloc() *Frame {
return (*Frame)(unsafe.Pointer(C.av_frame_alloc()))
}
func Data(f *Frame) *uint8 {
return (*uint8)(unsafe.Pointer((*C.uint8_t)(unsafe.Pointer(&f.data))))
}
My Golang code
package main
import "github.com/giorgisio/goav/avutil"
func main() {
videoFrame := avutil.AvFrameAlloc()
data := avutil.Data(videoFrame)
fmt.Println(data) // here i want the values from data[0] to data[7], but how?
}
Since the library author did not construct a slice header for you to work with you will instead need to cast the return value you get to an unsafe.Pointer and then to a uintptr this will allow you to perform pointer arithmetic on it to get elements later in memory.
Here's some example code that should run as-is on the go playground.
package main
import (
"fmt"
"unsafe"
)
func main() {
nums := []uint8{1, 2, 3, 4, 5, 6, 7, 8}
val := &nums[0] // val is the equivalent of the *uint8 the Data function returns
ptr := unsafe.Pointer(val)
sixthVal := (*uint8)(unsafe.Pointer(uintptr(ptr) + 5*unsafe.Sizeof(*val)))
fmt.Println("Sixth element:", *sixthVal)
}
Of course, you will need to be very certain you know how many elements there are so that you do not access invalid memory.
I'm currently writing a Go wrapper around a C library. That C library uses opaque struct pointers to hide information across the interface. However, the underlying implementation stores size_t values in there. This leads to runtime errors in the resulting program.
A minimum working example to reproduce the problem looks like this:
main.go:
package main
/*
#include "stddef.h"
// Create an opaque type to hide the details of the underlying data structure.
typedef struct HandlePrivate *Handle;
// In reality, the implementation uses a type derived from size_t for the Handle.
Handle getInvalidPointer() {
size_t actualHandle = 1;
return (Handle) actualHandle;
}
*/
import "C"
// Create a temporary slice containing invalid pointers.
// The idea is that the local variable slice can be garbage collected at the end of the function call.
// When the slice is scanned for linked objects, the GC comes across the invalid pointers.
func getTempSlice() {
slice := make([]C.Handle, 1000000)
for i, _ := range slice {
slice[i] = C.getInvalidPointer()
}
}
func main() {
getTempSlice()
}
Running this program will lead to the following error
runtime: writebarrierptr *0xc42006c000 = 0x1
fatal error: bad pointer in write barrier
[...stack trace omitted...]
Note that the errors disappear when the GC is disabled by setting the environment variable GOGC=off.
My question is which is the best way to solve or work around this problem. The library stores integer values in pointers for the sake of information hiding and this seems to confuse the GC. For obvious reasons I don't want to start messing with the library itself but rather absorb this behaviour in my wrapping layer.
My environment is Ubuntu 16.04, with gcc 5.4.0 and Go 1.9.2.
Documentation of cgo
I can reproduce the error for go1.8.5 and go1.9.2. I cannot reproduce the error for tip: devel +f01b928 Sat Nov 11 06:17:48 2017 +0000 (effectively go1.10alpha).
// Create a temporary slice containing invalid pointers.
// The idea is that the local variable slice can be garbage collected at the end of the function call.
// When the slice is scanned for linked objects, the GC comes across the invalid pointers.
A Go mantra is do not ignore errors. However, you seem to assume that that the GC will gracefully ignore errors. The GC should complain loudly (go1.8.5 and go1.9.2). At worst, with undefined behavior that may vary from release to release, the GC may appear to ignore errors (go devel).
The Go compiler sees a pointer and the Go runtime GC expects a valid pointer.
// go tool cgo
// type _Ctype_Handle *_Ctype_struct_HandlePrivate
// var handle _Ctype_Handle
var handle C.Handle
// main._Ctype_Handle <nil> 0x0
fmt.Fprintf(os.Stderr, "%[1]T %[1]v %[1]p\n", handle)
slice := make([]C.Handle, 1000000)
for i, _ := range slice {
slice[i] = C.getInvalidPointer()
}
Use type uintptr. For example,
package main
import "unsafe"
/*
#include "stddef.h"
// Create an opaque type to hide the details of the underlying data structure.
typedef struct HandlePrivate *Handle;
// In reality, the implementation uses a type derived from size_t for the Handle.
Handle getInvalidPointer() {
size_t actualHandle = 1;
return (Handle) actualHandle;
}
*/
import "C"
// Create a temporary slice of C pointers as Go integer type uintptr.
func getTempSlice() {
slice := make([]uintptr, 1000000)
for i, _ := range slice {
slice[i] = uintptr(unsafe.Pointer(C.getInvalidPointer()))
}
}
func main() {
getTempSlice()
}
I'm trying to call a C function from Go with cgo to read an error message. The function produces a message of an unknown length less than 256 bytes.
Working example in C:
char message[ERROR_SIZE]; //256
last_error( message, sizeof(message) );
printf( "message: %s\n", message );
My attempt in Go (not working):
var ptr *C.char
C.last_error(ptr, ERROR_SIZE)
var message = C.GoString(ptr)
fmt.Printf("message: %s\n", message)
When the go code is run, the message is empty. Does the go version need to preallocate space for the message? How to do this?
Update after comment by LPs to pass an array. This works, but seems a bit awkward:
var buf [ERROR_SIZE]byte
var ptr = (*C.char)(unsafe.Pointer(&buf[0]))
C.last_error(ptr, len(buf))
var message = C.GoString(ptr)
fmt.Printf("message: %s\n", message)
Is there a simpler way?
In your first example you are passing a nil pointer, so there is no allocated memory for C.last_error to write the output to (and luckily, it appears to just do nothing).
You need to allocate the memory somehow, and the most straightforward way to do that in Go is to use a slice, rather than create an array with a static size.
buf := make([]byte, ERROR_SIZE)
C.last_error((*C.char)(unsafe.Pointer(&buf[0])), len(buf))
// While C.GoString will find the terminating null if it's there,
// there's no reason to copy the string in C, and allocate another slice.
if i := bytes.IndexByte(buf, 0); i >= 0 {
buf = buf[:i]
}
fmt.Printf("message: %s\n", buf)
Trying out Google protocol buffers for my code in C language.
messagefile.proto
===================
mesage othermessage
{
optional string otherstring = 1;
}
message onemessage
{
optional string messagestring = 1;
optional int32 aninteger = 2;
optional othermessage otr_message= 3;
}
==============================================
--> protoc-c messagefile.proto --c_out=./
this resulted in two files
--> messagefile.pb-c.c and messagefile.pb-c.h
Now my code file which would try to use the
simpleexample.c
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>
#include "messagefile.pb-c.h"
#include <stdbool.h>
int main(int argc, const char * argv[])
{
onemessage msg = ONE__MESSAGE__INIT; //from generated .h code file
void *buf;
unsigned int len;
char *ptr;
//integer initialization
msg.has_aninteger = true;
msg.aninteger = 1;
//accessing the string in onemessage
msg.messagestring = malloc(sizeof("a simple string"));
strncpy(msg.messagestring,"a simple string",strlen("a simple string"));
//trying to initialize the string in the nested structure othermessage
msg.otr_message = malloc(sizeof(othermessage));
msg.otr_message->otherstring = malloc(sizeof("a not so simple string"));
strncpy(msg.otr_message->otherstring,"a not so simple string",strlen("a not so simple string"));
//lets find the length of the packed structure
len = one_message__get_packed_size(&msg); //from generated .h code
//lets arrange for as much size as len
buf = malloc(len);
//lets get the serialized structure in buf
one_message__pack_to_buffer(&msg,buf); //from generated code
//write it to a stream, for now the screen
fwrite(buf,len,1,stdout);
//free buffer
free(buf);
return 0;
}
I compile it as gcc -o testout messagefile.pb-c.c simpleexample.c -lprotobuf-c
The Problem I am facing is when trying to initialize the nested othermessage variables and then call the get_packed_size it throws a segmentation fault.
I tried various combinations and I can say that whenever having strings in a nested class, I am facing problem to access those using google protoc.
Am i missing something? Is there anything wrong.
Can anyone please help.
note:There might be a few general syntax errors please ignore them.
ThankYou.
note:There might be a few general syntax errors please ignore them.
Err... they are kinda hard to ignore since your code does not compile :-)
Anyway, apart from the syntax errors, you need to make several corrections to your code. In order to use the field otr_message, it is not sufficient to just malloc() it. You also need to initialize it so the headers in the message get the right values. This is done with init(), like this:
//trying to initialize the string in the nested structure othermessage
msg.otr_message = malloc(sizeof(othermessage));
othermessage__init(msg.otr_message);
Then you use the wrong function to do the packing to your own array. As explained here, you need to use pack() as opposed to pack_to_buffer(), like this:
//lets get the serialized structure in buf
onemessage__pack(&msg,buf); //from generated code
Finally, your strncpy() invocations have a mistake. The length calculated with strlen() does not include the null terminator, which you do need. So you need to take strlen()+1 or use sizeof(), like this:
strncpy(msg.messagestring,"a simple string",sizeof("a simple string"));
After making those changes, the example worked for me:
$ ./testout
a simple string
a not so simple string
I would like to be able to use my own memory allocation function for certain data structures (real valued vectors and arrays) in R. The reason for this is that I need my data to be 64bit aligned and I would like to use the numa library for having control over which memory node is used (I'm working on compute nodes with four 12-core AMD Opteron 6174 CPUs).
Now I have two functions for allocating and freeing memory: numa_alloc_onnode and numa_free (courtesy of this thread). I'm using R version 3.1.1, so I have access to the function allocVector3 (src/main/memory.c), which seems to me as the intended way of adding a custom memory allocator. I also found the struct R_allocator in src/include/R_ext
However it is not clear to me how to put these pieces together. Let's say, in R, I want the result res of an evaluation such as
res <- Y - mean(Y)
to be saved in a memory area allocated with my own function, how would I do this? Can I integrate allocVector3 directly at the R level? I assume I have to go through the R-C interface. As far as I know, I cannot just return a pointer to the allocated area, but have to pass the result as an argument. So in R I call something like
n <- length(Y)
res <- numeric(length=1)
.Call("R_allocate_using_myalloc", n, res)
res <- Y - mean(Y)
and in C
#include <R.h>
#include <Rinternals.h>
#include <numa.h>
SEXP R_allocate_using_myalloc(SEXP R_n, SEXP R_res){
PROTECT(R_n = coerceVector(R_n, INTSXP));
PROTECT(R_res = coerceVector(R_res, REALSXP));
int *restrict n = INTEGER(R_n);
R_allocator_t myAllocator;
myAllocator.mem_alloc = numa_alloc_onnode;
myAllocator.mem_free = numa_free;
myAllocator.res = NULL;
myAllocator.data = ???;
R_res = allocVector3(REALSXP, n, myAllocator);
UNPROTECT(2);
}
Unfortunately I cannot get beyond a variable has incomplete type 'R_allocator_t' compilation error (I had to remove the .data line since I have no clue as to what I should put there). Does any of the above code make sense? Is there an easier way of achieving what I want to? It seems a bit odd to have to allocate a small vector in R and the change its location in C just to be able to both control the memory allocation and have the vector available in R...
I'm trying to avoid using Rcpp, as I'm modifying a fairly large package and do not want to convert all C calls and thought that mixing different C interfaces could perform sub-optimally.
Any help is greatly appreciated.
I made some progress in solving my problem and I would like to share in case anyone else encounters a similar situation. Thanks to Kevin for his comment. I was missing the include statement he mentions. Unfortunately this was only one among many problems.
dyn.load("myAlloc.so")
size <- 3e9
myBigmat <- .Call("myAllocC", size)
print(object.size(myBigmat), units = "auto")
rm(myBigmat)
#include <R.h>
#include <Rinternals.h>
#include <R_ext/Rallocators.h>
#include <numa.h>
typedef struct allocator_data {
size_t size;
} allocator_data;
void* my_alloc(R_allocator_t *allocator, size_t size) {
((allocator_data*)allocator->data)->size = size;
return (void*) numa_alloc_local(size);
}
void my_free(R_allocator_t *allocator, void * addr) {
size_t size = ((allocator_data*)allocator->data)->size;
numa_free(addr, size);
}
SEXP myAllocC(SEXP a) {
allocator_data* my_allocator_data = malloc(sizeof(allocator_data));
my_allocator_data->size = 0;
R_allocator_t* my_allocator = malloc(sizeof(R_allocator_t));
my_allocator->mem_alloc = &my_alloc;
my_allocator->mem_free = &my_free;
my_allocator->res = NULL;
my_allocator->data = my_allocator_data;
R_xlen_t n = asReal(a);
SEXP result = PROTECT(allocVector3(REALSXP, n, my_allocator));
UNPROTECT(1);
return result;
}
For compiling the c code, I use R CMD SHLIB -std=c99 -L/usr/lib64 -lnuma myAlloc.c. As far as I can tell, this works fine. If anyone has improvements/corrections to offer, I'd be happy to include them.
One requirement from the original question that remains unresolved is the alignment issue. The block of memory returned by numa_alloc_local is correctly aligned, but other fields of the new VECTOR_SEXPREC (eg. the sxpinfo_struct header) push back the start of the data array. Is it somehow possible to align this starting point (the address returned by REAL())?
R has, in memory.c:
main/memory.c
84:#include <R_ext/Rallocators.h> /* for R_allocator_t structure */
so I think you need to include that header as well to get the custom allocator (RInternals.h merely declares it, without defining the struct or including that header)