I am trying to interface with a Windows dll using Go. The dll function I want to use accepts a pointer to a byte array. Therefore I need to give it that byte array.
I am using the syscall libary to call the dll, as demonstrated here. My basic requirements are:
I am given the required size for the byte array
I create the byte array
I must get a pointer to the byte array
I then pass the pointer to the Windows dll
I can't figure out how to create a byte array in go, and get a pointer to it. This is obviously an unsafe operation, and the unsafe library can be helpful, but I need to create a dynamic-length byte array in the first place. Creating a slice with "make" doesn't help me, unless I can get a pointer to the slice's backing array.
Has anyone else encountered this or have any ideas?
I think syscall.ComputerName implementation https://golang.org/src/syscall/syscall_windows.go#395 would be a good example. It uses uint16s, not bytes, but otherwise ...
In your case it would be ptr := &myslice[0].
Alex
Well I found one gross solution. Apparently the structure of a slice contains a pointer to the backing byte array, the length of the backing byte array, and then the capacity of the backing byte array.
I am only interested in a pointer to the byte array, so I only need the first member of the slice's internal data.
Go's unsafe.Pointer will not cast a slice to an unsafe pointer, but it will cast a pointer to a slice as an unsafe pointer. Since I can cast an unsafe pointer to any old type of pointer I want, I can cast it to a pointer-to-a-pointer, which recovers the first member of the slice's internal data.
Here's a working example. I wanted a uintptr but you could cast it to any pointer type.
package main
import (
"fmt"
"unsafe"
)
func main() {
// Arbitrary size
n := 4
// Create a slice of the correct size
m := make([]int, n)
// Use convoluted indirection to cast the first few bytes of the slice
// to an unsafe uintptr
mPtr := *(*uintptr)(unsafe.Pointer(&m))
// Check it worked
m[0] = 987
// (we have to recast the uintptr to a *int to examine it)
fmt.Println(m[0], *(*int)(unsafe.Pointer(mPtr)))
}
If you wanted a *int instead, you could do mPtr := *(**int)(unsafe.Pointer(&m))
This works as long as a slice maintains this internal data structure. I am definitely open to a more robust solution that doesn't depend on the structure of Go's internals.
Related
I am writing a program that has an array which size I know, it is a fixed instruction set. Each entry in the array maps to an struct of the opcode and some metadata, such as the function that implements the opcode and its name, a String.
I need to allocate the array before I actually compute and fill the instruction set on each opcode.
Rust won't let me allocate such array statically, even if I know the size and when the initialized state does NOT require ANY pointers, as I would like all strings to be empty, which means no allocations and you could have a "zero value" of the string that is preferably fine to be copied.
Any suggestions? Is there anyway to do it?
Seems Rust really likes to you use a Vec for everything? Even when static memory, which should be preferable on a non GC program language, would have been possible in many cases such as this? This is confusing.
From the rust reference:
If the length operand has a value greater than 1 then this requires that the type of the repeat operand is Copy or that it must be a path to a constant item.
Therefore, to initialize an array using [VALUE; NUM] syntax, either the type implements the Copy trait or that VALUE is a const.
As pointed out by Chayim Friedman, you can use code like below:
const EMPTY: Instruction = Instruction { name: String::new() };
static ARRAY: [Instruction; 5] = [EMPTY; 5];
However, if you are working with a fixed set of data, it might be easier to just populate the list with predefined names, using &'static str.
If there is a need to create an array from a value that cannot be made to a constant and is not Copy, you can use std::array::from_fn like so:
let value = String::new();
let array: [_; 6] = std::array::from_fn(|_| value.clone());
I am trying to initialize a struct of C array in go side.
I am new to cgo. Still trying to understand the use case.
test.h
typedef struct reply {
char *name;
reply_cb callback_fn;
} reply_t;
typedef struct common {
char *name;
int count;
reply_t reply[];
} common_t;
int
init_s (common_t *service);
test.go
name := C.CString("ABCD")
defer C.free(unsafe.Pointer(name))
num := C.int(3)
r := [3]C.reply_t{{C.CString("AB"), (C.s_cb)(unsafe.Pointer(C.g_cb))},
{C.CString("BC"), (C.s_cb)(unsafe.Pointer(C.g_cb))},
{C.CString("CD"), (C.s_cb)(unsafe.Pointer(C.g_cb))}}
g := C.common_t{
name: name,
count: num,
reply : r,
}
rc := C.init_s(&g)
I am getting error on "reply: r" unknown field 'r' in struct literal of type
Any help will be appreciated. The goal is initialize and then use it values in C init_s for processing.
You cannot use a flexible array field from Go: https://go-review.googlesource.com/c/go/+/12864/.
I think the reasonong is simple: this wart of C normally requires you to perform a trick of allocating a properly-aligned memory buffer long enough to accomodate for the sizeof(struct_type) itself at the beginning of that buffer plus sizeof(array_member[0]) * array_element_count bytes. This does not map to Go's type system because in it, structs have fixed size known at compile time. If Go would not hide reply from the definition, it would refer to a zero-length field you cannot do anything useful with anyway—see #20275.
Don't be deceived by code examples where a flexible array member field is initialized with a literal: as torek pointed out, it's a GCC extension, but what is more important, it requires work on part of the compiler—that is, it analyzes the literal, understands the context it appeared in and generates a code which allocates large enough memory block to accomodate both the struct and all the members of the flexible array.
The initialization of the array in your Go code may look superficially similar but it has an important difference: it allocates a separate array which has nothing to do with the memory block of the struct it's supposed to be "put into".
What's more Go's array are different beasts than C's: in C, arrays are pointers in disguise, in Go, arrays are first-class citizens and when you assign an array or pass it to a function call, the whole array is copied by value—as opposed to "decaying into a pointer"—in C's terms.
So even if the Go compiler would not hide the reply field, assignment to it would fail.
I think you cannot directly use values of this type from Go without additional helper code written in C. For instance, to initialize values of common_t, you would write a C helper which would first allocate a memory buffer long enough and then expose to the Go code a pair of pointers: to the beginning of the buffer (of type *C.common_t), and to the first element of the array—as *C.reply_t.
If this C code is the code you own, I'd recommend to just get rid of the flexible array and maintain a pointer to a "normal" array in the reply field.
Yes, this would mean extra pointer chasing for the CPU but it will be simpler to interoperate with Go.
I am attempting to return a dynamically declared array from a function; thus far I am returning a structure to hold a pointer to the memory block that malloc() assigned for the array AND an integer to store the length of the array.
This made me wonder; How does the C Compiler(or whatever) handle an automatic array declared in a program?
eg.
main()
{
//delcare an array holding 3 elements
int array[] = {1,2,3};
/*variable to hold length of array
*size of array / size of 1st element in the array == length of the array
*this will == 3
*/
int array_Length = (sizeof(array))/(sizeof(*array));
//call malloc for a block of memory to hold 3 integers(worth of memory)
int* ptr = malloc(3*(sizeof(int)));
/*not exactly sure what this formula means when using a pointer???
*but it seems to always == 1
*/
int dynamic_array_length = (sizeof(ptr))/(sizeof(*ptr));
return 0;
}
My point is, the sizeof() operator somehow knows that the automatically declared array has 3 integers within it.
Or more generally:
sizeof(array)
where array is (N x type_size)
N is the number of elements within the array
type_size is the number of bytes of memory used to store the data type
Are automatic arrays stored with additional information about their size/length?
Are dynamic arrays stored differently? (I know that we control when a dynamic variable is freed from memory)
Operator sizeof is a compile-time construct (with the exception of VLA arguments). It tells you the object size in bytes because it knows the exact compile-time object type. And when you know the exact type the size is also immediately known. There's no need to separately store the number of elements anywhere.
Your declaration
int array[] = {1,2,3};
is equivalent to
int array[3] = {1,2,3};
meaning that array has type int[3]. So your sizeof(array) is interpreted as sizeof(int[3]), which is immediately known to the compiler.
sizeof does not know and does not care about any "dynamic arrays" of yours. All it cares about is that in sizeof(ptr) operator sizeof is applied to a pointer. So it evaluates to pointer size.
sizeof(...) is not a function call. It doesn't actually execute at runtime - that value is replaced at the compile time, so what's actually compiled is:
int array_length = 3;
The calculation of dynamic_array_length is incorrect. You divide the size of a pointer by the size of int. Which in your case happens to be the same and get 1 as a result.
Your dynamic array is stored differently - the pointer (on the stack) is separate from the data (on the heap). The first array is just data on the stack - the memory address is constant (for that stack frame) and gets used where needed.
Disregarding VLAs, the array size of an automatic array is a fact completely known at compile time, and is actually a part of the type of the variable. sizeof is a query (resolved at compile time) to the type system, which is a thing that exists only in the compiler internal data structures while it is compiling. The result is the actual variable size, which is treated essentially as if it was directly written in the source code.
Things about sizeof has been discussed enough, and we know it is a compile-time action.
But in fact, It is true that there is something about size is stored and used in run-time for dynamic variables. Otherwise the free can not do its work correctly.
Here is a good reference: how-do-malloc-and-free-work
You are hitting one of the inconsistencies of C. An array is just a pointer, except where it has been declared. The result of sizeof (somearray) is different, depending upon whether or not sizeof is used in the function defining somearray. In, C, as soon as you move away from the definition, there is no knowledge of anything other than the type of object in the array. That is just one of the many reasons why C programming is so error prone.
Most programming languages other than C or those derived from C, maintain an array descriptor that includes the number of dimensions, the number of elements, and, in some cases, the array bounds.
In the case of dynamic arrays, the library adds overhead, usually before, to the memory returned (sometimes additional overhead is added at the end). This is used so that library can know how much memory is freed.
I'm writing a library and I want to return an array (or write to an array) of an unspecific type to the caller. The type can vary, depending on who calls - I can, however, create as many objects of said type from within my function. One way would be that the caller creates an array and the callee fills that - however, there is no way of telling how long this array is going to be. (Is there a way that the callee makes the caller's array bigger? Remember, the callee only sees x interface{}...)
The other way which I chose because I don't see how above is possible, is that the caller gives me the pointer of his specific type and I redirect it to the array of objects which I created.
Below is my solution. My question: why is the array empty after the function call? They are pointing to the same array after my operation, they should be the same. Am I overlooking something? I thought about GC, but it couldn't be that fast, could it?
http://play.golang.org/p/oVoPx5Nf84
package main
import "unsafe"
import "reflect"
import "log"
func main() {
var x []string
log.Printf("before: %v, %p", x, x)
manipulate(&x)
log.Printf("after: %v, %p", x, x)
}
func manipulate(target interface{}) {
new := make([]string, 0, 10)
new = append(new, "Hello", "World")
log.Printf("new: %v, %p", new, new)
p := unsafe.Pointer(reflect.ValueOf(target).Pointer())
ptr := unsafe.Pointer(reflect.ValueOf(new).Pointer())
*(*unsafe.Pointer)(p) = ptr
}
First of all, unsafe is usually a bad idea. So is reflection, but unsafe is at least an order of magnitude worse.
Here is your example using pure reflection (http://play.golang.org/p/jTJ6Mhg8q9):
package main
import (
"log"
"reflect"
)
func main() {
var x []string
log.Printf("before: %v, %p", x, x)
manipulate(&x)
log.Printf("after: %v, %p", x, x)
}
func manipulate(target interface{}) {
t := reflect.Indirect(reflect.ValueOf(target))
t.Set(reflect.Append(t, reflect.ValueOf("Hello"), reflect.ValueOf("World")))
}
So, why didn't your unsafe way work? Unsafe is extremely tricky and at times requires understanding the internals. First, some misconceptions you have:
You are using arrays: you are not, you are using slices. Slices are a view of an array. They contain within them a pointer to the data, a length, and a capacity. They are stucts internally and not pure pointers.
Pointer returns the pointer only if it is a pointer: it can actually return a value for many types like a slice.
From http://golang.org/pkg/reflect/#Value.Pointer:
If v's Kind is Slice, the returned pointer is to the first element of the slice. If the slice is nil the returned value is 0. If the slice is empty but non-nil the return value is non-zero.
Arrays are pointers: in Go, arrays are actually values. That means they are copied when passed to other functions or assigned. It also means the .Pointer method wouldn't work.
You are assigning a pointer to an array to a slice type. By luck, the implementation of slices used internally has the data pointer first so you are actually setting the internal array pointer used by the slice. I must stress that is is effectively pure accident. Even still, you are not setting the length and capacity of the slice so it still prints zero elements.
Unsafe lets you do things at such a low level that the actual results aren't really defined. It is best to stay away from it unless you really know what you are doing. Even then, be aware that things can can change and what works today may not work in the next version of Go or another implementation.
How do you allocate an array in Go with a run-time size?
The following code is illegal:
n := 1
var a [n]int
you get the message prog.go:12: invalid array bound n (or similar), whereas this works fine:
const n = 1
var a [n]int
The trouble is, I might not know the size of the array I want until run-time.
(By the way, I first looked in the question How to implement resizable arrays in Go for an answer, but that is a different question.)
The answer is you don't allocate an array directly, you get Go to allocate one for you when creating a slice.
The built-in function make([]T, length, capacity) creates a slice and the array behind it, and there is no (silly) compile-time-constant-restriction on the values of length and capacity. As it says in the Go language specification:
A slice created with make always allocates a new, hidden array to which the returned slice value refers.
So we can write:
n := 12
s := make([]int, n, 2*n)
and have an array allocated size 2*n, with s a slice initialised to be the first half of it.
I'm not sure why Go doesn't allocate the array [n]int directly, given that you can do it indirectly, but the answer is clear: "In Go, use slices rather than arrays (most of the time)."