How do you allocate an array in Go with a run-time size?
The following code is illegal:
n := 1
var a [n]int
you get the message prog.go:12: invalid array bound n (or similar), whereas this works fine:
const n = 1
var a [n]int
The trouble is, I might not know the size of the array I want until run-time.
(By the way, I first looked in the question How to implement resizable arrays in Go for an answer, but that is a different question.)
The answer is you don't allocate an array directly, you get Go to allocate one for you when creating a slice.
The built-in function make([]T, length, capacity) creates a slice and the array behind it, and there is no (silly) compile-time-constant-restriction on the values of length and capacity. As it says in the Go language specification:
A slice created with make always allocates a new, hidden array to which the returned slice value refers.
So we can write:
n := 12
s := make([]int, n, 2*n)
and have an array allocated size 2*n, with s a slice initialised to be the first half of it.
I'm not sure why Go doesn't allocate the array [n]int directly, given that you can do it indirectly, but the answer is clear: "In Go, use slices rather than arrays (most of the time)."
Related
Say I have a class field
a : array of array of double;
it is allocated using nested SetLength calls.
SetLength(a,100);
for i := 0 to length(a)-1 do SetLength( a[i], 100 );
On object Destroy is it necessary to loop through the first level of array to free the second level items of it?
for i := 0 to length(a)-1 do a[i] := NIL;
Is it necessary or the compiler handles freeing of multi dimensional dynamic arrays too?
Dynamic arrays are managed by the compiler. When a dynamic array's reference count drops to zero, it is automatically freed.
This is equally true for multidimensional dynamic arrays of any dimension, at every level.
So when your field a goes out of scope, this dynamic array's reference count is reduced by one. If the new reference count is zero, the array is freed, and so the reference counts of all elements a[0], a[1], ..., a[High(a)] are reduced by one. And, again, if they reach zero, they are freed too.
You don't need to do anything.
You can set the size of a multi-dimensional dynamic array with a single SetLength() call. No need to use any loop.
You do this by simply calling
SetLength(DynamicArray, FirstDimensionSize, SecondDimensionSize, ..NthDimensionSize).
So, in your case, you could set the initial size of your dynamic array by simply using:
SetLength(a,100,100);
And then when freeing up the array, you just call:
SetLength(a,0,0);
If I implement a queue like this...
package main
import(
"fmt"
)
func PopFront(q *[]string) string {
r := (*q)[0]
*q = (*q)[1:len(*q)]
return r
}
func PushBack(q *[]string, a string) {
*q = append(*q, a)
}
func main() {
q := make([]string, 0)
PushBack(&q, "A")
fmt.Println(q)
PushBack(&q, "B")
fmt.Println(q)
PushBack(&q, "C")
fmt.Println(q)
PopFront(&q)
fmt.Println(q)
PopFront(&q)
fmt.Println(q)
}
... I end up with an array ["A", "B", "C"] that has no slices pointing to the first two elements. Since the "start" pointer of a slice can never be decremented (AFAIK), those elements can never be accessed.
Is Go's garbage collector smart enough to free them?
Slices are just descriptors (small struct-like data structures) which if not referenced will be garbage collected properly.
The underlying array for a slice (to which the descriptor points to) on the other hand is shared between all slices that are created by reslicing it: quoting from the Go Language Specification: Slice Types:
A slice, once initialized, is always associated with an underlying array that holds its elements. A slice therefore shares storage with its array and with other slices of the same array; by contrast, distinct arrays always represent distinct storage.
Therefore if at least one slice exists, or a variable holding the array (if a slice was created by slicing the array), it will not be garbage collected.
Official Statement about this:
The blog post Go Slices: usage and internals By Andrew Gerrand clearly states this behaviour:
As mentioned earlier, re-slicing a slice doesn't make a copy of the underlying array. The full array will be kept in memory until it is no longer referenced. Occasionally this can cause the program to hold all the data in memory when only a small piece of it is needed.
...
Since the slice references the original array, as long as the slice is kept around the garbage collector can't release the array.
Back to your example
While the underlying array will not be freed, note that if you add new elements to the queue, the built-in append function occasionally might allocate a new array and copy the current elements to the new – but copying will only copy the elements of the slice and not the whole underlying array! When such a reallocation and copying occurs, the "old" array may be garbage collected if no other reference exists to it.
Also another very important thing is that if an element is popped from the front, the slice will be resliced and not contain a reference to the popped element, but since the underlying array still contains that value, the value will also remain in memory (not just the array). It is recommended that whenever an element is popped or removed from your queue (slice/array), always zero it (its respective element in the slice) so the value will not remain in memory needlessly. This becomes even more critical if your slice contains pointers to big data structures.
func PopFront(q *[]string) string {
r := (*q)[0]
(*q)[0] = "" // Always zero the removed element!
*q = (*q)[1:len(*q)]
return r
}
This is mentioned Slice Tricks wiki page:
Delete without preserving order
a[i] = a[len(a)-1]
a = a[:len(a)-1]
NOTE If the type of the element is a pointer or a struct with pointer fields, which need to be garbage collected, the above implementations of Cut and Delete have a potential memory leak problem: some elements with values are still referenced by slice a and thus can not be collected.
No. At the time of this writing, the Go garbage collector (GC) is not smart enough to collect the beginning of an underlying array in a slice, even if it is inaccessible.
As mentioned by others here, a slice (under the hood) is a struct of exactly three things: a pointer to its underlying array, the length of the slice (values accessible without reslicing), and the capacity of the slice (values accessible by reslicing). On the Go blog, slice internals are discussed at length. Here is another article I like about Go memory layouts.
When you reslice and cut off the tail end of a slice, it is obvious (upon understanding the internals) that the underlying array, the pointer to the underlying array, and the slice's capacity are all left unchanged; only the slice length field is updated. When you re-slice and cut off the beginning of a slice, you are really changing the pointer to the underlying array along with the length and capacity. In this case, it is generally unclear (based on my readings) why the GC does not clean up this inaccessible part of the underlying array because you cannot re-slice the array to access it again. My assumption is that the underlying array is treated as one block of memory from the GC's point of view. If you can point to any part of the underlying array, the entire thing is ineligible for deallocation.
I know what you're thinking... like the true computer scientist you are, you may want some proof. I'll indulge you:
https://goplay.space/#tDBQs1DfE2B
As mentioned by others and as shown in the sample code, using append can cause a reallocation and copy of the underlying array, which allows the old underlying array to be garbage collected.
Simple question, simple answer: No. (But if you keep pushing the slice will at some point overflow its underlying array then the unused elements become available to be freed.)
Contrary to what I'm reading, Golang certainly seems to garbage collect at least unused slices starting sections. The following test case provides evidence.
In the first case the slice is set to slice[:1] in each iteration. In the comparison case, it skips that step.
The second case dwarfs the memory consumed in the first case. But why?
func TestArrayShiftMem(t *testing.T) {
slice := [][1024]byte{}
mem := runtime.MemStats{}
mem2 := runtime.MemStats{}
runtime.GC()
runtime.ReadMemStats(&mem)
for i := 0; i < 1024*1024*1024*1024; i++ {
slice = append(slice, [1024]byte{})
slice = slice[1:]
runtime.GC()
if i%(1024) == 0 {
runtime.ReadMemStats(&mem2)
fmt.Println(mem2.HeapInuse - mem.HeapInuse)
fmt.Println(mem2.StackInuse - mem.StackInuse)
fmt.Println(mem2.HeapAlloc - mem.HeapAlloc)
}
}
}
func TestArrayShiftMem3(t *testing.T) {
slice := [][1024]byte{}
mem := runtime.MemStats{}
mem2 := runtime.MemStats{}
runtime.GC()
runtime.ReadMemStats(&mem)
for i := 0; i < 1024*1024*1024*1024; i++ {
slice = append(slice, [1024]byte{})
// slice = slice[1:]
runtime.GC()
if i%(1024) == 0 {
runtime.ReadMemStats(&mem2)
fmt.Println(mem2.HeapInuse - mem.HeapInuse)
fmt.Println(mem2.StackInuse - mem.StackInuse)
fmt.Println(mem2.HeapAlloc - mem.HeapAlloc)
}
}
}
Output Test1:
go test -run=.Mem -v .
...
0
393216
21472
^CFAIL github.com/ds0nt/cs-mind-grind/arrays 1.931s
Output Test3:
go test -run=.Mem3 -v .
...
19193856
393216
19213888
^CFAIL github.com/ds0nt/cs-mind-grind/arrays 2.175s
If you disable garbage collection on the first test, indeed memory skyrockets. The resulting code looks like this:
func TestArrayShiftMem2(t *testing.T) {
debug.SetGCPercent(-1)
slice := [][1024]byte{}
mem := runtime.MemStats{}
mem2 := runtime.MemStats{}
runtime.GC()
runtime.ReadMemStats(&mem)
// 1kb per
for i := 0; i < 1024*1024*1024*1024; i++ {
slice = append(slice, [1024]byte{})
slice = slice[1:]
// runtime.GC()
if i%(1024) == 0 {
fmt.Println("len, cap:", len(slice), cap(slice))
runtime.ReadMemStats(&mem2)
fmt.Println(mem2.HeapInuse - mem.HeapInuse)
fmt.Println(mem2.StackInuse - mem.StackInuse)
fmt.Println(mem2.HeapAlloc - mem.HeapAlloc)
}
}
}
I read somewhere that Ada allows a function only to return a single item. Since an array can hold multiple items does this mean that I can return the array as a whole or must I return only a single index of the array?
Yes, an Ada function can return an array - or a record.
There can be a knack to using it, though. For example, if you are assigning the return value to a variable, the variable must be exactly the right size to hold the array, and there are two common ways of achieving that.
1) Fixed size array - cleanest way is to define an array type, e.g.
type Vector is new Array(1..3) of Integer;
function Unit_Vector return Vector;
A : Vector;
begin
A := Unit_Vector;
...
2) Unconstrained array variables.
These are arrays whose size is determined at runtime by the initial assignment to them. Subsequent assignments to them will fail unless the new value happens to have the same size as the old. The trick is to use a declare block - a new scope - so that each assignment to the unconstrained variable is its first assignment. For example:
for i in 1 .. last_file loop
declare
text : String := Read_File(Filename(i));
-- the size of "text" is determined by the file contents
begin
-- process the text here.
for j in text'range loop
if text(j) = '*' then
...
end loop;
end
end loop;
One warning : if the array size is tens of megabytes or more, it may not be successfully allocated on the stack. So if this construct raises Storage_Error exceptions, and you can't raise the stack size, you may need to use access types, heap allocation via "new" and deallocation as required.
Yes, an Ada function can return an array. For example, an Ada String is "A one-dimensional array type whose component type is a character type." Several of the functions defined in Ada.Strings.Fixed—including Insert, Delete, Head, Tail and Trim—return a String.
Is there a way we can copy every element from one multidimensional array to another multidimensional array by just doing one memcpy operation?
int array1[4][4][4][4];
int array2[4][4][4][4];
int main()
{
memset(&array1,1,sizeof(array1));
memset(&array2,0,sizeof(array2));
printf_all("value in array2 %d \n",array2[1][1][1][1]);
memcpy(&array2,&array1,sizeof(array2));
printf("memcopied in array2 from array1 \n");
printf("value in array2 %d \n",array2[1][1][1][1]); //not printing 1
}
Your code is correct. You should not expect the output to show you a value of 1. You should expect it to show you a value of 16843009, assuming a 4 byte int.
The reason is: you are filling array1 with bytes of value 1, not with ints of value 1. i.e. binary 00000001000000010000000100000001 (0x01010101) is being filled into all the int elements with your memset operation.
So regardless of the size of int on your machine (unless it's a single byte!) you should not expect to see the value 1.
I hope this helps.
Yes, your code should already be correct.
You have to consider memory layout when doing this. The arrays are all in one block, multi dimensional is essentially a math trick done by the compiler.
Your code says copy this memory content to the other memory block. since both share the same layout they will contain the same values.
The following code also just copies the values, but access is handled differently so you would have to think about how to get the order of elements correct.
int array1[4][4][4][4]; //elements 256
int array2[256];
int main()
{
memcpy(&array2,&array1,sizeof(array1)); //will also copy
// original access via: a + 4 * b + 16 * c + 64 * d
}
Multidimensional array in C is a flat block of memory with no internal structure. Memory layout of a multidimensional array is exactly the same as that of a 1-dimensional array of the same total size. The multidimensional interface is implemented through simple index recalculation. You can always memcpy the whole multidimensional array exactly as you do it in your code.
This, of course, only applies to built-in multidimensional arrays, explicitly declared as such (as in your code sample). If you implement a hand-made multidimensional array as an array of pointers to sub-arrays, that data structure will not be copyable in one shot with memcpy.
However, apparently you have some misconceptions about how memset works. Your memset(&array1,1,sizeof(array1)); will not fill the array with 1s, meaning that your code is not supposed to print 1 regardless of which array you print. memset interprets target memory as an array of chars, not as an array of ints.
memset can be used to set memory to zero. As for non-zero values, memset is generally unsuitable for initializing arrays of any type other than char.
I am trying to interface with a Windows dll using Go. The dll function I want to use accepts a pointer to a byte array. Therefore I need to give it that byte array.
I am using the syscall libary to call the dll, as demonstrated here. My basic requirements are:
I am given the required size for the byte array
I create the byte array
I must get a pointer to the byte array
I then pass the pointer to the Windows dll
I can't figure out how to create a byte array in go, and get a pointer to it. This is obviously an unsafe operation, and the unsafe library can be helpful, but I need to create a dynamic-length byte array in the first place. Creating a slice with "make" doesn't help me, unless I can get a pointer to the slice's backing array.
Has anyone else encountered this or have any ideas?
I think syscall.ComputerName implementation https://golang.org/src/syscall/syscall_windows.go#395 would be a good example. It uses uint16s, not bytes, but otherwise ...
In your case it would be ptr := &myslice[0].
Alex
Well I found one gross solution. Apparently the structure of a slice contains a pointer to the backing byte array, the length of the backing byte array, and then the capacity of the backing byte array.
I am only interested in a pointer to the byte array, so I only need the first member of the slice's internal data.
Go's unsafe.Pointer will not cast a slice to an unsafe pointer, but it will cast a pointer to a slice as an unsafe pointer. Since I can cast an unsafe pointer to any old type of pointer I want, I can cast it to a pointer-to-a-pointer, which recovers the first member of the slice's internal data.
Here's a working example. I wanted a uintptr but you could cast it to any pointer type.
package main
import (
"fmt"
"unsafe"
)
func main() {
// Arbitrary size
n := 4
// Create a slice of the correct size
m := make([]int, n)
// Use convoluted indirection to cast the first few bytes of the slice
// to an unsafe uintptr
mPtr := *(*uintptr)(unsafe.Pointer(&m))
// Check it worked
m[0] = 987
// (we have to recast the uintptr to a *int to examine it)
fmt.Println(m[0], *(*int)(unsafe.Pointer(mPtr)))
}
If you wanted a *int instead, you could do mPtr := *(**int)(unsafe.Pointer(&m))
This works as long as a slice maintains this internal data structure. I am definitely open to a more robust solution that doesn't depend on the structure of Go's internals.