I want to implement a method to convert a interface{} slice to a interface{} array which has equal length to the given slice. It's similar to below:
func SliceToArray(in []interface{}) (out interface{}) {
...
}
// out's type is [...]interface{} and len(out)==len(in)
How can I implement this method?
EDIT:
Any possible to use reflect.ArrayOf to implement this?
Use reflect.ArrayOf to create the array type given the slice element type. Use reflect.New to create a value of that type. Use reflect.Copy to copy from the slice to the array.
func SliceToArray(in interface{}) interface{} {
s := reflect.ValueOf(in)
if s.Kind() != reflect.Slice {
panic("not a slice")
}
t := reflect.ArrayOf(s.Len(), s.Type().Elem())
a := reflect.New(t).Elem()
reflect.Copy(a, s)
return a.Interface()
}
Run it on the Playground
This function is useful for creating a map key from a slice and other scenarios where a comparable value is required. Otherwise, it's usually best to use a slice when the length can be arbitrary.
Go 1.20 (Q1 2023) should facilitate such a conversion, after issue 46505: "spec: allow conversion from slice to array"
This is addressed with CL 430415, 428938 (type), 430475 (reflect) and 429315 (spec)
Conversions from slice to array or array pointer
Converting a slice to an array yields an array containing the elements of the underlying array of the slice.
Similarly, converting a slice to an array pointer yields a pointer to the underlying array of the slice.
In both cases, if the length of the slice is less than the length of the array, a run-time panic occurs.
s := make([]byte, 2, 4)
a0 := ([0]byte)(s)
a1 := ([1]byte)(s[1:]) // a1[0] == s[1]
a2 := ([2]byte)(s) // a2[0] == s[0]
a4 := ([4]byte)(s) // panics: len([4]byte) > len(s)
Related
I was reading this article which explains how slices in Go are implemented under the hood:
https://medium.com/swlh/golang-tips-why-pointers-to-slices-are-useful-and-how-ignoring-them-can-lead-to-tricky-bugs-cac90f72e77b
At the end of the article is this snippet of Go code:
func main() {
slice:= make([]string, 1, 3)
func(slice []string){
slice=slice[1:3]
slice[0]="b"
slice[1]="b"
fmt.Print(len(slice))
fmt.Print(slice)
}(slice)
fmt.Print(len(slice))
fmt.Print(slice)
}
My first guess was this would print:
2 [b b]3 [ b b]
In fact it prints:
2[b b]1[]
Which suggests that when anonymous function creates a new local slice by slicing the one passed to it as argument causes a new underlying array to be allocated for the slice. I've confirmed this with this modified version of code:
func main() {
slice := make([]string, 1, 3)
hdr := (*reflect.SliceHeader)(unsafe.Pointer(&slice))
fmt.Printf("adress of underlying array in main: %p\n", unsafe.Pointer(hdr.Data))
func(slice []string) {
hdr := (*reflect.SliceHeader)(unsafe.Pointer(&slice))
fmt.Printf("adress of underlying array in func before slicing: %p\n", unsafe.Pointer(hdr.Data))
slice = slice[1:3]
slice[0] = "b"
slice[1] = "b"
hdr = (*reflect.SliceHeader)(unsafe.Pointer(&slice))
fmt.Printf("adress of underlying array in func after slicing: %p\n", unsafe.Pointer(hdr.Data))
fmt.Print(len(slice))
fmt.Println(slice)
}(slice)
fmt.Print(len(slice))
fmt.Println(slice)
}
Which prints:
adress of underlying array in main: 0xc0000121b0
adress of underlying array in func before slicing: 0xc0000121b0
adress of underlying array in func after slicing: 0xc0000121c0
2[b b]
1[]
My question is: why is the slicing operation in anonymous function causing a new array to be allocated? My understanding was that if in main we are creating a slice with capacity 3, an underlying array with length 3 is created and when anonymous function is manipulating the slice, it is manipulating the same underlying array.
As mkopriva mentioned, the reslicing does not reallocate anything. Reallocation happens only when appending new values would exceed the slice's capacity (source).
The output you got is because the elements you are "able to see" in a slice (including when you print it) depend on its length:
The number of elements is called the length of the slice and is never negative. [...] The length of a slice s can be discovered by the built-in function len;
The original slice constructed with slice := make([]string, 1, 3), has length=1, so when you print it, the output will be the one element at position 0 of the backing array, which is an empty string.
With this code:
slice = slice[1:3]
slice[0] = "b"
slice[1] = "b"
you are effectively mutating the elements at position 1 and 2 of the backing array, none of which is an element of the original slice.
If you reslice the original slice up to capacity — thus extending its length, it will print what you expect:
slice = slice[:cap(slice)]
fmt.Println(len(slice)) // 3
fmt.Println(slice) // [ b b]
// ^ first is empty string
I need to unmarshal some data, and in one case, the marshaled data represents a map with a byte array as a key.
Slices are not allowed as map keys, however arrays are. But the problem here is that arrays can't be created with a non-constant size as far as I can tell.
Example:
package main
import (
"fmt"
"reflect"
)
func getHashable(value interface{}) interface{} {
rfl := reflect.ValueOf(value)
if rfl.Kind() == reflect.Slice && rfl.Type().Elem().Kind() == reflect.Uint8 {
slice, ok := value.([]uint8)
if !ok {
panic(fmt.Errorf("Could not coerce to []uint8"))
}
var arr [len(slice)]uint8 // This fails
copy(arr, slice)
value = arr
}
return value
}
func unmarshalMap(serialized []byte) map[interface{}]interface{} {
result := make(map[interface{}]interface{})
for len(serialized) > 0 {
var value interface{}
key, bytesConsumed := deserializeValue(serialized)
serialized = serialized[bytesConsumed:]
value, bytesConsumed = deserializeValue(serialized)
serialized = serialized[bytesConsumed:]
result[getHashable(key)] = value
}
}
If deserializeValue() returns a []byte, then it can't be stored as a key in the result map. An array would work, but I can't create an array because I don't know what size I'll need until runtime, and it only allows compile time constants.
A simplified version https://play.golang.org/p/wkYGs3S-uSD fails with the error
./prog.go:15:12: non-constant array bound len(slice)
How can I use an array of bytes that I've unmarshaled as a key in a map in Go?
While use a string is clearly the better way, if the code you don't control uses a byte array as key, here is how you can turn the byte slice into an array as an interface using reflect.
varr := reflect.New(reflect.ArrayOf(len(slice), reflect.TypeOf(uint8(0))))
reflect.Copy(varr.Elem(), reflect.ValueOf(slice))
return varr.Elem().Interface()
Please consider other options before using this.
Playground: https://play.golang.org/p/CXsxZwgjiRR
Use string instead of a fixed size array of bytes. A string can hold an arbitrary sequence of bytes.
func getHashable(value interface{}) interface{} {
rfl := reflect.ValueOf(value)
if rfl.Kind() == reflect.Slice && rfl.Type().Elem().Kind() == reflect.Uint8 {
value = string(rfl.Bytes())
}
return value
}
If you are only need to handle []byte and not named types for []byte, use type assertions instead of reflection:
func getHashable(value interface{}) interface{} {
switch value := value.(type) {
case []byte:
return string(value)
default:
return value
}
}
If the user of the map needs to distinguish string keys from keys created form []byte, define a string type to distinguish those values:
type convertedSlice string
Replace use of string() conversion in the code above with convertedSlice().
The application can check for converted keys with:
_, ok := key.(convertedSlice) // ok is true if key is converted slice.
and convert the keys back to []byte with:
cv, ok := key.(convertedSice)
if ok {
key = []byte(cv)
}
A basic question that I'm struggling to find an answer for as there are a lot of answers about how to join two slices using the append function and the spread operator which erroneously use the word 'array'.
I am new to Go and have made the assumption that using sized arrays is good practice where the size is known. However I am struggling to work with arrays as I can't figure out how to do simple operations such as concatenation. Here is some code.
var seven [7]int
five := [5]int{1,2,3,4,5}
two := [2]int{6,7}
//this doesn't work as both the inputs and assignment are the wrong type
seven = append(five,two)
//this doesn't work as the assignment is still the wrong type
seven = append(five[:],two[:])
//this works but I'm not using arrays anymore so may as well use slices everywhere and forget sizing
seven2 := append(five[:],two[:])
As far as I can see I can either just give up on arrays and use slices exclusively or I could write a loop to explicitly construct the new array. Is there a third option?
append() can only be used to append elements to a slice. If you have an array, you can't pass that directly to append().
What you may do is slice the array, so you get a slice (which will use the array as its backing store), and you can use that slice as the target and source of elements.
For example:
s := seven[:0]
s = append(s, five[:]...)
s = append(s, two[:]...)
fmt.Println(seven)
This will print (try it on the Go Playground):
[1 2 3 4 5 6 7]
Also note that since append() returns the resulting slice, it's possible to write all this in one line:
_ = append(append(seven[:0], five[:]...), two[:]...)
(Storing the result is not needed here because we have and want to use only the backing array, but in general that is not the case.)
This outputs the same, try it on the Go Playground. Although this isn't very readable, so it's not worth compacting it into a single line.
Although when you have the target array, "appending" arrays is nothing more than copying them to the target, to the proper position. For that, you may use the builtin copy() function too. Note that the copy() function also accepts only slices, so you have to slice the arrays here too.
copy(seven[:], five[:])
copy(seven[len(five):], two[:])
fmt.Println(seven)
This will output the same. Try this one on the Go Playground.
You can use copy
copy(seven[:], five[:])
copy(seven[5:], two[:])
fmt.Printf("%v\n", seven)
> [1 2 3 4 5 6 7]
You can concatenate two arrays in go using copy function
package main
import "fmt"
func main() {
five := [5]int{1, 2, 3, 4, 5}
two := [2]int{6, 7}
var n [len(five) + len(two)]int
copy(n[:], five[:])
copy(n[len(five):], two[:])
fmt.Println(n)
}
https://blog.golang.org/go-slices-usage-and-internals
Golang runtime used to check whether current index exceeds the maximum possible.
On the side of array, it look ups its type (which contain its len and reference to the element type), because that's type, that can be registered only at compile time.
// each array mention with unique size creates new type
array := [5]byte{1,2,3,4,5}
On the side of slice, it look ups their header which looks like:
type slice {
data *byte
len int
cap int // capacity, the maximum possible index
}
As you can see, any slice is a single structure with data and len, cap fields, meanwhile array is just single pointer to data (*byte).
When you trying to convert array to slice, it just creates slice header and fills fields with:
slice := array[:]
==
slice := Slice{}
slice.data = array
slice.len = type_of(array).len
slice.cap = type_of(array).len
you can do that simply by converting array into slice:
arr1 := [...]int {1,2,3,}
arr2 := [...]int {4,5,6, }
//arr3 = arr1 + arr2 // not allowed
// converting arrays into slice
slc_arr1, slc_arr2 := arr1[:], arr2[:]
slc_arr3 := make([]int, 0)
slc_arr3 = append(slc_arr1, slc_arr2...)
fmt.Println(slc_arr3) // [1 2 3 4 5 6]
There is a more general way of appending an array of any type(once Golang has generics, but for now this solution is specific to strings. Just change the type as appropriate). The notion of Fold comes from Functional Programming. Note I have also included a filter function which also uses Fold. The solution is not stack safe but in many cases that does not matter. It can be made stack safe with trampolining. At the end is an example of its usage.
func FoldRightStrings(as, z []string, f func(string, []string) []string) []string {
if len(as) > 1 { //Slice has a head and a tail.
h, t := as[0], as[1:len(as)]
return f(h, FoldRightStrings(t, z, f))
} else if len(as) == 1 { //Slice has a head and an empty tail.
h := as[0]
return f(h, FoldRightStrings([]string{}, z, f))
}
return z
}
func FilterStrings(as []string, p func(string) bool) []string {
var g = func(h string, accum []string) []string {
if p(h) {
return append(accum, h)
} else {
return accum
}
}
return FoldRightStrings(as, []string{}, g)
}
func AppendStrings(as1, as2 []string) []string {
var g = func(h string, accum []string) []string {
return append(accum, h)
}
return FoldRightStrings(as1, as2, g)
}
func TestAppendStringArrays(t *testing.T) {
strings := []string{"a","b","c"}
bigarray := AppendStrings(AppendStrings(strings, strings),AppendStrings(strings, strings))
if diff := deep.Equal(bigarray, []string{"a","b","c","c","b","a","a","b","c","c","b","a"}); diff != nil {
t.Error(diff)
}
}
To declare an empty slice, with a non-fixed size,
is it better to do:
mySlice1 := make([]int, 0)
or:
mySlice2 := []int{}
Just wondering which one is the correct way.
The two alternative you gave are semantically identical, but using make([]int, 0) will result in an internal call to runtime.makeslice (Go 1.16).
You also have the option to leave it with a nil value:
var myslice []int
As written in the Golang.org blog:
a nil slice is functionally equivalent to a zero-length slice, even though it points to nothing. It has length zero and can be appended to, with allocation.
A nil slice will however json.Marshal() into "null" whereas an empty slice will marshal into "[]", as pointed out by #farwayer.
None of the above options will cause any allocation, as pointed out by #ArmanOrdookhani.
They are equivalent. See this code:
mySlice1 := make([]int, 0)
mySlice2 := []int{}
fmt.Println("mySlice1", cap(mySlice1))
fmt.Println("mySlice2", cap(mySlice2))
Output:
mySlice1 0
mySlice2 0
Both slices have 0 capacity which implies both slices have 0 length (cannot be greater than the capacity) which implies both slices have no elements. This means the 2 slices are identical in every aspect.
See similar questions:
What is the point of having nil slice and empty slice in golang?
nil slices vs non-nil slices vs empty slices in Go language
As an addition to #ANisus' answer...
below is some information from the "Go in action" book, which I think is worth mentioning:
Difference between nil & empty slices
If we think of a slice like this:
[pointer] [length] [capacity]
then:
nil slice: [nil][0][0]
empty slice: [addr][0][0] // points to an address
nil slice
They’re useful when you want to represent a slice that doesn’t exist, such as when an exception occurs in a function that returns a slice.
// Create a nil slice of integers.
var slice []int
empty slice
Empty slices are useful when you want to represent an empty collection, such as when a database query returns zero results.
// Use make to create an empty slice of integers.
slice := make([]int, 0)
// Use a slice literal to create an empty slice of integers.
slice := []int{}
Regardless of whether you’re using a nil slice or an empty slice, the built-in functions append, len, and cap work the same.
Go playground example:
package main
import (
"fmt"
)
func main() {
var nil_slice []int
var empty_slice = []int{}
fmt.Println(nil_slice == nil, len(nil_slice), cap(nil_slice))
fmt.Println(empty_slice == nil, len(empty_slice), cap(empty_slice))
}
prints:
true 0 0
false 0 0
Empty slice and nil slice are initialized differently in Go:
var nilSlice []int
emptySlice1 := make([]int, 0)
emptySlice2 := []int{}
fmt.Println(nilSlice == nil) // true
fmt.Println(emptySlice1 == nil) // false
fmt.Println(emptySlice2 == nil) // false
As for all three slices, len and cap are 0.
In addition to #ANisus' answer
When using the official Go MongoDb Driver, a nil slice will also marshal into "null" whereas an empty slice will marshal into "[]".
When using using the community supported MGO driver, both nil and empty slices will be marshalled into "[]".
Reference: https://jira.mongodb.org/browse/GODRIVER-971
I am trying to write an application that reads RPM files. The start of each block has a Magic char of [4]byte.
Here is my struct
type Lead struct {
Magic [4]byte
Major, Minor byte
Type uint16
Arch uint16
Name string
OS uint16
SigType uint16
}
I am trying to do the following:
lead := Lead{}
lead.Magic = buffer[0:4]
I am searching online and not sure how to go from a slice to an array (without copying). I can always make the Magic []byte (or even uint64), but I was more curious on how would I go from type []byte to [4]byte if needed to?
The built in method copy will only copy a slice to a slice NOT a slice to an array.
You must trick copy into thinking the array is a slice
copy(varLead.Magic[:], someSlice[0:4])
Or use a for loop to do the copy:
for index, b := range someSlice {
varLead.Magic[index] = b
}
Or do as zupa has done using literals. I have added onto their working example.
Go Playground
You have allocated four bytes inside that struct and want to assign a value to that four byte section. There is no conceptual way to do that without copying.
Look at the copy built-in for how to do that.
Try this:
copy(lead.Magic[:], buf[0:4])
Tapir Liui (auteur de Go101) twitte:
Go 1.18 1.19 1.20 will support conversions from slice to array: golang/go issues 46505.
So, since Go 1.18,the slice copy2 implementation could be written as:
*(*[N]T)(d) = [N]T(s)
or, even simpler if the conversion is allowed to present as L-values:
[N]T(d) = [N]T(s)
Without copy, you can convert, with the next Go 1.17 (Q3 2021) a slice to an array pointer.
This is called "un-slicing", giving you back a pointer to the underlying array of a slice, again, without any copy/allocation needed:
See golang/go issue 395: spec: convert slice x into array pointer, now implemented with CL 216424/, and commit 1c26843
Converting a slice to an array pointer yields a pointer to the underlying array of the slice.
If the length of the slice is less than the length of the array,
a run-time panic occurs.
s := make([]byte, 2, 4)
s0 := (*[0]byte)(s) // s0 != nil
s2 := (*[2]byte)(s) // &s2[0] == &s[0]
s4 := (*[4]byte)(s) // panics: len([4]byte) > len(s)
var t []string
t0 := (*[0]string)(t) // t0 == nil
t1 := (*[1]string)(t) // panics: len([1]string) > len(s)
So in your case, provided Magic type is *[4]byte:
lead.Magic = (*[4]byte)(buffer)
Note: type aliasing will work too:
type A [4]int
var s = (*A)([]int{1, 2, 3, 4})
Why convert to an array pointer? As explained in issue 395:
One motivation for doing this is that using an array pointer allows the compiler to range check constant indices at compile time.
A function like this:
func foo(a []int) int
{
return a[0] + a[1] + a[2] + a[3];
}
could be turned into:
func foo(a []int) int
{
b := (*[4]int)(a)
return b[0] + b[1] + b[2] + b[3];
}
allowing the compiler to check all the bounds once only and give compile-time errors about out of range indices.
Also:
One well-used example is making classes as small as possible for tree nodes or linked list nodes so you can cram as many of them into L1 cache lines as possible.
This is done by each node having a single pointer to a left sub-node, and the right sub-node being accessed by the pointer to the left sub-node + 1.
This saves the 8-bytes for the right-node pointer.
To do this you have to pre-allocate all the nodes in a vector or array so they're laid out in memory sequentially, but it's worth it when you need it for performance.
(This also has the added benefit of the prefetchers being able to help things along performance-wise - at least in the linked list case)
You can almost do this in Go with:
type node struct {
value int
children *[2]node
}
except that there's no way of getting a *[2]node from the underlying slice.
Go 1.20 (Q1 2023): this is addressed with CL 430415, 428938 (type), 430475 (reflect) and 429315 (spec).
Go 1.20
You can convert from a slice to an array directly with the usual conversion syntax T(x). The array's length can't be greater than the slice's length:
func main() {
slice := []int64{10, 20, 30, 40}
array := [4]int64(slice)
fmt.Printf("%T\n", array) // [4]int64
}
Go 1.17
Starting from Go 1.17 you can directly convert a slice to an array pointer. With Go's type conversion syntax T(x) you can do this:
slice := make([]byte, 4)
arrptr := (*[4]byte)(slice)
Keep in mind that the length of the array must not be greater than the length of the slice, otherwise the conversion will panic.
bad := (*[5]byte)(slice) // panics: slice len < array len
This conversion has the advantage of not making any copy, because it simply yields a pointer to the underlying array.
Of course you can dereference the array pointer to obtain a non-pointer array variable, so the following also works:
slice := make([]byte, 4)
var arr [4]byte = *(*[4]byte)(slice)
However dereferencing and assigning will subtly make a copy, since the arr variable is now initialized to the value that results from the conversion expression. To be clear (using ints for simplicity):
v := []int{10,20}
a := (*[2]int)(v)
a[0] = 500
fmt.Println(v) // [500 20] (changed, both point to the same backing array)
w := []int{10,20}
b := *(*[2]int)(w)
b[0] = 500
fmt.Println(w) // [10 20] (unchanged, b holds a copy)
One might wonder why the conversion checks the slice length and not the capacity (I did). Consider the following program:
func main() {
a := []int{1,2,3,4,5,6}
fmt.Println(cap(a)) // 6
b := a[:3]
fmt.Println(cap(a)) // still 6
c := (*[3]int)(b)
ptr := uintptr(unsafe.Pointer(&c[0]))
ptr += 3 * unsafe.Sizeof(int(0))
i := (*int)(unsafe.Pointer(ptr))
fmt.Println(*i) // 4
}
The program shows that the conversion might happen after reslicing. The original backing array with six elements is still there, so one might wonder why a runtime panic occurs with (*[6]int)(b) where cap(b) == 6.
This has actually been brought up. It's worth to remember that, unlike slices, an array has fixed size, therefore it needs no notion of capacity, only length:
a := [4]int{1,2,3,4}
fmt.Println(len(a) == cap(a)) // true
You might be able to do the whole thing with one read, instead of reading individually into each field. If the fields are fixed-length, then you can do:
lead := Lead{}
// make a reader to dispense bytes so you don't have to keep track of where you are in buffer
reader := bytes.NewReader(buffer)
// read into each field in Lead, so Magic becomes buffer[0:4],
// Major becomes buffer[5], Minor is buffer[6], and so on...
binary.Read(reader, binary.LittleEndian, &lead)
Don't. Slice itself is suffice for all purpose. Array in go lang should be regarded as the underlying structure of slice. In every single case, use only slice. You don't have to array yourself. You just do everything by slice syntax. Array is only for computer. In most cases, slice is better, clear in code. Even in other cases, slice still is sufficient to reflex your idea.