shuffle array in Go - arrays

I tried to translate the following Python code to Go
import random
list = [i for i in range(1, 25)]
random.shuffle(list)
print(list)
but found my Go version lengthy and awkward because there is no shuffle function and I had to implement interfaces and convert types.
What would be an idiomatic Go version of my code?

dystroy's answer is perfectly reasonable, but it's also possible to shuffle without allocating any additional slices.
for i := range slice {
j := rand.Intn(i + 1)
slice[i], slice[j] = slice[j], slice[i]
}
See this Wikipedia article for more details on the algorithm. rand.Perm actually uses this algorithm internally as well.

As your list is just the integers from 1 to 25, you can use Perm :
list := rand.Perm(25)
for i, _ := range list {
list[i]++
}
Note that using a permutation given by rand.Perm is an effective way to shuffle any array.
dest := make([]int, len(src))
perm := rand.Perm(len(src))
for i, v := range perm {
dest[v] = src[i]
}

Since 1.10 Go includes an official Fisher-Yates shuffle function.
Documentation: pkg/math/rand/#Shuffle
math/rand: add Shuffle
Shuffle uses the Fisher-Yates algorithm.
Since this is new API, it affords us the opportunity
to use a much faster Int31n implementation that mostly avoids division.
As a result, BenchmarkPerm30ViaShuffle is
about 30% faster than BenchmarkPerm30,
despite requiring a separate initialization loop
and using function calls to swap elements.
See also the original CL 51891
First, as commented by shelll:
Do not forget to seed the random, or you will always get the same order.
For example rand.Seed(time.Now().UnixNano())
Example:
words := strings.Fields("ink runs from the corners of my mouth")
rand.Shuffle(len(words), func(i, j int) {
words[i], words[j] = words[j], words[i]
})
fmt.Println(words)

Answer by Evan Shaw has a minor bug. If we iterate through the slice from lowest index to highest, to get a uniformly (pseudo) random shuffle, according to the same article, we must choose a random integer from interval [i,n) as opposed to [0,n+1).
That implementation will do what you need for larger inputs, but for smaller slices, it will perform a non-uniform shuffle.
To utilize rand.Intn(), we can do:
for i := len(slice) - 1; i > 0; i-- {
j := rand.Intn(i + 1)
slice[i], slice[j] = slice[j], slice[i]
}
following the same algorithm from Wikipedia article.

Maybe you can also use the following function:
func main() {
slice := []int{10, 12, 14, 16, 18, 20}
Shuffle(slice)
fmt.Println(slice)
}
func Shuffle(slice []int) {
r := rand.New(rand.NewSource(time.Now().Unix()))
for n := len(slice); n > 0; n-- {
randIndex := r.Intn(n)
slice[n-1], slice[randIndex] = slice[randIndex], slice[n-1]
}
}

When using the math/rand package, do not forget to set a source
// Random numbers are generated by a Source. Top-level functions, such as
// Float64 and Int, use a default shared Source that produces a deterministic
// sequence of values each time a program is run. Use the Seed function to
// initialize the default Source if different behavior is required for each run.
So I wrote a Shuffle function that takes this into consideration:
import (
"math/rand"
)
func Shuffle(array []interface{}, source rand.Source) {
random := rand.New(source)
for i := len(array) - 1; i > 0; i-- {
j := random.Intn(i + 1)
array[i], array[j] = array[j], array[i]
}
}
And to use it:
source := rand.NewSource(time.Now().UnixNano())
array := []interface{}{"a", "b", "c"}
Shuffle(array, source) // [c b a]
If you would like to use it, you can find it here https://github.com/shomali11/util

Raed's approach is very inflexible because of []interface{} as input. Here is more convenient version for go>=1.8:
func Shuffle(slice interface{}) {
rv := reflect.ValueOf(slice)
swap := reflect.Swapper(slice)
length := rv.Len()
for i := length - 1; i > 0; i-- {
j := rand.Intn(i + 1)
swap(i, j)
}
}
Example usage:
rand.Seed(time.Now().UnixNano()) // do it once during app initialization
s := []int{1, 2, 3, 4, 5}
Shuffle(s)
fmt.Println(s) // Example output: [4 3 2 1 5]
And also, don't forget that a little copying is better than a little dependency

Use Shuffle() from the math/rand library.
Here's an example:
package main
import (
"fmt"
"math/rand"
"strings"
)
func main() {
words := strings.Fields("ink runs from the corners of my mouth")
rand.Shuffle(len(words), func(i, j int) {
words[i], words[j] = words[j], words[i]
})
fmt.Println(words)
}
Since it comes from the math/rand library it needs to be seeded. See here for more details.

Related

Setlength in Delphi 10.2.3 for array like types

Although I found some postings concerning the subject setlength I couldn't find out a solution for my problem:
With the code below - which is part of a much bigger program -
type TVektor2=array[20] of extended;
TElement2=array[20] of String;
Procedure Sort_Shell2(
element1X: TElement2; zahlX: TVektor2; var Element2X : TElement2;
var zahl2X : TVektor2);
var
bis, i, j, k, min : LongInt; l, laenge : single;
h,s,w,h1,h2, ElemX: string;
e : array[20] of String;
begin
laenge := 5; // just an example
SetLength(Element1X, 3); /// Error
//DynArraySetlength(e,l,1); /// how?
bis := High(e);
k := bis shr 1;// div 2
while (k > 0) do
begin
for i := 0 to (bis - k) do
begin
j := i;
h1 := e[j]; //I use this because before I had an Acces violation
h2 := e[j + k]; // using directly e[j] := e[j+k];
while (j >= 0) and (h1 > h2) do
begin
h := h1;
l:=zahlx[j]; //str(l:5:3,S);showmessage(h + s);
e[j] :=e[j + k];
zahlx[j] := zahlx[j+ k];
e[j + k] := h;
zahlx[j+ k]:=l;
if j > k then
Dec(j, k)
else
j := 0;
end; // {end while]
end; // { end for}
k := k shr 1; // div 2
end; // {end while}
Element2x:=e; zahl2x :=zahlx;
end;
I get the error 'incompatible types' if I try the setlength command like this.
I tried - with a for next loop -to attribute to each position of the static array (with 20 entries) or also to the correponding dynamic array and then to use setlength.
But it didn't work. Is there some casting possible to transform TElement2 to an array ? (since it is already an array!)
Why isn't it possible to use a simple static array[1..20] of strings = a, set for each position a[i] = TElement2[i] and use setlength(a,5)?
If I use DynArraySetLength(Pointer, typeInfo,dimCnt, lengthVec) what must I use for these variables?
I don't know nearly anything about Pointers and I have no idea for such problem what parameters I must use to get an array of a given length starting with the given TElement2 array. By the way, in general, is it a good idea to use dynamic arrays?
By the way there might also be an error in this sorting routine because it doesn't work well...
Can any one help me?
In order to use a Dynamic Array in Delphi you have to declare an array like this:
TElement2=array of String;
and not TElement2=array[20] of String or TElement2=array[1..20] of String;
If you declare TElement2 that way then SetLength(element1X, 3); is going to work.
moreover when you assign at the bottom of the code
Element2x:=e;
it's not going to compile unless both variable aren't declared of the same type:
e : TElement2;

Create slice of range with large intervals in Go

I want to get a slice in Go which looks like this:
[100, 200, 300, 400, 500]
In Python I would do this:
l = range(100, 600, 100)
I know I can do this in Go:
l := []int{}
for i:=100; i<600; i+=100{
l = append(l, i)
}
but isn't there anything simpler to create this slice?
Do it the same way Python does:
func pyrange(start, end, step int) []int {
// TODO: Error checking to make sure parameters are all valid,
// else you could get divide by zero in make and other errors.
rtn := make([]int, 0, (end-start)/step)
for i := start; i < end; i += step {
rtn = append(rtn, i)
}
return rtn
}
With a function.
Obviously only worth the time if you need to do this often. By default Go does not include a function like this, so you need to write your own (or find a third party library) if you want one.

Most idiomatic way to select elements from an array in Golang?

I have an array of strings, and I'd like to exclude values that start in foo_ OR are longer than 7 characters.
I can loop through each element, run the if statement, and add it to a slice along the way. But I was curious if there was an idiomatic or more golang-like way of accomplishing that.
Just for example, the same thing might be done in Ruby as
my_array.select! { |val| val !~ /^foo_/ && val.length <= 7 }
There is no one-liner as you have it in Ruby, but with a helper function you can make it almost as short.
Here's our helper function that loops over a slice, and selects and returns only the elements that meet a criteria captured by a function value:
func filter(ss []string, test func(string) bool) (ret []string) {
for _, s := range ss {
if test(s) {
ret = append(ret, s)
}
}
return
}
Starting with Go 1.18, we can write it generic so it will work with all types, not just string:
func filter[T any](ss []T, test func(T) bool) (ret []T) {
for _, s := range ss {
if test(s) {
ret = append(ret, s)
}
}
return
}
Using this helper function your task:
ss := []string{"foo_1", "asdf", "loooooooong", "nfoo_1", "foo_2"}
mytest := func(s string) bool { return !strings.HasPrefix(s, "foo_") && len(s) <= 7 }
s2 := filter(ss, mytest)
fmt.Println(s2)
Output (try it on the Go Playground, or the generic version: Go Playground):
[asdf nfoo_1]
Note:
If it is expected that many elements will be selected, it might be profitable to allocate a "big" ret slice beforehand, and use simple assignment instead of the append(). And before returning, slice the ret to have a length equal to the number of selected elements.
Note #2:
In my example I chose a test() function which tells if an element is to be returned. So I had to invert your "exclusion" condition. Obviously you may write the helper function to expect a tester function which tells what to exclude (and not what to include).
Have a look at robpike's filter library. This would allow you to do:
package main
import (
"fmt"
"strings"
"filter"
)
func isNoFoo7(a string) bool {
return ! strings.HasPrefix(a, "foo_") && len(a) <= 7
}
func main() {
a := []string{"test", "some_other_test", "foo_etc"}
result := Choose(a, isNoFoo7)
fmt.Println(result) // [test]
}
Interestingly enough the README.md by Rob:
I wanted to see how hard it was to implement this sort of thing in Go, with as nice an API as I could manage. It wasn't hard.
Having written it a couple of years ago, I haven't had occasion to use it once. Instead, I just use "for" loops.
You shouldn't use it either.
So the most idiomatic way according to Rob would be something like:
func main() {
a := []string{"test", "some_other_test", "foo_etc"}
nofoos := []string{}
for i := range a {
if(!strings.HasPrefix(a[i], "foo_") && len(a[i]) <= 7) {
nofoos = append(nofoos, a[i])
}
}
fmt.Println(nofoos) // [test]
}
This style is very similar, if not identical, to the approach any C-family language takes.
Today, I stumbled on a pretty idiom that surprised me. If you want to filter a slice in place without allocating, use two slices with the same backing array:
s := []T{
// the input
}
s2 := s
s = s[:0]
for _, v := range s2 {
if shouldKeep(v) {
s = append(s, v)
}
}
Here's a specific example of removing duplicate strings:
s := []string{"a", "a", "b", "c", "c"}
s2 := s
s = s[:0]
var last string
for _, v := range s2 {
if len(s) == 0 || v != last {
last = v
s = append(s, v)
}
}
If you need to keep both slices, simply replace s = s[:0] with s = nil or s = make([]T, 0, len(s)), depending on whether you want append() to allocate for you.
There are a couple of nice ways to filter a slice without allocations or new dependencies. Found in the Go wiki on Github:
Filter (in place)
n := 0
for _, x := range a {
if keep(x) {
a[n] = x
n++
}
}
a = a[:n]
And another, more readable, way:
Filtering without allocating
This trick uses the fact that a slice shares the same backing array
and capacity as the original, so the storage is reused for the
filtered slice. Of course, the original contents are modified.
b := a[:0]
for _, x := range a {
if f(x) {
b = append(b, x)
}
}
For elements which must be garbage collected, the following code can
be included afterwards:
for i := len(b); i < len(a); i++ {
a[i] = nil // or the zero value of T
}
One thing I'm not sure about is whether the first method needs clearing (setting to nil) the items in slice a after index n, like they do in the second method.
EDIT: the second way is basically what MicahStetson described in his answer. In my code I use a function similar to the following, which is probably as good as it gets in terms on performance and readability:
func filterSlice(slice []*T, keep func(*T) bool) []*T {
newSlice := slice[:0]
for _, item := range slice {
if keep(item) {
newSlice = append(newSlice, item)
}
}
// make sure discarded items can be garbage collected
for i := len(newSlice); i < len(slice); i++ {
slice[i] = nil
}
return newSlice
}
Note that if items in your slice are not pointers and don't contain pointers you can skip the second for loop.
There isn't an idiomatic way you can achieve the same expected result in Go in one single line as in Ruby, but with a helper function you can obtain the same expressiveness as in Ruby.
You can call this helper function as:
Filter(strs, func(v string) bool {
return strings.HasPrefix(v, "foo_") // return foo_testfor
}))
Here is the whole code:
package main
import "strings"
import "fmt"
// Returns a new slice containing all strings in the
// slice that satisfy the predicate `f`.
func Filter(vs []string, f func(string) bool) []string {
vsf := make([]string, 0)
for _, v := range vs {
if f(v) && len(v) > 7 {
vsf = append(vsf, v)
}
}
return vsf
}
func main() {
var strs = []string{"foo1", "foo2", "foo3", "foo3", "foo_testfor", "_foo"}
fmt.Println(Filter(strs, func(v string) bool {
return strings.HasPrefix(v, "foo_") // return foo_testfor
}))
}
And the running example: Playground
you can use the loop as you did and wrap it to a utils function for reuse.
For multi-datatype support, copy-paste will be a choice. Another choice is writing a generating tool.
And final option if you want to use lib, you can take a look on https://github.com/ledongthuc/goterators#filter that I created to reuse aggregate & transform functions.
It requires the Go 1.18 to use that support generic + dynamic type you want to use with.
filteredItems, err := Filter(list, func(item int) bool {
return item % 2 == 0
})
filteredItems, err := Filter(list, func(item string) bool {
return item.Contains("ValidWord")
})
filteredItems, err := Filter(list, func(item MyStruct) bool {
return item.Valid()
})
It also supports Reduce in case you want to optimize the way you select.
Hope it's useful with you!
"Select Elements from Array" is also commonly called a filter function. There's no such thing in go. There are also no other "Collection Functions" such as map or reduce. For the most idiomatic way to get the desired result, I find https://gobyexample.com/collection-functions a good reference:
[...] in Go it’s common to provide collection functions if and when they are specifically needed for your program and data types.
They provide an implementation example of the filter function for strings:
func Filter(vs []string, f func(string) bool) []string {
vsf := make([]string, 0)
for _, v := range vs {
if f(v) {
vsf = append(vsf, v)
}
}
return vsf
}
However, they also say, that it's often ok to just inline the function:
Note that in some cases it may be clearest to just inline the
collection-manipulating code directly, instead of creating and calling
a helper function.
In general, golang tries to only introduce orthogonal concepts, meaning that when you can solve a problem one way, there shouldn't be too many more ways to solve it. This adds simplicity to the language by only having a few core concepts, such that not every developer uses a different subset of the language.
Take a look at this library: github.com/thoas/go-funk
It provides an implementation of a lot of life-saving idioms in Go (including filtering of elements in array for instance).
r := funk.Filter([]int{1, 2, 3, 4}, func(x int) bool {
return x%2 == 0
}
Here is an elegant example of both Fold and Filter that uses recursion to accomplish filtering. FoldRight is also generally useful. It is not stack safe but could be made so with trampolining. Once Golang has generics it can be entirely generalized for any 2 types:
func FoldRightStrings(as, z []string, f func(string, []string) []string) []string {
if len(as) > 1 {//Slice has a head and a tail.
h, t := as[0], as[1:len(as)]
return f(h, FoldRightStrings(t, z, f))
} else if len(as) == 1 {//Slice has a head and an empty tail.
h := as[0]
return f(h, FoldRightStrings([]string{}, z, f))
}
return z
}
func FilterStrings(as []string, p func(string) bool) []string {
var g = func(h string, accum []string) []string {
if p(h) {
return append(accum, h)
} else {
return accum
}
}
return FoldRightStrings(as, []string{}, g)
}
Here is an example of its usage to filter out all the strings with length < 8
var p = func(s string) bool {
if len(s) < 8 {
return true
} else {
return false
}
}
FilterStrings([]string{"asd","asdfas","asdfasfsa","asdfasdfsadfsadfad"}, p)
I`m developing this library: https://github.com/jose78/go-collection. PLease try this example to filter elements:
package main
import (
"fmt"
col "github.com/jose78/go-collection/collections"
)
type user struct {
name string
age int
id int
}
func main() {
newMap := generateMapTest()
if resultMap, err := newMap.FilterAll(filterEmptyName); err != nil {
fmt.Printf("error")
} else {
fmt.Printf("Result: %v\n", resultMap)
result := resultMap.ListValues()
fmt.Printf("Result: %v\n", result)
fmt.Printf("Result: %v\n", result.Reverse())
fmt.Printf("Result: %v\n", result.JoinAsString(" <---> "))
fmt.Printf("Result: %v\n", result.Reverse().JoinAsString(" <---> "))
result.Foreach(simpleLoop)
err := result.Foreach(simpleLoopWithError)
if err != nil {
fmt.Println(err)
}
}
}
func filterEmptyName(key interface{}, value interface{}) bool {
user := value.(user)
return user.name != "empty"
}
func generateMapTest() (container col.MapType) {
container = col.MapType{}
container[1] = user{"Alvaro", 6, 1}
container[2] = user{"Sofia", 3, 2}
container[3] = user{"empty", 0, -1}
return container
}
var simpleLoop col.FnForeachList = func(mapper interface{}, index int) {
fmt.Printf("%d.- item:%v\n", index, mapper)
}
var simpleLoopWithError col.FnForeachList = func(mapper interface{}, index int) {
if index > 0 {
panic(fmt.Sprintf("Error produced with index == %d\n", index))
}
fmt.Printf("%d.- item:%v\n", index, mapper)
}
Result of execution:
Result: map[1:{Alvaro 6 1} 2:{Sofia 3 2}]
Result: [{Sofia 3 2} {Alvaro 6 1}]
Result: [{Alvaro 6 1} {Sofia 3 2}]
Result: {Sofia 3 2} <---> {Alvaro 6 1}
Result: {Alvaro 6 1} <---> {Sofia 3 2}
0.- item:{Sofia 3 2}
1.- item:{Alvaro 6 1}
0.- item:{Sofia 3 2}
Recovered in f Error produced with index == 1
ERROR: Error produced with index == 1
Error produced with index == 1
The DOC currently are located in wiki section of the project. You can try it in this link. I hope you like it...
REgaRDS...

Generate array with with range of integers

I have two values:
[3:6]
I was trying to play something around in Golang but I can't manage to find a good method to create an array according to those values.
This what I would like to achieve:
[3,4,5,6]
You can make use of the for ... range construct to make it more compact and maybe even faster:
lo, hi := 3, 6
s := make([]int, hi-lo+1)
for i := range s {
s[i] = i + lo
}
As a matter of curiosity, the loop can be implemented without a loop variable, but it will be slower, and the code longer. By decrementing hi:
for ; hi >= lo; hi-- {
s[hi-len(s)+1] = hi
}
Or incrementing lo:
for ; lo <= hi; lo++ {
s[len(s)-1-hi+lo] = lo
}
I was looking for a similar answer and #icza's answer was a great start. I ended up creating a helper function based on icza's answer:
func createNumbers(lo int, hi int) []int {
s := make([]int, hi-lo+1)
for i:= range s {
s[i] = i + lo
}
return s
}

What's the slice element access complexity in Go?

I thought it to be O(1), but this is from a pprof output:
140 140 176: var lastSB byte = s[lenSMinusOne]
88 88 177: var lastSuffixB byte = suffix[lenSuffixMinusOne]
and by average length of s is greater than length of suffix. Thus, this shows that accessing an element takes longer if the slice is bigger?
The function is:
func hasSuffix(s, suffix []byte) bool {
lenSMinusOne := len(s) - 1
lenSuffixMinusOne := len(suffix) - 1
var lastSB byte = s[lenSMinusOne]
var lastSuffixB byte = suffix[lenSuffixMinusOne]
if lenSMinusOne < lenSuffixMinusOne {
return false
} else if lastSB != lastSuffixB {
return false
} else {
for i := 0; i < lenSuffixMinusOne ; i++ {
if suffix[i] != s[lenSMinusOne-lenSuffixMinusOne+i] {
return false
}
}
}
return true
}
UPDATE:
To reproduce the results install fetch which uses go-porterstemmer fork with a large corpus(I use a 440mb file).
pprof collects samples during program execution to illuminate hotspots. Use the testing package and go test to run benchmarks.
As you should expect, the following benchmark shows that there is no difference between reading the 2nd element of a slice on average and reading the 2691st element of a slice on average, 13439773 ns/op versus 13460864 ns/op for 904,061 byte slice elements. Both benchmarks use the same underlying data arrays. Indexing a slice is O(1).
In your example, you are reading from two different underlying data arrays with different access patterns (outer versus inner loop). On modern processors, which have sophisticated memory management and optimization, you shouldn't expect the same results.
$ go version
go version devel +3ae7a530dd4e Sat Dec 28 09:37:54 2013 -0800 linux/amd64
$ go test -bench=IndexWord
904061 2 2690.8131199111563
testing: warning: no tests to run
PASS
BenchmarkIndexWordLong 100 13460864 ns/op
BenchmarkIndexWordShort 100 13439773 ns/op
ok bench 7.814s
$
.
package main
import (
"bytes"
"fmt"
"io/ioutil"
"testing"
)
var (
Words [][]byte
ShortLen = 2
)
func IndexWord(b *testing.B, words [][]byte) {
b.ResetTimer()
b.StartTimer()
var char byte
for i := 0; i < b.N; i++ {
for _, word := range words {
char = word[len(word)-1]
}
}
_ = char
}
func BenchmarkIndexWordLong(b *testing.B) {
words := make([][]byte, len(Words))
for i, word := range Words {
words[i] = word
}
IndexWord(b, words)
}
func BenchmarkIndexWordShort(b *testing.B) {
words := make([][]byte, len(Words))
for i, word := range Words {
if len(word) > ShortLen {
word = word[:ShortLen]
}
words[i] = word
}
IndexWord(b, words)
}
func init() {
// The Complete Works of William Shakespeare
// http://www.gutenberg.org/cache/epub/100/pg100.txt
text, err := ioutil.ReadFile(`/home/peter/pg100.txt`)
if err != nil {
panic(err)
}
var n, short, long int64
Words = bytes.Fields(text)
for i, word := range Words {
word = bytes.Repeat(word, 600) // Requires 4GB memory
Words[i] = word
n++
long += int64(len(word))
shortLen := ShortLen
if len(word) < ShortLen {
shortLen = len(word)
}
short += int64(shortLen)
}
fmt.Println(n, float64(short)/float64(len(Words)), float64(long)/float64(len(Words)))
}
The code for your hasSuffix function looks like a direct port from another language; it doesn't look like it is written for Go. Here's my rewrite.
func hasSuffix(s, suffix []byte) bool {
if len(s) < len(suffix) {
return false
}
s = s[len(s)-len(suffix):]
for i, x := range suffix {
if x != s[i] {
return false
}
}
return true
}
Also, Go has a bytes.HasSuffix function.
Package bytes
func HasSuffix
func HasSuffix(s, suffix []byte) bool
HasSuffix tests whether the byte slice s ends with suffix.
Slice access is O(1), but memory access on modern computers can take orders of magnitude more or less time depending on whether the value is cached. Without seeing your code, most likely this is why one memory access is slower than another.
Another possibility are that one of your slices is an array and the index is constant, meaning that bounds checking isn't necessary.

Resources