Processing arrays in Go parallel gives unexpected results - arrays

I am interested to calculate correlations in parallel in Go. The main problem I have is that all the Go processes seems to execute exactly the same calculation. I reproduced here the problem with a very simple example.
I obtain :
4 + 50 = 54
4 + 50 = 54
4 + 50 = 54
instead of :
1 + 20 = 21
2 + 30 = 32
3 + 40 = 43
If I move up "wg.Wait()" I obtain the good result but no parallelism :(
Thank's in advance for your comments !
package main
import (
"fmt"
"runtime"
"sync"
)
func process_array(x, y int) int {
r := x + y
return r
}
func main() {
a1 := []int{0, 1, 2, 3, 4}
a2 := []int{10, 20, 30, 40, 50}
runtime.GOMAXPROCS(8)
var wg sync.WaitGroup
for i := 1; i < 4 ; i++ {
wg.Add(1)
go func() {
defer wg.Done()
x :=process_array(a1[i],a2[i])
fmt.Println(a1[i],"+", a2[i],"=", x)
}()
//wg.Wait() give the good result
//but it is not parallel processing
// 1 + 20 = 21
// 2 + 30 = 32
// 3 + 40 = 43
}
wg.Wait() // give a repetition of the same result :
// 4 + 50 = 54
// 4 + 50 = 54
// 4 + 50 = 54
}

You're accessing the same copy of i in all goroutines. The output you see is because the loop happens to finish before any of the goroutines start executing.
This means that i has the same value in all goroutines, i.e. the last value it had in the loop.
Passing i as an argument to each of your goroutines, thereby operating on a copy per goroutine instead, solves this problem.
The reason you saw the result you expected when you added wg.Wait() in the loop is because you then introduced synchronization, waiting for the goroutine to finish before starting the next one. That means the execution was in fact serial, not parallell.
Here's the updated code, which works as you'd expect:
package main
import (
"fmt"
"runtime"
"sync"
)
func process_array(x, y int) int {
r := x + y
return r
}
func main() {
a1 := []int{0, 1, 2, 3, 4}
a2 := []int{10, 20, 30, 40, 50}
runtime.GOMAXPROCS(8)
var wg sync.WaitGroup
for i := 1; i < 4; i++ {
wg.Add(1)
go func(i int) {
defer wg.Done()
x := process_array(a1[i], a2[i])
fmt.Println(a1[i], "+", a2[i], "=", x)
}(i)
//wg.Wait() give the good result
//but it is not parallel processing
// 1 + 20 = 21
// 2 + 30 = 32
// 3 + 40 = 43
}
wg.Wait() // give a repetition of the same result :
// 4 + 50 = 54
// 4 + 50 = 54
// 4 + 50 = 54
}

Related

Batching at most 10 at a time over input

I'm trying and failing to accomplish the simple task of batching over input, at most 10 at a time. The follow code almost works:
func batchMe(input []int) {
fmt.Println("Length", len(input), len(input)/10)
for i := 0; i <= len(input)/10; i++ {
from := i * 10
to := (i + 1) * 10
if len(input) < to {
to = len(input)
}
fmt.Println("Batch", i, input[from:to])
}
But you can see from https://play.golang.org/p/_UgFD1iDyse that it prints:
Length 10 1
Batch 0 [1 2 3 4 5 6 7 8 9 10]
Batch 1 []
I don't want it to print Batch 1 in the case of 10 elements!
Perhaps there is a code simplification here?
Elegant solution from Tv on #go-nuts looks like:
for len(input) > 0 {
n := 10
if n > len(input) {
n = len(input)
}
chunk := input[:n]
input = input[n:]
fmt.Println("Batch", chunk)
}
https://play.golang.org/p/Y3U8dUD7Zrr

How to generate a sequence of numbers

I want to generate a sequence of numbers in Go but I can't find any built-in functions for this.
Basically I want the equivalent of PHP's range function in Golang:
array range ( mixed $start , mixed $end [, number $step = 1 ] )
It would be useful when creating a slice/array of numeric types and you want to populate/initialize it with a numeric sequence.
There is no equivalent to PHP's range in the Go standard library. You have to create one yourself. The simplest is to use a for loop:
func makeRange(min, max int) []int {
a := make([]int, max-min+1)
for i := range a {
a[i] = min + i
}
return a
}
Using it:
a := makeRange(10, 20)
fmt.Println(a)
Output (try it on the Go Playground):
[10 11 12 13 14 15 16 17 18 19 20]
Also note that if the range is small, you can use a composite literal:
a := []int{1, 2, 3}
fmt.Println(a) // Output is [1 2 3]
1- You may use:
//Create a slice containing a range of elements.
//
// start: First value of the sequence.
// end: The sequence is ended upon reaching the end value.
// step: step will be used as the increment between elements in the sequence.
// step should be given as a positive number.
//
//Return Values: Returns a slice of elements from start to end, inclusive.
func NewSlice(start, end, step int) []int {
if step <= 0 || end < start {
return []int{}
}
s := make([]int, 0, 1+(end-start)/step)
for start <= end {
s = append(s, start)
start += step
}
return s
}
Try it on The Go Playground:
package main
import "fmt"
//Create a slice containing a range of elements.
//
// start: First value of the sequence.
// end: The sequence is ended upon reaching the end value.
// step: step will be used as the increment between elements in the sequence.
// step should be given as a positive number.
//
//Return Values: Returns a slice of elements from start to end, inclusive.
func NewSlice(start, end, step int) []int {
if step <= 0 || end < start {
return []int{}
}
s := make([]int, 0, 1+(end-start)/step)
for start <= end {
s = append(s, start)
start += step
}
return s
}
func main() {
s := []int{0, 1, 2, 3, 4, 5, 6, 7, 8, 9}
fmt.Println(s) // [0 1 2 3 4 5 6 7 8 9]
fmt.Println(NewSlice(10, 19, 1)) // [10 11 12 13 14 15 16 17 18 19]
fmt.Println(NewSlice(10, 28, 2)) // [10 12 14 16 18 20 22 24 26 28]
fmt.Println(NewSlice(-10, -1, 1)) // [-10 -9 -8 -7 -6 -5 -4 -3 -2 -1]
}
2- You may use:
// Returns a slice of elements with exact count.
// step will be used as the increment between elements in the sequence.
// step should be given as a positive, negative or zero number.
func NewSlice(start, count, step int) []int {
s := make([]int, count)
for i := range s {
s[i] = start
start += step
}
return s
}
Try it on The Go Playground:
package main
import "fmt"
func NewSlice(start, count, step int) []int {
s := make([]int, count)
for i := range s {
s[i] = start
start += step
}
return s
}
func main() {
s := []int{0, 1, 2, 3, 4, 5, 6, 7, 8, 9}
fmt.Println(s) // [0 1 2 3 4 5 6 7 8 9]
fmt.Println(NewSlice(10, 10, 1)) // [10 11 12 13 14 15 16 17 18 19]
fmt.Println(NewSlice(10, 10, 2)) // [10 12 14 16 18 20 22 24 26 28]
fmt.Println(NewSlice(-1, 10, -1)) // [-1 -2 -3 -4 -5 -6 -7 -8 -9 -10]
fmt.Println(NewSlice(20, 10, 0)) // [20 20 20 20 20 20 20 20 20 20]
}
package main
import "fmt"
func main() {
var a [11]int
for i := 1; i < len(a); i++ {
a[i] = i
}
fmt.Print(a)
}
You will get:
[0 1 2 3 4 5 6 7 8 9 10]
You can try this code GenerateSequenceInt, it's like Python's range:
package main
import (
"fmt"
"errors"
)
func GenerateSequenceInt(begin, end, step int) (sequence []int){
if step == 0 {
panic(errors.New("step must not be zero"))
}
count := 0
if (end > begin && step > 0) || (end < begin && step < 0) {
count = (end-step-begin)/step + 1
}
sequence = make([]int, count)
for i := 0; i < count; i, begin = i+1, begin+step {
sequence[i] = begin
}
return
}
func main() {
seq1 := GenerateSequenceInt(-1,11,-3)
fmt.Println(seq1)
seq2 := GenerateSequenceInt(1,-1,3)
fmt.Println(seq2)
seq3 := GenerateSequenceInt(1,1,1)
fmt.Println(seq3)
seq4 := GenerateSequenceInt(1, 11, 2)
fmt.Println(seq4)
seq5 := GenerateSequenceInt(1, -11, -2)
fmt.Println(seq5)
}
When you create an array/slice of number in golang, it is automatically populated with 0s. You could use the index instead of the value if you need a 0 based range that increases by 1.
nums := make([]int, 10)
for i := range nums {
//do something with index
fmt.Println(i)
}
If you needed a different starting point (not zero) and step (not 1) of the range, you could calculate it in a loop. For example, for a range of 10 numbers starting at 100, increasing by 5
nums := make([]int, 10)
for i := range nums {
nums[i] = 100 + i*5
}
Wrapping it as a function that returns a slice of numbers
func Range(start int, max int, step int) []int {
count := (max - start) / step
nums := make([]int, count)
for i := range nums {
nums[i] = start + i*step
}
return nums
}
Call it as
nums = Range(-50, 150, 10)
fmt.Println(nums)

How many iterations of the while loop must be made in order to locate it?

Im having some trouble trying to find out why the correct answer to this question is 4. Could anyone be kind enough to briefly explain why? Thanks in advanced! Here's the question:
Consider the array a with values as shown:
4, 7, 19, 25, 36, 37, 50, 100, 101, 205, 220, 271, 306, 321
where 4 is a [0] and 321 is a [13] . Suppose that the search method is called with
first = 0 and last = 13 to locate the key 205. How many iterations of the while loop must be made in order to locate it?
My guess is that you have to use a binary search here, as the items are sorted.
Given this array
[0] [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13]
4, 7, 19, 25, 36, 37, 50, 100, 101, 205, 220, 271, 306, 321
You initialize with:
left and right indexes: l = 0, r = 14 (= length of array)
Then you need these iterations:
m = (l + r) / 2 = (0 + 14) / 2 = 7
[m = 7] = 100 is < 205 ==> l = 7 + 1
m = (l + r) / 2 = (8 + 14) / 2 = 11
[m = 11] = 271 is > 205 ==> r = 11 - 1
m = (l + r) / 2 = (8 + 10) / 2 = 9
[m = 9] = 205 is = 205 ==> result = [9]
= 3 iterations!
However, a slight change to the algorithm can change the number of iterations. If you take r = N-1 instead of N as initial value then you get:
m = (l + r) / 2 = (0 + 13) / 2 = 6 (integer division)
[m = 6] = 50 is < 205 ==> l = 6 + 1
m = (l + r) / 2 = (7 + 13) / 2 = 10
[m = 10] = 220 is > 205 ==> r = 10 - 1
m = (l + r) / 2 = (7 + 9) / 2 = 8
[m = 8] = 101 is < 205 ==> l = 8 + 1
m = (l + r) / 2 = (9 + 9) / 2 = 9
[m = 9] = 205 is = 205 ==> result = [9]
= 4 iterations!
So the result depends on implementation details. Both variants are correct. Take care to choose the appropriate loop condition (I think l < r for the first and l <= r for the second algorithm.
Just go from the last index.
You start with index 13, which the first iteration you go to index 12, on the 4th iteration you are on index 9, which equals to 205.

Vectorize 2d convolution on matlab

I got this Code for computing two dimensional convolution for two given arrays.
[r,c] = size(x);
[m,n] = size(y);
h = rot90(y, 2);
center = floor((size(h)+1)/2);
Rep = zeros(r + m*2-2, c + n*2-2);
return
for x1 = m : m+r-1
for y1 = n : n+r-1
Rep(x1,y1) = x(x1-m+1, y1-n+1);
end
end
B = zeros(r+m-1,n+c-1);
for x1 = 1 : r+m-1
for y1 = 1 : n+c-1
for i = 1 : m
for j = 1 : n
B(x1, y1) = B(x1, y1) + (Rep(x1+i-1, y1+j-1) * h(i, j));
end
end
end
end
How can i vectorize it , so no for loops exist ?
Thanks in advance.
Here's what I came up with:
%// generate test matrices
x = randi(12, 4, 5)
y = [2 2 2;
2 0 2;
2 2 2]
[r,c] = size(x);
%[m,n] = size(y); %// didn't use this
h = rot90(y, 2);
center = floor((size(h)+1)/2);
Rep = zeros(size(x)+size(h)-1); %// create image of zeros big enough to pad x
Rep(center(1):center(1)+r-1, center(2):center(2)+c-1) = x; %// and copy x into the middle
%// all of this can be compressed onto one line, if desired
%// I'm just breaking it out into steps for clarity
CRep = im2col(Rep, size(h), 'sliding'); %// 'sliding' is the default, but just to be explicit
k = h(:); %// turn h into a column vector
BRow = bsxfun(#times, CRep, k); %// multiply k times each column of CRep
B = reshape(sum(BRow), r, c) %// take the sum of each column and reshape to match x
T = conv2(Rep, h, 'valid') %// take the convolution using conv2 to check
assert(isequal(B, T), 'Result did not match conv2.');
Here are the results of a sample run:
x =
11 12 11 2 8
5 9 2 3 2
7 9 3 4 8
7 10 8 5 4
y =
2 2 2
2 0 2
2 2 2
B =
52 76 56 52 14
96 120 106 80 50
80 102 100 70 36
52 68 62 54 34
T =
52 76 56 52 14
96 120 106 80 50
80 102 100 70 36
52 68 62 54 34

Inserting One Row Each Time in a Sequence from Matrix into Another Matrix After Every nth Row in Matlab

I have matrix A and matrix B. Matrix A is 100*3. Matrix B is 10*3. I need to insert one row from matrix B each time in a sequence into matrix A after every 10th row. The result would be Matrix A with 110*3. How can I do this in Matlab?
Here's another indexing-based approach:
n = 10;
C = [A; B];
[~, ind] = sort([1:size(A,1) n*(1:size(B,1))+.5]);
C = C(ind,:);
For canonical purposes, here's how you'd do it via loops. This is a bit inefficient since you're mutating the array at each iteration, but it's really simple to read. Given that your two matrices are stored in A (100 x 3) and B (10 x 3), you would do:
out = [];
for idx = 1 : 10
out = [out; A((idx-1)*10 + 1 : 10*idx,:); B(idx,:)];
end
At each iteration, we pick out 10 rows of A and 1 row of B and we concatenate these 11 rows onto out. This happens 10 times, resulting in 330 rows with 3 columns.
Here's an index-based approach:
%//pre-allocate output matrix
matrixC = zeros(110, 3);
%//create index array for the locations in matrixC that would be populated by matrixB
idxArr = (1:10) * 11;
%//place matrixB into matrixC
matrixC(idxArr,:) = matrixB;
%//place matrixA into matrixC
%//setdiff is used to exclude indexes already populated by values from matrixB
matrixC(setdiff(1:110, idxArr),:) = matrixA;
And just for fun here's the same approach sans magic numbers:
%//define how many rows to take from matrixA at once
numRows = 10;
%//get dimensions of input matrices
lengthA = size(matrixA, 1);
lengthB = size(matrixB, 1);
matrixC = zeros(lengthA + lengthB, 3);
idxArr = (1:lengthB) * (numRows + 1);
matrixC(idxArr,:) = matrixB;
matrixC(setdiff(1:size(matrixC, 1), idxArr),:) = matrixA;
Just for fun... Now with more robust test matrices!
A = ones(3, 100);
A(:) = 1:300;
A = A.'
B = ones(3, 10);
B(:) = 1:30;
B = B.' + 1000
C = reshape(A.', 3, 10, []);
C(:,end+1,:) = permute(B, [2 3 1]);
D = permute(C, [2 3 1]);
E = reshape(D, 110, 3)
Input:
A =
1 2 3
4 5 6
7 8 9
10 11 12
13 14 15
16 17 18
19 20 21
22 23 24
25 26 27
28 29 30
31 32 33
34 35 36
...
B =
1001 1002 1003
1004 1005 1006
...
Output:
E =
1 2 3
4 5 6
7 8 9
10 11 12
13 14 15
16 17 18
19 20 21
22 23 24
25 26 27
28 29 30
1001 1002 1003
31 32 33
34 35 36
...
Thanks to #Divakar for pointing out my previous error.
Solution Code
Here's an implementation based on logical indexing also known as masking and must be pretty efficient when working with large arrays -
%// Get sizes of A and B
[M,d] = size(A);
N = size(B,1);
%// Mask of row indices where rows from A would be placed
mask_idx = reshape([true(A_cutrow,M/A_cutrow) ; false(1,N)],[],1);
%// Pre-allocate with zeros:
%// http://undocumentedmatlab.com/blog/preallocation-performance
out(M+N,d) = 0;
%// Insert A and B using mask and ~mask
out(mask_idx,:) = A;
out(~mask_idx,:) = B;
Benchmarking
%// Setup inputs
A = rand(100000,3);
B = rand(10000,3);
A_cutrow = 10;
num_iter = 200; %// Number of iterations to be run for each approach
%// Warm up tic/toc.
for k = 1:50000
tic(); elapsed = toc();
end
disp(' ------------------------------- With MASKING')
tic
for iter = 1:num_iter
[M,d] = size(A);
N = size(B,1);
mask_idx = reshape([true(A_cutrow,M/A_cutrow) ; false(1,N)],[],1);
out(M+N,d) = 0;
out(mask_idx,:) = A;
out(~mask_idx,:) = B;
clear out
end
toc, clear mask_idx N M d iter
disp(' ------------------------------- With SORT')
tic
for iter = 1:num_iter
C = [A; B];
[~, ind] = sort([1:size(A,1) A_cutrow*(1:size(B,1))+.5]);
C = C(ind,:);
end
toc, clear C ind iter
disp(' ------------------------------- With RESHAPE+PERMUTE')
tic
for iter = 1:num_iter
[M,d] = size(A);
N = size(B,1);
C = reshape(A.', d, A_cutrow , []);
C(:,end+1,:) = permute(B, [2 3 1]);
D = permute(C, [2 1 3]);
out = reshape(permute(D,[1 3 2]),M+N,[]);
end
toc, clear out D C N M d iter
disp(' ------------------------------- With SETDIFF')
tic
for iter = 1:num_iter
lengthA = size(A, 1);
lengthB = size(B, 1);
matrixC = zeros(lengthA + lengthB, 3);
idxArr = (1:lengthB) * (A_cutrow + 1);
matrixC(idxArr,:) = B;
matrixC(setdiff(1:size(matrixC, 1), idxArr),:) = A;
end
toc, clear matrixC idxArr lengthA lengthB
disp(' ------------------------------- With FOR-LOOP')
tic
for iter = 1:num_iter
[M,d] = size(A);
N = size(B,1);
Mc = M/A_cutrow;
out(M+N,d) = 0;
for idx = 1 : Mc
out( 1+(idx-1)*(A_cutrow +1): idx*(A_cutrow+1), :) = ...
[A( 1+(idx-1)*A_cutrow : idx*A_cutrow , : ) ; B(idx,:)];
end
clear out
end
toc
Runtimes
Case #1: A as 100 x 3 and B as 10 x 3
------------------------------- With MASKING
Elapsed time is 4.987088 seconds.
------------------------------- With SORT
Elapsed time is 5.056301 seconds.
------------------------------- With RESHAPE+PERMUTE
Elapsed time is 5.170416 seconds.
------------------------------- With SETDIFF
Elapsed time is 35.063020 seconds.
------------------------------- With FOR-LOOP
Elapsed time is 12.118992 seconds.
Case #2: A as 100000 x 3 and B as 10000 x 3
------------------------------- With MASKING
Elapsed time is 1.167707 seconds.
------------------------------- With SORT
Elapsed time is 2.667149 seconds.
------------------------------- With RESHAPE+PERMUTE
Elapsed time is 2.603110 seconds.
------------------------------- With SETDIFF
Elapsed time is 3.153900 seconds.
------------------------------- With FOR-LOOP
Elapsed time is 19.822912 seconds.
Please note that num_iter was different for these two cases, as the idea was to keep the runtimes > 1 sec mark to compensate for tic-toc overheads.

Resources