Julia : Cartesian product of multiple arrays - arrays

I would like to compute a product iterator using Iterators.jl.
Let's say I have an array of UnitRanges tab with a priori unknown size.
I would like to compute the cartesian product of the elements of tab.
For example if tab length is 2 and tab[1] = a and tab[2] = b I want to compute product(a,b) from Iterators.jl.
I want to make a generic function that compute the cartesian product of every component in tab.
I tried something like this
prod = tab[1]
for i in tab[2:end]
prod = product(prod,i)
end
However if tab is length 3, components a,b and c, I obtain in prod elements under the form (1,(3,2)) and not (1,3,2). With 1 element of c, 3 element of b and 2 element of a.

In v0.5, there is now Base.product, which is much better than Iterators.product.
It can handle as many arrays as needed, and it even has a shape:
julia> collect(Base.product([1, 2], [3, 4]))
2×2 Array{Tuple{Int64,Int64},2}:
(1,3) (1,4)
(2,3) (2,4)
julia> collect(Base.product(1:5, 1:3, 1:2, 1:2))
5×3×2×2 Array{NTuple{4,Int64},4}:
[:, :, 1, 1] =
(1,1,1,1) (1,2,1,1) (1,3,1,1)
(2,1,1,1) (2,2,1,1) (2,3,1,1)
(3,1,1,1) (3,2,1,1) (3,3,1,1)
(4,1,1,1) (4,2,1,1) (4,3,1,1)
(5,1,1,1) (5,2,1,1) (5,3,1,1)
[:, :, 2, 1] =
(1,1,2,1) (1,2,2,1) (1,3,2,1)
(2,1,2,1) (2,2,2,1) (2,3,2,1)
(3,1,2,1) (3,2,2,1) (3,3,2,1)
(4,1,2,1) (4,2,2,1) (4,3,2,1)
(5,1,2,1) (5,2,2,1) (5,3,2,1)
[:, :, 1, 2] =
(1,1,1,2) (1,2,1,2) (1,3,1,2)
(2,1,1,2) (2,2,1,2) (2,3,1,2)
(3,1,1,2) (3,2,1,2) (3,3,1,2)
(4,1,1,2) (4,2,1,2) (4,3,1,2)
(5,1,1,2) (5,2,1,2) (5,3,1,2)
[:, :, 2, 2] =
(1,1,2,2) (1,2,2,2) (1,3,2,2)
(2,1,2,2) (2,2,2,2) (2,3,2,2)
(3,1,2,2) (3,2,2,2) (3,3,2,2)
(4,1,2,2) (4,2,2,2) (4,3,2,2)
(5,1,2,2) (5,2,2,2) (5,3,2,2)
The shape is extremely useful for map. For instance, here's how to create a multiplication table using Base.product:
julia> map(prod, Base.product(1:9, 1:9))
9×9 Array{Int64,2}:
1 2 3 4 5 6 7 8 9
2 4 6 8 10 12 14 16 18
3 6 9 12 15 18 21 24 27
4 8 12 16 20 24 28 32 36
5 10 15 20 25 30 35 40 45
6 12 18 24 30 36 42 48 54
7 14 21 28 35 42 49 56 63
8 16 24 32 40 48 56 64 72
9 18 27 36 45 54 63 72 81
Of course, if you don't need the shape, then you are free to ignore it — it will still iterate properly.
And Base.product is fast too!

Related

Splitting a large matrix

I'm using Julia to ingest a large two dimensional data array (data) of size 1000 x 32768; I need to break up the array into smaller square arrays along both dimensions. For instance, I would like to break data into a grid of smaller, square arrays similar to the following image:
Note that no pixels get left out -- when another square cannot be fit in along either axis, the last possible array of square pixels is returned as another array (hence the shifted pink squares on the right hand side).
Currently, I'm doing this through a function I built to decimate the raw dataset:
function decimate_square(data,fraction=4)
# Read size of input data / calculate length of square side
sy,sx = size(data)
square_side = Int(round(sy/fraction))
# Number of achievable full squares
itersx,itersy = [Int(floor(s/square_side)) for s in [sx,sy]]
# Find left/right X values
for ix in 1:itersx
if ix!=itersx
# Full sliding square can be calculated
left = square_side*(ix-1) + 1
right = square_side*(ix)
else
# Capture last square of data
left = sx-square_side + 1
right = sx
end
# Find top/bottom Y values for each X
for iy in 1:itersy
if iy!=itersy
# Full sliding square can be calculated
top = square_side*(iy-1) + 1
bottom = square_side*(iy)
else
# Capture last square of data
top = sy-square_side + 1
bottom = sy
end
# Record data in 3d stack
cursquare = data[top:bottom,left:right]
if (ix==1)&&(iy==1); global dstack=cursquare
else; dstack=cat(dstack,cursquare,dims=3)
end
end
end
return dstack
end
Which currently takes ~20 seconds to run:
rand_arr = rand(1000,32768)
t1 = Dates.now()
dec_arr = decimate_square(rand_arr)
t2 = Dates.now()
#info(t2-t1)
[ Info: 19666 milliseconds
This is the biggest bottleneck of my analysis. Is there a pre-built function that I can use, or is there a more efficient way to decimate my array?
You can take views as Przemyslaw Szufel suggests, and the CartesianIndex type comes in handy for selecting blocks of the matrix.
julia> function squareviews(data, fraction = 4)
squareside = floor(Int, size(data, 1) / fraction)
[#view(M[CartesianIndex(ix-squareside+1, iy-squareside+1):CartesianIndex(ix, iy)])
for ix in squareside:squareside:size(data, 1),
iy in squareside:squareside:size(data, 2)]
end
squareviews (generic function with 2 methods)
julia> result = squareviews(M)
4×40 Matrix{SubArray{Int64, 2, Matrix{Int64}, Tuple{UnitRange{Int64}, UnitRange{Int64}}, false}}:
[346 392 … 746 429; 380 193 … 476 757; … ; 424 329 … 285 427; 591 792 … 710 891] … [758 916 … 7 185; 26 846 … 631 808; … ; 945 713 … 875 137; 793 655 … 400 322]
[55 919 … 402 728; 292 238 … 266 636; … ; 62 490 … 913 126; 293 475 … 492 20] [53 8 … 146 365; 216 673 … 157 909; … ; 955 635 … 332 945; 354 913 … 922 272]
[278 966 … 128 334; 700 560 … 226 701; … ; 529 398 … 17 674; 237 830 … 4 788] [239 274 … 983 911; 591 669 … 762 675; … ; 213 949 … 917 903; 336 890 … 633 578]
[723 483 … 135 283; 729 579 … 1000 942; … ; 987 383 … 764 544; 682 942 … 376 179] [370 859 … 444 566; 34 106 … 320 161; … ; 310 41 … 868 349; 719 341 … 718 800]
This divides the data matrix into blocks such that result[2, 3] gives the square that is 2nd from the top and 3rd from the left. (My matrix M was 100x1000 in size, so there are 100/25 = 4 blocks vertically and 1000/25 = 40 blocks horizontally.)
If you want the results linearly like in your original function, you can instead have the second line of the function be:
julia> function squareviews(data, fraction = 4)
squareside = floor(Int, size(data, 1) / fraction)
[#view(M[CartesianIndex(ix-squareside+1, iy-squareside+1):CartesianIndex(ix, iy)])
for iy in squareside:squareside:size(data, 2)
for ix in squareside:squareside:size(data, 1)]
end
squareviews (generic function with 2 methods)
julia> squareviews(M)
160-element Vector{SubArray{Int64, 2, Matrix{Int64}, Tuple{UnitRange{Int64}, UnitRange{Int64}}, false}}:
(Note the subtle changes in the for syntax - the iy comes before ix here, there's no comma, and there's an extra for.)
This returns a vector of square matrices (views).
Your original function returned a three-dimensional matrix, in which you'd access values as originalresult[i, j, k]. Here, the equivalent would be result[k][i, j].
There's a lot of stuff going on in your code with is not recommended and making things slow. Here's a somewhat idiomatic solution, with the additional bonus of generalizing to arbitrary ranks:
julia> function square_indices(data; fraction=4)
splits = cld.(size(data), fraction)
return Iterators.map(CartesianIndices, Iterators.product(Iterators.partition.(axes(data), splits)...))
end
square_indices (generic function with 1 method)
The result of this is an iterator over CartesianIndices, which are objects that you can use to index your squares. Either the regular data[ix], or view(data, ix), which does not create a copy. (Different fractions per dimension are possible, try test_square_indices(println, 4, 4, 4; fraction=(2, 1, 1)).)
And to see whether it works as expected:
julia> function test_square_indices(f, s...; fraction=4)
arr = reshape(1:prod(s), s...)
for ix in square_indices(arr; fraction)
f(view(arr, ix))
end
end
test_square_indices (generic function with 1 method)
julia> # just try this on some moderatly costly function
#btime test_square_indices(v -> inv.(v), 1000, 32768)
81.980 ms (139 allocations: 250.01 MiB)
julia> test_square_indices(println, 9)
[1, 2, 3]
[4, 5, 6]
[7, 8, 9]
julia> test_square_indices(println, 9, 5)
[1 10; 2 11; 3 12]
[4 13; 5 14; 6 15]
[7 16; 8 17; 9 18]
[19 28; 20 29; 21 30]
[22 31; 23 32; 24 33]
[25 34; 26 35; 27 36]
[37; 38; 39;;]
[40; 41; 42;;]
[43; 44; 45;;]
julia> reshape(1:9*5, 9, 5)
9×5 reshape(::UnitRange{Int64}, 9, 5) with eltype Int64:
1 10 19 28 37
2 11 20 29 38
3 12 21 30 39
4 13 22 31 40
5 14 23 32 41
6 15 24 33 42
7 16 25 34 43
8 17 26 35 44
9 18 27 36 45
julia> test_square_indices(println, 4, 4, 4; fraction=2)
[1 5; 2 6;;; 17 21; 18 22]
[3 7; 4 8;;; 19 23; 20 24]
[9 13; 10 14;;; 25 29; 26 30]
[11 15; 12 16;;; 27 31; 28 32]
[33 37; 34 38;;; 49 53; 50 54]
[35 39; 36 40;;; 51 55; 52 56]
[41 45; 42 46;;; 57 61; 58 62]
[43 47; 44 48;;; 59 63; 60 64]
julia> reshape(1:4*4*4, 4, 4, 4)
4×4×4 reshape(::UnitRange{Int64}, 4, 4, 4) with eltype Int64:
[:, :, 1] =
1 5 9 13
2 6 10 14
3 7 11 15
4 8 12 16
[:, :, 2] =
17 21 25 29
18 22 26 30
19 23 27 31
20 24 28 32
[:, :, 3] =
33 37 41 45
34 38 42 46
35 39 43 47
36 40 44 48
[:, :, 4] =
49 53 57 61
50 54 58 62
51 55 59 63
52 56 60 64
Here's a bit of an illustration of how this works:
julia> data = reshape(1:9*5, 9, 5); fraction = 3;
julia> size(data)
(9, 5)
julia> # chunk sizes
splits = cld.(size(data), fraction)
(3, 2)
julia> # every dimension chunked
Iterators.partition.(axes(data), splits) .|> collect
(UnitRange{Int64}[1:3, 4:6, 7:9], UnitRange{Int64}[1:2, 3:4, 5:5])
julia> # cross product of all chunks
Iterators.product(Iterators.partition.(axes(data), splits)...) .|> collect
3×3 Matrix{Vector{UnitRange{Int64}}}:
[1:3, 1:2] [1:3, 3:4] [1:3, 5:5]
[4:6, 1:2] [4:6, 3:4] [4:6, 5:5]
[7:9, 1:2] [7:9, 3:4] [7:9, 5:5]
You could just go with views. Suppose you want to slice your data into 64 matrices, each having size 1000 x 512. In that case you could do:
dats = view.(Ref(rand_arr),Ref(1:1000), [range(1+(i-1)*512,i*512) for i in 1:64])
The time for this on my machine is 600 nanoseconds:
julia> #btime view.(Ref($rand_arr),Ref(1:1000), [range(1+(i-1)*512,i*512) for i in 1:64]);
595.604 ns (3 allocations: 4.70 KiB)

Is there a way to rotate 3D arrays in Julia?

I am trying to rotate a 3D array in julia as if it represents a physical object in 3D space. Essentially, I want to know if there is a way to rotate an array by increments of 90 degrees along the x-, y-, and/or the z-axis.
In 2D it would be something like this if I were to rotate counter-clockwise...
1 2 3 3 6 9
4 5 6 -----> 2 5 8
7 8 9 1 4 7
I would want the same logic to apply in 3D as well.
Any help is appreciated.
For two-dimensional Matrices you have functions such as rotl90, rotr90 and rot180. Those can be combined with mapslices to get the desired effect. For an example below is a rotation over dimensions 1 and 2 for each cut of array in the dimension 3.
julia> A=collect(reshape(1:27,3,3,3))
3×3×3 Array{Int64,3}:
[:, :, 1] =
1 4 7
2 5 8
3 6 9
[:, :, 2] =
10 13 16
11 14 17
12 15 18
[:, :, 3] =
19 22 25
20 23 26
21 24 27
julia> mapslices(rotr90,A,dims=[1,2])
3×3×3 Array{Int64,3}:
[:, :, 1] =
3 2 1
6 5 4
9 8 7
[:, :, 2] =
12 11 10
15 14 13
18 17 16
[:, :, 3] =
21 20 19
24 23 22
27 26 25

How to fill an array by row in Julia

I would like to fill an Array object by row in the Julia language.
The reshape function wants to fill by column (Julia is column major).
julia> reshape(1:15, 3,5)
3x5 Array{Int64,2}:
1 4 7 10 13
2 5 8 11 14
3 6 9 12 15
Is there a way to persuade it to fill by row? It feels like there should be an obvious answer, but I've not found one.
One suggestion:
julia> reshape(1:15, 5, 3) |> transpose
3x5 Array{Int64,2}:
1 2 3 4 5
6 7 8 9 10
11 12 13 14 15
With array comprehension:
julia> [i+5*j for j=0:2,i=1:5]
3x5 Array{Int64,2}:
1 2 3 4 5
6 7 8 9 10
11 12 13 14 15
Ah, it's just more than 10x times faster than other suggestion (actually, an embarrassing 100x on my initial benchmark).
permutedims is another choice when dealing with more general multi-way arrays.
julia> permutedims(reshape(1:24, 2,3,4), [2,1,3])
3x2x4 Array{Int64,3}:
[:, :, 1] =
1 2
3 4
5 6
[:, :, 2] =
7 8
9 10
11 12
[:, :, 3] =
13 14
15 16
17 18
[:, :, 4] =
19 20
21 22
23 24
however, it's slowest among other suggestions in your specific case.

Data field shifting through a vector of data in matlab

I need to create a data field that will go through a vector. Data field is constant length, and it is going through the data vector shifting data field with data field length. I need the mean value of that field (A vector) that corresponds to a mean value of another field (B vector).
Example:
A=[1 5 7 8 9 10 11 13 15 18 19 25 28 30 35 40 45 48 50 51];
B=[2 4 8 9 12 15 16 18 19 20 25 27 30 35 39 40 45 48 50 55];
I want to do next:
A=[{1 5 7 8 9} 10 11 13 15 18 19 25 28 30 35 40 45 48 50 51];
B=[{2 4 8 9 12} 15 16 18 19 20 25 27 30 35 39 40 45 48 50 55];
I want to take data from field of 5 points and get mean value. And then shift whole data field with data field length.
A=[1 5 7 8 9 {10 11 13 15 18} 19 25 28 30 35 40 45 48 50 51];
B=[2 4 8 9 12 {15 16 18 19 20} 25 27 30 35 39 40 45 48 50 55];
I need two vectors, C and D with mean values of this method.
C=[6 13.4 27.4 45.2];
D=[7 17.6 31.2 47.6];
I started something with
n = length(A);
for k = 1:n
....
but nothing I tried worked.
reshape the vector into a 5-row matrix and then compute the mean of each column:
C = mean(reshape(A,5,[]),1);
D = mean(reshape(B,5,[]),1)

Matlab: reshape 3-dimensional array into 2-dimensional array

I have a 3x3x2000 array of rotation matrices that I need to transform into a 2000x9 array.
I think I have to use a combination of permute() and reshape(), but I don't get the correct output order.
This is what I need:
First row of 3x3 array needs to be columns 1:3 in the output
Second row of 3x3 array needs to be columns 4:6 in the output
Third row of 3x3 array needs to be columns 7:9 in the output
I have tried all possible combinations of numbers 1 2 3 in the following code:
out1 = permute(input, [2 3 1]);
out2 = reshape(out1, [2000 9]);
But I always end up with the wrong order. Any tips for a Matlab newbie?
How about a simple for-loop?
for i=1:size(myinput,3)
myoutput(i,:)=[myinput(1,:,i) myinput(2,:,i) myinput(3,:,i)];
% or
% myoutput(i,:)=reshape(myinput(:,:,i),[],9);
end
It's not simple as using permute and reshape, but it is transparent and easier for debugging. Once everything in your program runs perfectly, you can consider to rewrite such for-loops in your code...
You had a mix-up in your permute
a = reshape(1:9*6, 3, 3, []);
a is a 3x3x6 matrix, each
a(:,:,i) = 9*(i-1) + [1 4 7
2 5 8
3 6 9];
So
out1 = permute(a, [3,1,2]);
out2 = reshape(out1, [], 9);
Or in one line
out3 = reshape(permute(a, [3,1,2]), [], 9);
So
out2 = out3 =
1 2 3 4 5 6 7 8 9
10 11 12 13 14 15 16 17 18
19 20 21 22 23 24 25 26 27
28 29 30 31 32 33 34 35 36
37 38 39 40 41 42 43 44 45
46 47 48 49 50 51 52 53 54

Resources