Related
I'm using Julia to ingest a large two dimensional data array (data) of size 1000 x 32768; I need to break up the array into smaller square arrays along both dimensions. For instance, I would like to break data into a grid of smaller, square arrays similar to the following image:
Note that no pixels get left out -- when another square cannot be fit in along either axis, the last possible array of square pixels is returned as another array (hence the shifted pink squares on the right hand side).
Currently, I'm doing this through a function I built to decimate the raw dataset:
function decimate_square(data,fraction=4)
# Read size of input data / calculate length of square side
sy,sx = size(data)
square_side = Int(round(sy/fraction))
# Number of achievable full squares
itersx,itersy = [Int(floor(s/square_side)) for s in [sx,sy]]
# Find left/right X values
for ix in 1:itersx
if ix!=itersx
# Full sliding square can be calculated
left = square_side*(ix-1) + 1
right = square_side*(ix)
else
# Capture last square of data
left = sx-square_side + 1
right = sx
end
# Find top/bottom Y values for each X
for iy in 1:itersy
if iy!=itersy
# Full sliding square can be calculated
top = square_side*(iy-1) + 1
bottom = square_side*(iy)
else
# Capture last square of data
top = sy-square_side + 1
bottom = sy
end
# Record data in 3d stack
cursquare = data[top:bottom,left:right]
if (ix==1)&&(iy==1); global dstack=cursquare
else; dstack=cat(dstack,cursquare,dims=3)
end
end
end
return dstack
end
Which currently takes ~20 seconds to run:
rand_arr = rand(1000,32768)
t1 = Dates.now()
dec_arr = decimate_square(rand_arr)
t2 = Dates.now()
#info(t2-t1)
[ Info: 19666 milliseconds
This is the biggest bottleneck of my analysis. Is there a pre-built function that I can use, or is there a more efficient way to decimate my array?
You can take views as Przemyslaw Szufel suggests, and the CartesianIndex type comes in handy for selecting blocks of the matrix.
julia> function squareviews(data, fraction = 4)
squareside = floor(Int, size(data, 1) / fraction)
[#view(M[CartesianIndex(ix-squareside+1, iy-squareside+1):CartesianIndex(ix, iy)])
for ix in squareside:squareside:size(data, 1),
iy in squareside:squareside:size(data, 2)]
end
squareviews (generic function with 2 methods)
julia> result = squareviews(M)
4×40 Matrix{SubArray{Int64, 2, Matrix{Int64}, Tuple{UnitRange{Int64}, UnitRange{Int64}}, false}}:
[346 392 … 746 429; 380 193 … 476 757; … ; 424 329 … 285 427; 591 792 … 710 891] … [758 916 … 7 185; 26 846 … 631 808; … ; 945 713 … 875 137; 793 655 … 400 322]
[55 919 … 402 728; 292 238 … 266 636; … ; 62 490 … 913 126; 293 475 … 492 20] [53 8 … 146 365; 216 673 … 157 909; … ; 955 635 … 332 945; 354 913 … 922 272]
[278 966 … 128 334; 700 560 … 226 701; … ; 529 398 … 17 674; 237 830 … 4 788] [239 274 … 983 911; 591 669 … 762 675; … ; 213 949 … 917 903; 336 890 … 633 578]
[723 483 … 135 283; 729 579 … 1000 942; … ; 987 383 … 764 544; 682 942 … 376 179] [370 859 … 444 566; 34 106 … 320 161; … ; 310 41 … 868 349; 719 341 … 718 800]
This divides the data matrix into blocks such that result[2, 3] gives the square that is 2nd from the top and 3rd from the left. (My matrix M was 100x1000 in size, so there are 100/25 = 4 blocks vertically and 1000/25 = 40 blocks horizontally.)
If you want the results linearly like in your original function, you can instead have the second line of the function be:
julia> function squareviews(data, fraction = 4)
squareside = floor(Int, size(data, 1) / fraction)
[#view(M[CartesianIndex(ix-squareside+1, iy-squareside+1):CartesianIndex(ix, iy)])
for iy in squareside:squareside:size(data, 2)
for ix in squareside:squareside:size(data, 1)]
end
squareviews (generic function with 2 methods)
julia> squareviews(M)
160-element Vector{SubArray{Int64, 2, Matrix{Int64}, Tuple{UnitRange{Int64}, UnitRange{Int64}}, false}}:
(Note the subtle changes in the for syntax - the iy comes before ix here, there's no comma, and there's an extra for.)
This returns a vector of square matrices (views).
Your original function returned a three-dimensional matrix, in which you'd access values as originalresult[i, j, k]. Here, the equivalent would be result[k][i, j].
There's a lot of stuff going on in your code with is not recommended and making things slow. Here's a somewhat idiomatic solution, with the additional bonus of generalizing to arbitrary ranks:
julia> function square_indices(data; fraction=4)
splits = cld.(size(data), fraction)
return Iterators.map(CartesianIndices, Iterators.product(Iterators.partition.(axes(data), splits)...))
end
square_indices (generic function with 1 method)
The result of this is an iterator over CartesianIndices, which are objects that you can use to index your squares. Either the regular data[ix], or view(data, ix), which does not create a copy. (Different fractions per dimension are possible, try test_square_indices(println, 4, 4, 4; fraction=(2, 1, 1)).)
And to see whether it works as expected:
julia> function test_square_indices(f, s...; fraction=4)
arr = reshape(1:prod(s), s...)
for ix in square_indices(arr; fraction)
f(view(arr, ix))
end
end
test_square_indices (generic function with 1 method)
julia> # just try this on some moderatly costly function
#btime test_square_indices(v -> inv.(v), 1000, 32768)
81.980 ms (139 allocations: 250.01 MiB)
julia> test_square_indices(println, 9)
[1, 2, 3]
[4, 5, 6]
[7, 8, 9]
julia> test_square_indices(println, 9, 5)
[1 10; 2 11; 3 12]
[4 13; 5 14; 6 15]
[7 16; 8 17; 9 18]
[19 28; 20 29; 21 30]
[22 31; 23 32; 24 33]
[25 34; 26 35; 27 36]
[37; 38; 39;;]
[40; 41; 42;;]
[43; 44; 45;;]
julia> reshape(1:9*5, 9, 5)
9×5 reshape(::UnitRange{Int64}, 9, 5) with eltype Int64:
1 10 19 28 37
2 11 20 29 38
3 12 21 30 39
4 13 22 31 40
5 14 23 32 41
6 15 24 33 42
7 16 25 34 43
8 17 26 35 44
9 18 27 36 45
julia> test_square_indices(println, 4, 4, 4; fraction=2)
[1 5; 2 6;;; 17 21; 18 22]
[3 7; 4 8;;; 19 23; 20 24]
[9 13; 10 14;;; 25 29; 26 30]
[11 15; 12 16;;; 27 31; 28 32]
[33 37; 34 38;;; 49 53; 50 54]
[35 39; 36 40;;; 51 55; 52 56]
[41 45; 42 46;;; 57 61; 58 62]
[43 47; 44 48;;; 59 63; 60 64]
julia> reshape(1:4*4*4, 4, 4, 4)
4×4×4 reshape(::UnitRange{Int64}, 4, 4, 4) with eltype Int64:
[:, :, 1] =
1 5 9 13
2 6 10 14
3 7 11 15
4 8 12 16
[:, :, 2] =
17 21 25 29
18 22 26 30
19 23 27 31
20 24 28 32
[:, :, 3] =
33 37 41 45
34 38 42 46
35 39 43 47
36 40 44 48
[:, :, 4] =
49 53 57 61
50 54 58 62
51 55 59 63
52 56 60 64
Here's a bit of an illustration of how this works:
julia> data = reshape(1:9*5, 9, 5); fraction = 3;
julia> size(data)
(9, 5)
julia> # chunk sizes
splits = cld.(size(data), fraction)
(3, 2)
julia> # every dimension chunked
Iterators.partition.(axes(data), splits) .|> collect
(UnitRange{Int64}[1:3, 4:6, 7:9], UnitRange{Int64}[1:2, 3:4, 5:5])
julia> # cross product of all chunks
Iterators.product(Iterators.partition.(axes(data), splits)...) .|> collect
3×3 Matrix{Vector{UnitRange{Int64}}}:
[1:3, 1:2] [1:3, 3:4] [1:3, 5:5]
[4:6, 1:2] [4:6, 3:4] [4:6, 5:5]
[7:9, 1:2] [7:9, 3:4] [7:9, 5:5]
You could just go with views. Suppose you want to slice your data into 64 matrices, each having size 1000 x 512. In that case you could do:
dats = view.(Ref(rand_arr),Ref(1:1000), [range(1+(i-1)*512,i*512) for i in 1:64])
The time for this on my machine is 600 nanoseconds:
julia> #btime view.(Ref($rand_arr),Ref(1:1000), [range(1+(i-1)*512,i*512) for i in 1:64]);
595.604 ns (3 allocations: 4.70 KiB)
I have a 2D matrix with in the 1st dimension different channels, and in the 2nd dimension time samples. I want to rearrange this to a 3D matrix, with in the 1st and 2nd dimension channels, and in the 3rd time samples.
The channels have to mapped according to a certain mapping. Right now I am using a for-loop to do so, but what would be a no-loop solution?
N_samples = 1000;
N_channels = 64;
channel_mapping = reshape(1:64, [8 8]).';
% Results in mapping: (can also be random)
% 1 2 3 4 5 6 7 8
% 9 10 11 12 13 14 15 16
% 17 18 19 20 21 22 23 24
% 25 26 27 28 29 30 31 32
% 33 34 35 36 37 38 39 40
% 41 42 43 44 45 46 47 48
% 49 50 51 52 53 55 55 56
% 57 58 59 60 61 62 63 64
data = rand(N_channels, N_samples);
data_grid = NaN(8,8, N_samples);
for k = 1:N_samples
tmp = data(:, k);
data_grid(:, :, k) = tmp(channel_mapping);
end
You can do it in one go as follows:
data_grid = reshape(data(channel_mapping, :), 8, 8, []);
I have the following multidimensional array:
a(:,:,1) =
11 12 13
21 22 23
a(:,:,2) =
21 22 23
31 32 33
a(:,:,3) =
31 32 33
41 42 43
The end result looks as follows:
e =
11 21 31
12 22 32
13 23 33
21 31 41
22 32 42
23 33 43
In order to obtain array e, I did the following:
b=permute(a,[2,1,3])
c=reshape(b,1,6,[])
d=permute(c,[2,1,3])
Is there a cleaner or more elegant way of achieving the same result?
Here permute used once:
b = permute(a,[2 1 3]);
result = reshape(b, [], 3)
You can do it like the following:
reshape([a(:,:,1);a(:,:,2);a(:,:,3)].',6,3)
I would like to compute a product iterator using Iterators.jl.
Let's say I have an array of UnitRanges tab with a priori unknown size.
I would like to compute the cartesian product of the elements of tab.
For example if tab length is 2 and tab[1] = a and tab[2] = b I want to compute product(a,b) from Iterators.jl.
I want to make a generic function that compute the cartesian product of every component in tab.
I tried something like this
prod = tab[1]
for i in tab[2:end]
prod = product(prod,i)
end
However if tab is length 3, components a,b and c, I obtain in prod elements under the form (1,(3,2)) and not (1,3,2). With 1 element of c, 3 element of b and 2 element of a.
In v0.5, there is now Base.product, which is much better than Iterators.product.
It can handle as many arrays as needed, and it even has a shape:
julia> collect(Base.product([1, 2], [3, 4]))
2×2 Array{Tuple{Int64,Int64},2}:
(1,3) (1,4)
(2,3) (2,4)
julia> collect(Base.product(1:5, 1:3, 1:2, 1:2))
5×3×2×2 Array{NTuple{4,Int64},4}:
[:, :, 1, 1] =
(1,1,1,1) (1,2,1,1) (1,3,1,1)
(2,1,1,1) (2,2,1,1) (2,3,1,1)
(3,1,1,1) (3,2,1,1) (3,3,1,1)
(4,1,1,1) (4,2,1,1) (4,3,1,1)
(5,1,1,1) (5,2,1,1) (5,3,1,1)
[:, :, 2, 1] =
(1,1,2,1) (1,2,2,1) (1,3,2,1)
(2,1,2,1) (2,2,2,1) (2,3,2,1)
(3,1,2,1) (3,2,2,1) (3,3,2,1)
(4,1,2,1) (4,2,2,1) (4,3,2,1)
(5,1,2,1) (5,2,2,1) (5,3,2,1)
[:, :, 1, 2] =
(1,1,1,2) (1,2,1,2) (1,3,1,2)
(2,1,1,2) (2,2,1,2) (2,3,1,2)
(3,1,1,2) (3,2,1,2) (3,3,1,2)
(4,1,1,2) (4,2,1,2) (4,3,1,2)
(5,1,1,2) (5,2,1,2) (5,3,1,2)
[:, :, 2, 2] =
(1,1,2,2) (1,2,2,2) (1,3,2,2)
(2,1,2,2) (2,2,2,2) (2,3,2,2)
(3,1,2,2) (3,2,2,2) (3,3,2,2)
(4,1,2,2) (4,2,2,2) (4,3,2,2)
(5,1,2,2) (5,2,2,2) (5,3,2,2)
The shape is extremely useful for map. For instance, here's how to create a multiplication table using Base.product:
julia> map(prod, Base.product(1:9, 1:9))
9×9 Array{Int64,2}:
1 2 3 4 5 6 7 8 9
2 4 6 8 10 12 14 16 18
3 6 9 12 15 18 21 24 27
4 8 12 16 20 24 28 32 36
5 10 15 20 25 30 35 40 45
6 12 18 24 30 36 42 48 54
7 14 21 28 35 42 49 56 63
8 16 24 32 40 48 56 64 72
9 18 27 36 45 54 63 72 81
Of course, if you don't need the shape, then you are free to ignore it — it will still iterate properly.
And Base.product is fast too!
I have a 3x3x2000 array of rotation matrices that I need to transform into a 2000x9 array.
I think I have to use a combination of permute() and reshape(), but I don't get the correct output order.
This is what I need:
First row of 3x3 array needs to be columns 1:3 in the output
Second row of 3x3 array needs to be columns 4:6 in the output
Third row of 3x3 array needs to be columns 7:9 in the output
I have tried all possible combinations of numbers 1 2 3 in the following code:
out1 = permute(input, [2 3 1]);
out2 = reshape(out1, [2000 9]);
But I always end up with the wrong order. Any tips for a Matlab newbie?
How about a simple for-loop?
for i=1:size(myinput,3)
myoutput(i,:)=[myinput(1,:,i) myinput(2,:,i) myinput(3,:,i)];
% or
% myoutput(i,:)=reshape(myinput(:,:,i),[],9);
end
It's not simple as using permute and reshape, but it is transparent and easier for debugging. Once everything in your program runs perfectly, you can consider to rewrite such for-loops in your code...
You had a mix-up in your permute
a = reshape(1:9*6, 3, 3, []);
a is a 3x3x6 matrix, each
a(:,:,i) = 9*(i-1) + [1 4 7
2 5 8
3 6 9];
So
out1 = permute(a, [3,1,2]);
out2 = reshape(out1, [], 9);
Or in one line
out3 = reshape(permute(a, [3,1,2]), [], 9);
So
out2 = out3 =
1 2 3 4 5 6 7 8 9
10 11 12 13 14 15 16 17 18
19 20 21 22 23 24 25 26 27
28 29 30 31 32 33 34 35 36
37 38 39 40 41 42 43 44 45
46 47 48 49 50 51 52 53 54